- The Desktop
In particular, the use of 'Desktop' as the default downloads folder rather than the far-too-obvious 'Downloads', making 'the bit that appears behind all the other windows' cluttered with files. Showing mounted devices like iPods on the desktop sometimes makes sense, but essentially it should just be a place for the files you're currently working on, before they have a proper home.
Firefox on all operating systems lets you change the default folder for downloads, but on OS X the 'Open With' command still respects the system 'downloads folder' setting for some reason (it's already filed as a bug), so you have to change the setting in Safari.
Anyone whose installer adds an application-launching icon to the Desktop should have to pay a fine, reduced only slightly when the installer offers an option to add an icon that's not off by default.
In Gnome you can use gconf-editor to change the show_desktop setting in apps→nautilus→preferences, to disable the desktop completely. There's an option to use your Home folder as the desktop too.
- The Trash/Recycle Bin/Wastebasket
Trying to keep a tidy screen is much harder when there's an icon in the corner that says "Empty Me!" whenever you delete a file. In Ubuntu you can remove it from the panel and still use "Empty Wastebasket" from the File menu in Nautilus; in Windows you can hide it from the desktop (!) with TweakUI; in Mac OS X you can replace the icon with a transparent one using CandyBar.