I use this command:

sudo /Developer/Library/uninstall-devtools --mode=all

Edit (1 year later):

If you've downloaded Xcode from the App Store, it's self-contained, as @mediaslave suggests. You can just drag it to the trash or use AppZapper (or a similar utility) to remove the developer tools.

If you're looking to update it, you can also do that straight through the App Store.