You should be pointing it towards the Developer directory, not the Xcode application bundle. Run this:

sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer

With recent versions of Xcode, you can go to XcodePreferences…Locations and pick one of the options for Command Line Tools to set the location.