Did You Start ssh-agent?

You might need to start ssh-agent before you run the ssh-add command:

eval `ssh-agent -s`
ssh-add

Note that this will start the agent for msysgit Bash on Windows. If you're using a different shell or operating system, you might need to use a variant of the command, such as those listed in the other answers.

See the following answers:

  1. ssh-add complains: Could not open a connection to your authentication agent
  2. Git push requires username and password (contains detailed instructions on how to use ssh-agent)
  3. How to run (git/ssh) authentication agent?.
  4. Could not open a connection to your authentication agent

To automatically start ssh-agent and allow a single instance to work in multiple console windows, see Start ssh-agent on login.

Why do we need to use eval instead of just ssh-agent?

To find out why, see Robin Green's answer.

Public vs Private Keys

Also, whenever I use ssh-add, I always add private keys to it. The file ~/.ssh/id_rsa.pub looks like a public key, I'm not sure if that will work. Do you have a ~/.ssh/id_rsa file? If you open it in a text editor, does it say it's a private key?