Prior to C# 5, you need to re-declare a variable inside the foreach - otherwise it is shared, and all your handlers will use the last string:

foreach (string list in lists)
{
    string tmp = list;
    Button btn = new Button();
    btn.Click += new EventHandler(delegate { MessageBox.Show(tmp); });
}

Significantly, note that from C# 5 onwards, this has changed, and specifically in the case offoreach, you do not need to do this any more: the code in the question would work as expected.

To show this not working without this change, consider the following:

string[] names = { "Fred", "Barney", "Betty", "Wilma" };
using (Form form = new Form())
{
    foreach (string name in names)
    {
        Button btn = new Button();
        btn.Text = name;
        btn.Click += delegate
        {
            MessageBox.Show(form, name);
        };
        btn.Dock = DockStyle.Top;
        form.Controls.Add(btn);
    }
    Application.Run(form);
}

Run the above prior to C# 5 , and although each button shows a different name, clicking the buttons shows "Wilma" four times.

This is because the language spec (ECMA 334 v4, 15.8.4) (before C# 5) defines:

foreach (V v in x) embedded-statement is then expanded to:

{
    E e = ((C)(x)).GetEnumerator();
    try {
        V v;
         while (e.MoveNext()) {
            v = (V)(T)e.Current;
             embedded-statement
        }
    }
    finally {
        … // Dispose e
    }
}

Note that the variable v (which is your list) is declared outside of the loop. So by the rules of captured variables, all iterations of the list will share the captured variable holder.

From C# 5 onwards, this is changed: the iteration variable (v) is scoped inside the loop. I don't have a specification reference, but it basically becomes:

{
    E e = ((C)(x)).GetEnumerator();
    try {
        while (e.MoveNext()) {
            V v = (V)(T)e.Current;
            embedded-statement
        }
    }
    finally {
        … // Dispose e
    }
}

Re unsubscribing; if you actively want to unsubscribe an anonymous handler, the trick is to capture the handler itself:

EventHandler foo = delegate {...code...};
obj.SomeEvent += foo;
...
obj.SomeEvent -= foo;

Likewise, if you want a once-only event-handler (such as Load etc):

EventHandler bar = null; // necessary for "definite assignment"
bar = delegate {
  // ... code
  obj.SomeEvent -= bar;
};
obj.SomeEvent += bar;

This is now self-unsubscribing ;-p