kubectl config unset takes a dot-delimited path. You can delete cluster/context/user entries by name. E.g.

kubectl config unset users.gke_project_zone_name

kubectl config unset contexts.aws_cluster1-kubernetes

kubectl config unset clusters.foobar-baz

Side note, if you teardown your cluster using cluster/kube-down.sh (or gcloud if you use Container Engine), it will delete the associated kubeconfig entries. There is also a planned kubectl config rework for a future release to make the commands more intuitive/usable/consistent.