From the man file:

When --cached is given, the staged content has to match either the tip of the branch or the file on disk, allowing the file to be removed from just the index.

So, for a single file:

git rm --cached mylogfile.log

and for a single directory:

git rm --cached -r mydirectory