I finally figured it out working on another problem. The problem was that my test couldn't find an import.
It looks like you get the above error if your test fails to import. This makes sense because the test suite can't import a broken test. At least I think this is what is going on because I fixed the import within my test file and sure enough it started working.
To validate your test case just try import the test case file in python console.
from project.apps.app1.tests import *