One way of keeping your test suites running fast is by organising them in a way that allows you to run the right tests at the right time. This might be running the faster, isolated tests to give you instant feedback in your TDD loop, or it might be running the most critical acceptance tests before you commit a changeset.
PHPUnit offers a number of ways to organise your test suite, but the docs are a little light on commentary, these are my thoughts.
Using the filesystem
The first and easiest way to get started is to use the filesystem. If you give
PHPUnit a directory as an argument, it will scan that directory (recursively)
for *Test.php files and then execute them. This means we can start to split
our test suite up, just by putting files in different places.
For the main Childcare.co.uk app's test suite, I have two top level PHPUnit directories.
The first is tests/unit and this is for isolated, fast, unit
tests. These tests tend not to interact with any external systems, quite often
not interacting with other classes, functions or components at all. They take
just a few milliseconds each to run and there are quite a lot of them.
The second directory is called tests/integration and this directory gets
everything else. I try not to get too caught up in naming the types of tests I
write, but depending on your way of thinking, this directory includes unit,
functional, integration, integrated, system and acceptance tests. The tests in here
exercise larger parts of the code and usually interact with third party systems
like databases or HTTP APIs. In a lot of cases, I use fakes to speed up these
tests,
but they're still slow, taking a couple of minutes to run the entire directory.
Under these directories, I tend to follow the file structure of the production
code as close as possible, though that quite often doesn't apply to the
acceptance tests that I run out of tests/integration, which tend to be
something like tests/integration/src/Childcare/<module>/NameOfFeature.php. If
I were starting green field today, I might have a separate top level directory
for some of those, but I don't lose sleep over it. Either way, having the tests
with a decent hierachy allows for flexibility in running just the tests for a
section/module/component as needed.
Failing fast
Given this simple split of faster tests and slower tests, I am able to run the
faster tests first, followed by the slow tests. In theory, this should mean I'm
getting the feedback I need quicker. To faciliate this, I use a simple
Makefile as a task runner.
.PHONY: check
check:
vendor/bin/phpunit tests/unit
vendor/bin/phpunit tests/integration
Using groups
Even given this separation, I still find I need some further organisation. Some
of those tests in tests/integration are really slow. I have a bunch of code
that interacts with the Facebook API and to make sure my code keeps in tune with
the API responses, I really want to have some tests that actually hit the API.
It turns out that creating test user accounts is quite slow and this slows my
tests down significantly, to the point where I start to get annoyed when I'm
running tests before a merge. Rather than creating another top level directory
to house the dozen or so tests that create Facebook users, I decided to use
PHPUnit's group feature to allow me to exclude these tests as I please.
/**
* @test
* @group facebook
*/
public function some_expensive_test() {}
Again, using our Makefile as a task runner, I make a recipe that runs the
tests without those facebook tests.
.PHONY: check check-quick
check-quick:
vendor/bin/phpunit tests/unit
vendor/bin/phpunit --stop-on-failure --exclude-group=facebook tests/integration
I have a few other groups excluded and I use this run regularly throughout the day,
only running the full suite occasionally. Our continuos integration
server always runs the full suite when I push code, so I can carry on developing
carefree with regards to our Facebook integration. Because I use this for rapid
feedback, I also use the --stop-on-failure switch. This can sometimes be
annoying if you've broken code in several places, but more times than not the
first failing test allows me to identify a smaller set of tests to run, make
fixes and run again.
Using @small, @medium and @large
One alternative to separating faster and slower tests in to directories is to
use the special @small, @medium and @large annotations. These are aliases
for @group small etc, but also have special meaning if you install the
phpunit/php-invoker package. With this package installed, if you run PHPUnit
with
--enforce-time-limit,
PHPUnit will mark these tests as risky if they do not execute within a set of
time thresholds.
Using test suites
Another way to organise the PHPUnit tests is to use test suites. This isn't something I currently utilise, but they are quite flexible and can mimic the behaviour I achieve using the Makefiles.
<phpunit bootstrap="src/autoload.php">
<testsuites>
<testsuite name="all">
<directory>tests/unit</directory>
<directory>tests/integration</directory>
</testsuite>
</testsuites>
</phpunit>
The first thing to note is that when using a test suite, PHPUnit will run the
tests in the order specified. This means with the example above, we get that
failing fast scenario as our faster tests/unit tests get run before the slow
tests/integration tests.
You can also get fairly specific about the files you want to include or even
exclude, so depending on the way you set your files and directories up, you can
achieve something like I did above with the facebook group.
<testsuite name="quick">
<directory>tests/unit</directory>
<directory>tests/integration</directory>
<exclude>tests/integration/facebook</exclude>
</testsuite>
I tend not to use test suites as I find the combination of directories and groups to be enough.
I plan to follow up on this post with a more detailed look at which tests to run at what time, drop your email in the box below to be notified when that one lands.