Zend Framework app's PHPUnit suite 3x faster
The background
My regular readers may have already noticed I don’t even dare writing code without TDD/BDD-ing it from left to right. Unfortunately, my current job is based on Zend Framework. I say “unfortunately”, as (amongst other drawbacks) ZF’s architecture is far from professional in terms of testability.
We currently have 600+ PHPUnit test cases, most of them for models and controllers, but quite some also for view helpers, bootstrap and even some views. The whole suite is taking around 90 seconds to run on my machine against a MySQL database (modelled with Zend_Db* in an active record/table data gateway manner).
In this post I’ll be showing how I managed to decrease the time from ~90 seconds to ~30 seconds.
The problem
As I’ve mentioned before, Zend Framework isn’t really encouraging developers to do TDD (on the other hand, this shouldn’t be a surprise in the PHP community). The application architecture is based on singletons all around (front controller, session, layout, helper broker, registry, etc.), the bootstrap doesn’t “separate the cacheable from the non-cacheable”, there’s no built-in support for dependency injection, the ORM is cumbersome to stub out.
Our approach used to involve calling Our_Db_Table_Abstract::truncate() on each table used by the given test case. Not only was this inelegant, mundane to maintain and easy to forget, but as it later turned out, it was the biggest performance bottleneck. We’d either put it at the end of the test case – which would not be executed if the assertion failed and we’d end up with a dirty db – or we’d put it in tearDown() – which meant it ran even if a particular test didn’t use the given table.
Another problem was controller tests. We needed to explicitly declare, which tables would be used by the test. Even with extra-simple syntax, it was still terribly inelegant. And if we’d forget about one table…
This is how it would look like in model tests:
And in the controller:
As you can see, there’s a lot of verbosity, duplication and noise in the code. We need to do plenty of “plumbing” for the tasks that should be automatic.
The solution
The solution I came up with was inspired by Rails, and other frameworks. It can be summarized as: wrap each test case in a transaction, rolling it back after the test case.
I implemented the solution using a PHPUnit listener, hooking up to startTest() and endTest() callbacks. Here’s the appropriate code fragment:
These callbacks wrap a single test case, so the cycle is:
startTest()setUp()test*()tearDown()endTest()
And here’s the appropriate fragment of phpunit.xml (we’re using PHPUnit 3.5):
Excellent, that’s an elegant solution with a properly separated concern. I got rid of all the ugly truncate()s and sped up the tests significantly.
Caveats
Of course, life’s not that full of sunshine and bunnies.
Problem #1
First problem I encountered, was bootstrapping the application inside the cycle mentioned above (anywhere within steps 2-4). This happens in controller, view, view helper and bootstrap test cases. Let’s go through a controller test case. We:
startTest(), which begins a transactionsetUp(), which bootstraps the application, which in turn puts a new ‘database’ object inZend_Registry- …
- …
endTest()callsrollBack(), which explodes withNo active transaction
This basically means that we’re running beginTransaction() and rollBack() on a different database connection.
One solution for this problem is to use a persistent PDO connection. This will most likely cause both methods to run on two different PHP objects, but same connection resource. To do this, just add the following setting to your database config:
params.persistent = true
I wanted something more robust, though, so I rollBack() manually right before bootstrapping and beginTransaction() immediately afterwards. It’s plumbing again, but at least it’s abstracted in a base class. Here’s an example:
Problem #2
Second caveat was quite obvious. Some of our tables were using MyISAM (which doesn’t support transactions), not InnoDB. I solved it by creating an appropriate migration, like so:
ALTER TABLE Country ENGINE = InnoDB;
Problem #3
Some tests were still failing after all these changes. The problem was that they were assuming primary key values for newly created rows, i.e. expected a new row to have an ID of 1. This was true when we were truncating, but is no longer true for rollbacks, since autoincrement counters are not reset. I refactored it in a following way:
Problem #4
Not really a problem for me, but I should probably mention this. We could observe the performance boost on all developers' machines with Ubuntu. Some of our colleagues are using Windows, and they had a very small gain in speed (around 5 seconds). But their benchmarks are generally tilted, since sometimes the suite would finish under 2 minutes, and sometimes after 5+ minutes. YMMV.
Conclusion
There you go, with a conceptually simple and elegant solution you can achieve quite a dramatic improvement in performance of PHPUnit suite for a Zend Framework application. My next idea is to cache the bootstrap input (i.e. routing). What’s yours?
And here’s the evidence for sceptics:
Before:
$ phpunit PHPUnit 3.5.0 by Sebastian Bergmann. ............................................................ 60 / 630 ............................................................ 120 / 630 ..............S.SSSS............................S........SS. 180 / 630 S........................................................... 240 / 630 ..................................S.S....................... 300 / 630 ............................................................ 360 / 630 .........................SS.......F.S...........S........... 420 / 630 .....................S...................................... 480 / 630 ..........................I................................. 540 / 630 ..............................................S............. 600 / 630 .............................. Time: 01:24, Memory: 136.75Mb FAILURES! Tests: 630, Assertions: 830, Failures: 1, Incomplete: 1, Skipped: 17.
After:
$ phpunit PHPUnit 3.5.0 by Sebastian Bergmann. ............................................................ 60 / 630 ............................................................ 120 / 630 ..............S.SSSS............................S........SS. 180 / 630 S........................................................... 240 / 630 ..................................S.S....................... 300 / 630 ............................................................ 360 / 630 .........................SS.......F.S...........S........... 420 / 630 .....................S...................................... 480 / 630 ..........................I................................. 540 / 630 ..............................................S............. 600 / 630 .............................. Time: 29 seconds, Memory: 131.50Mb FAILURES! Tests: 630, Assertions: 830, Failures: 1, Incomplete: 1, Skipped: 17.
