act:ualise | technology

In my previous article on performance testing, I suggested that incremental and iterative development of performance unit tests was an essential part of your overall performance testing strategy. However, performance testing the parts of a system is vastly different from testing the whole and will require some careful thinking to avoid too much focus on the small things. Since we’re not in the habit of doing this, how to begin grappling with this?

Side-door tests

I had never heard of side-door tests until I watched a presentation given by Jonathan Kohl @ Agile Vancouver 2008. His extremely hands-on approach to his QA test role fascinated me. He reviewed developer’s code frequently and aggressively; good stuff. He wrote code to test the developers code; now that was quite novel indeed.

He spoke of how he was stuck trying to recreate a problem; a query in the database was causing a locking issue when under load. His testing record/playback tool was itself not powerful enough to generate comparable loads on the test server; even if it could his test server was a poor mans version of live, nowhere near as performing. Enter the side-door test:

Employing a test to generate concurrent load on the service method executing the suspect query reproduced the locking problem. The test was left in place after to verify the problem was fixed. This seemed like a perfectly sensible way to fix problems, however it recently occurred to me that this could serve as a viable approach to implement a unit performance testing suite.

Some examples might be:

• Determine total logins per second for LoginService
• Determine total new registrations per second for RegistrationService
• Determine max concurrent trades per second for TradeService
• Determine total contending logins and registrations in 10 seconds for LoginController and RegistrationController

Developers can write these tests as either unit/integration tests in the traditional sense executed locally, or additionally as tests executed within the VM of a deployed application dedicated to performance testing.

Using focused tests a developer can more methodically ‘tune-backwards‘, identifying Tier and Technical Wait-points for saturation testing and the subsequent tuning to be made.

Early visibility and growing value

In the same way that the network effect of a growing suite of unit and integration tests leads to increasing confidence in functionality, an ever expanding suite of unit performance tests increases confidence in the performance of code.

At the inception of the project unit performance tests merely record timings on each new check-in: a history of select performance characteristics (saturation point, max concurrency, throughput, even contention) per functional area is collected. Then at some point in time the test may transition from recording to triggering a failure where a performance threshold is exceeded – precisely when this transition should occur is hard to say. It might make sense to do so when the record/playback version of a test which exercises the same code is automated.

Whatever the case, setting performance thresholds is not the end-game of unit performance testing. What’s important is ownership by developers, product owner and QAs to qualify the performance criteria which are to be met and understanding the required instrumentation for collecting the right breadth and depth of data vital for later analyses. Investing development teams in performance testing is key to ensuring that performance test automation is not discontinued.

Another effect ought to be that effort-intensive, record/playback test automation can be keenly focused on the highest value aspects of a system – the preceding process of identifying performance criteria during unit performance testing in fact provides a roadmap which can be followed to this end.