act:ualise | technology

I’ve done a lot of performance analysis and improvement in my time although it’s only in my last 3 projects that we started automating it.

In all cases a developer check-in would trigger the execution of a battery of stress/concurrency tests against a fully integrated as-live environment. Soak tests were scheduled nightly or over weekends against the same hardware. As much as possible we automated everything: deployment, db schema initialisation, data seeding, test execution and report generation.

Project Zero

We chose JMeter to build a series of tests which exercised selective system end-points, the controller’s actions surfaced as URLs. We used distributed JMeter tests to simulate huge numbers of concurrent clients from several machines. This was automated using Maven/Ant build scripts, with graphs published for each successful build charting metrics such as throughput vs response time.

• It was effort intensive : XML result translation into graphs was bespoke, the hardware infrastructure was also fairly heavyweight.
• We pushed JMeter to the brink – its implementation has finite limits with high levels of concurrency.
• Developers would write performance tests for qualified endpoints in parallel with code, unit and integration tests.
• CruiseControl triggered build failure noises where performance fell below some acceptable measure, but too many false negatives forced us to disable it.
• We weren’t testing the whole page, just the controller endpoints. Not great, but at least we tested through the whole stack.

With this approach we found and fixed several serious bottlenecks and other performance issues. However after the big wins the development effort moved on, and performance and load testing aspects went stale and unmaintained.

Our first Grails app

The next time we were commissioned into performance tuning we used JMeter much less. We tried out JChav which generated interesting charts but was insufficient by itself (and not maintained since 2006). We invested relatively less time into the process and had to timebox the overall effort to ensure the launch deadline.

• Developers were unable to implement performance tests in parallel with code due to delays delivering staging and production environments. Consequently all performance testing occurred several months into development, pre-launch.
• It was still costly to pull this setup together since the infrastructure assembly was effort intensive.
• We automated using Hudson though we didn’t trigger performance threshold fails.
• We were testing the whole page, not just end-points, generating more realistic system load.

Once again we identified and corrected numerous key performance problems over the following few weeks, then after launch the performance and load testing aspects grew stale from ever diminishing attention.

Grails application #2

The performance testing for the latest Grails project was developed more organically out of unit tests into a larger thing. This effort is ongoing at the time of writing.

• Performance tests developed in the course of development.
• HttpUnit used in development of a Grails plugin to spawn concurrent HttpClients for loading.
• All requests were for a complete page with all it’s assets.
• A model was developed to describe and replay anticipated usage scenarios using the above.
• We automated using Hudson.
• We instrumented runtime performance measurement using JAMON.
• Effort intensive: the plugin development, XML translation and visualisation combining JAMON and plugin result datasets.