Weighing the Costs of Load Testing

When Does Load Testing Make Sense?

Load testing often makes sense, but not always.

If you want to make sure that whatever testing approach you take generates meaningful results and a positive return on investment, here are a few aspects to consider.

The Costs of Site Failure

How much is reducing the risk of a site failure actually worth to you? In other words, what would be the cost of lost business (plus embarrassment and brand damage) that could potentially result if your site crashes during a high traffic event? If your site is valuable (or you expect it to be in the future) then the cost of a load-related failure is substantial – especially since this sort of failure by its very definition always happens at the worst possible time!

Take a moment and estimate a number (in actual money, ideally) for how much the risk of a site failure under load is actually worth to you. Knowing or estimating this number is critical when you’re sizing up a load testing effort and weighing the associated costs and benefits.

Furthermore, how extensively are you planning to test? Obviously, the more you test, the higher degree of confidence you will end up with – but there is a point of diminishing returns.

Generally speaking, an extensive load and stress testing effort may reduce the risk of traffic-related site failure by 80-95%, but even a quick volley of load tests can reduce your risk by 30-50%.

The Time Cost of Load Testing

Load testing takes time. You’ll need to formulate performance and scalability requirements, and someone will need to create test scripts and run the tests. It’s an iterative process that can sometimes take a few tries to get a successful test, and then more tests to evaluate the impact of performance tuning or other changes to your site.

Some of the more minimalist testing tools require additional layers of drudgery on top of that – aggregating data from disparate sources, creating charts and reports, and deploying and maintaining infrastructure.

Is it worth investing in a tool that gives you fully managed test infrastructure, automatic reporting, and easier scripting? Shortening the test-to-test feedback loop is the key to successful load testing, but it doesn’t come for free.

Deciding If Load Testing Is Worth It

Some load testing tools are free (as in beer and sometimes as in speech). Others require a subscription fee or upfront license cost. When you’re evaluating tools, remember to consider the infrastructure costs as well! If you end up running a free tool from 87 EC2 instances that you have to separately pay for, deploy, and maintain yourself, that’s definitely not free. If you aren’t careful, the “total cost of ownership” of a free tool can sometimes be surprisingly high.

A Few Thoughts

As a load testing vendor, it’s tempting to tell everyone they should load test extensively and incessantly. Of course, unless the sites and applications you’re testing are incredibly valuable and lucrative, that may not be the right answer.

Weigh the probability and severity of a load-related failure with the total cost of the investment you make to prevent it.