Running Successful Load Tests

No test or simulation can entirely eliminate the risk of a site failure. Even the most robust and scalable sites in the world, maintained by a billion dollar team of experts, run the risk of a black swan event (human error, acts of nature, or otherwise) bringing them down. So it’s never really about reducing the risk to absolute zero.

We can get pretty close, though, by load testing the environment with realistic traffic and fixing issues as they arise. And load testing doesn’t always have to take a Herculean effort, either.

If you get a few important things right in your load testing, it’s actually not that difficult to reduce the risk of a crash due to high traffic by 90% or even 99%.

The important things are…

#1: Realistically Modeling User Behavior

Do you understand how your users behave on the site?

Certain user actions can be a lot more time-consuming and expensive from your application’s point of view than others. As far as your site’s infrastructure is concerned, serving up static pages might be a breeze, but a shopping cart checkout flow would be more expensive because it requires maintaining user state, accessing a data store, communicating with a third party payment processor and fraud detection services, and so on.

A good load test should act out the primary user behaviors on your site as closely as possible, with an emphasis on the most risky and expensive ones. Create at least one test script for each common user behavior, and make sure to parameterize it when necessary, so that each user journey is unique. Parameterizing your script with unique user data is important. Unless your site is very simple, you’ll want to avoid the pitfall of just sending a flood of identical requests and calling it a load test.

#2: Understanding Your Scalability Requirements

The term requirements has a somewhat looser meaning when it comes to load testing than it does with functional testing. Because performance and scalability are multi-dimensional and multivariate, it can actually be quite tricky to state the requirements as pass-or-fail assertions.

You’ll probably start with rather simplistic requirements from your partners and stakeholders, if you’re lucky enough to even get that.

  • “The landing page should load in less than 1 second.”
  • “We need to process 7500 registrations per hour.”
  • “Response times should be less than 2 seconds.”
  • “We need to be able to handle 3X our previous peak load.”

These requirements all sound reasonable on the surface, right?

The problem with all of these requirements is they leave too many things open to interpretation. Some of them state the acceptable response time with no mention of load; others state the required throughput with no mention of acceptable response times. And not all of them describe what user behavior is being tested in the first place.

In fact, all of these requirements (as stated above) are likely to pass under some circumstances and not under others.

A more useful scalability requirement needs to specifically account for at least three things: the user behavior, the load on the site, and the acceptable response time.

With that in mind, let’s try rewriting the vague requirements above to be a bit more specific:

  • “The landing page should load in less than 1 second, when 5000 concurrent users are performing the 4-step Checkout flow.”
  • “The system needs to process 7500 registrations per hour, while maintaining an average response time of 1.5 seconds.”
  • “The 90th percentile response time should be less than 2 seconds, when 10,000 concurrent users are browsing the site on a variety of key pages.”
  • “We need to be able to handle 150,000 home page views per hour, with an average response time of 2 seconds or less.”

These requirements are more concrete because they now account for the dimensions of user behavior, load, and acceptable response time. It’s now easier to conclusively determine whether they passed or failed, and leaves much less open to assumption or misunderstanding.

#3: Testing in Your Real Environment

Needless to say, your site’s infrastructure and environment will have a big impact on its performance and ability to handle sustained levels of throughput.

Ideally, the environment you test should be running the same software and hardware (virtual or physical) as your production site. It can even be your production environment – as long as you’re aware of the risk that your load test might crash it.

If you’re unable or unwilling to load test your actual environment, it’s crucial to test one that’s as similar as possible. Attempting to extrapolate load test results to a different environment than the one that is tested is risky, and should be avoided! Scalability of complex systems is rarely linear.

#4: Correctly Interpreting Results

The outcome of a load test is impacted by many inputs and assumptions. When you share test results with your colleagues, we recommend calling out these assumptions as possible caveats, because they affect the applicability of your test results.

Pro Tip: As a tester, documenting all of the assumptions that went into your test is not only responsible, it’s also a good way to cover your ass!

Examples of inputs and assumptions that you should call out in your report include:

  • The specific user behaviors that you simulated in your test scripts
  • The servers, configuration, and network infrastructure that you tested
  • The specific version of your site or application that you tested
  • The amount of load both in terms of simulated users (bots) and throughput (transactions/sec, bytes/sec)

Remember, the outcome of your test is only as meaningful as the assumptions that went into it!

A meaningful load test is one where you can state, with a high degree of confidence, that given the assumptions listed above, the site passed or failed the requirements.

5: Choosing The Right Tool

There are many tools that can generate load against a server. Besides Loadster, people have run successful load tests with homegrown solutions, open source libraries and tools, and other commercial tools.

Depending on your constraints (time vs. money, simplicity vs. flexibility, etc.) you might choose a different tool or even a combination of tools. Up until now, this guide has mostly been tool-agnostic. From now on we’ll mostly focus on the specifics of load testing with Loadster.

Loadster is a strong choice when you have some budget for tools, and you don’t have unlimited time to deploy and configure your own test infrastructure, wrangle raw test result data, and draft reports manually. To borrow an old phrase, Loadster’s aim is to “make the easy things easy and the hard things possible”.

Let’s get started with Loadster!