Running Successful Load Tests
A load test is only as good as the assumptions you make going into it!
While no test can entirely eliminate the risk of a site crashing, you can greatly reduce the risk if you get these basic elements right.
The important things are…
Important Thing #1: Realistically Modeling User Behavior
Do you understand how your real users behave on the site?
Certain user actions can be a lot more expensive than others. As far as your server is concerned, serving up static pages might be a breeze, but a shopping cart checkout flow would be more expensive because it requires accessing some kind of datastore, communicating with a third party payment processor, and so on.
Your test should imitate the primary user behaviors on your site as closely as reasonably possible… especially the most expensive ones.
Important Thing #2: Understanding Your Scalability Requirements
The term requirements has a somewhat looser meaning when it comes to load testing than it does with functional testing. Because performance and scalability are multi-dimensional, it can be a bit tricky to state the requirements as binary pass/fail assertions.
We usually start with rather simplistic requirements from our partners and stakeholders, for example:
- “The average page should load in less than 1 second.”
- “The system needs to process 7500 registrations per hour.”
- “The vast majority of response times should be less than 2 seconds.”
- “We need to be able to handle triple our previous peak load, while still remaining responsive to end users.”
The problem with all of these requirements is they leave too many things open to interpretation. Some of them state the acceptable response time with no mention of load; others state the throughput with no mention of acceptable response times.
A useful scalability requirement needs to specifically account for both the load on your site and the acceptable response time.
Let’s try rewriting the vague requirements above to be a bit more specific:
- “The average page should load in less than 1 second, when 5000 concurrent users are performing the 4-step Checkout flow.”
- “The system needs to process 7500 registrations per hour, while maintaining an average response time of 1.5 seconds.”
- “The 90th percentile response time should be less than 2 seconds, when 10,000 concurrent users are browsing the site on a variety of key pages.”
- “We need to be able to handle 150,000 page views per hour, with an average response time of 2 seconds or less.”
These requirements are better because they now account for both the load and the acceptable response time.
Important Thing #3: Testing in Your Real Environment
Needless to say, your site’s infrastructure will have a big impact on its performance and ability to handle sustained levels of throughput.
Ideally, the environment you test should be running the same software and hardware (virtual or physical) as your production site. It can even be your production environment – as long as you’re aware of the risk and all the other stakeholders are too.
If you’re not able or willing to load test your actual environment, it’s crucial to test one that’s as similar as possible.
Attempting to extrapolate load test results to a different environment than the one that is tested is risky, and should be avoided if possible! Scalability of complex systems is rarely linear.
Important Thing #4: Correctly Interpreting Results
A load test, by its very nature, is influenced by many variables. When you share test results with your colleagues and stakeholders, we recommend calling out these variables as possible caveats, because they affect the applicability of your test results.
Pro Tip: As a tester, documenting all the assumptions that went into your test is also a good way to cover your ass!
Important variables/assumptions that you should call out in your report include:
- The user behaviors that you simulated in the test
- The site and network infrastructure that you tested
- The specific version of the site or application that you tested
- The amount of load both in terms of simulated users (v-users) and throughput
Remember, the validity of your test is only as good as the assumptions that went into it!
A meaningful load test is one where you can state, with a high degree of confidence, that given the assumptions listed above, the site did (or did not) meet the requirements.
Loadster Makes Things Easier
Load testing complex systems is no walk in the park, but Loadster helps you operate at a higher level by focusing on what really matters in your testing, instead of the mundane tasks.
If you’ve done load testing in the past, you’ll be pleasantly relieved to know that…
- You won’t have to write your scripts from the ground up in some low-level programming language
- You won’t have to build complex cloud orchestration tooling to spin up and manage your test instances
- You won’t have to sift through piles of raw data just to make a few paltry charts and reports
By doing all this for you, Loadster makes your job easier, and you can focus on getting actual meaningful results in as little time as possible.