Performance tests are a group of tests and a Performance test is part of that group. Weird, right? Anyway…
- Define what ‘performance’ means to you and your organization. Be very punctual about it, ensure that everybody understands it. It is open to (moronic) interpretations. Performance testing includes several different types of tests:
- Benchmark / Stress
- Performance
- Load (a.k.a Endurance)
- Define what you will be measuring, under what conditions and why you’ll be measuring it. This part may come from the requirements or it may not. Be very punctual about it. That definition is something you should stick to throughout the tests (and preferably as long as it makes sense to).
- What to measure? Examples: Download/Upload speed, messages/s , file uploads/s, issuance/s, queries/s, operations / s , response time
- Under what conditions? This is where you must define the environment and the setup. The more complex a product/service is, the more moving parts there are. Decide what the setup should be given the specifics of your context and goals.
- Establish a baseline
- Define the hardware or whatever resources. Doesn’t matter if the product/service is on-perm or in the “cloud”. There’s no running away from the hardware so you better write down what resources are available at the moment of the test. This could be pretty tricky or straightforward. Depends on the product/service design, architecture and complexity.
- Define what parameters you wish to monitor – CPU, memory, I/O usage, bandwidth & etc. This may or may not be relevant and it depends on your specific context and goals.
- Know your tools. By far something that is missed the most. People have tendency not to research the tools they are using, not knowing how they work, what they require & etc. Tools have bugs too – hence you need to know what your tool can do, on what configuration, when, how. Everything there is to know about the tool you must know. Else you risk having weird results and then waste away figuring out what went wrong.
Performance test
The primary goal of a performance test is to establish the maximum stable throughput (baseline) of a system for a specified amount of time. It is also a prerequisite for a Scalability test. The “art” is to figure out WTF that means and how it should be done. A common mistake is to run performance tests in “isolation”, meaning one decides to disable certain features of the system while running the test. That’s very bad practice unless it is the precise goal – to know how the system performance if it is half-functional. Another common mistake is running a Performance test before running Benchmark / Stress one.
A Performance test is meant to check how many successful operations per second (ops/s) a given system can do for a given amount of time. Those operations should be of the same kind – file uploads, emails sent, messages sent (per second for 2 days or 2000 or 2 gazillion). The payload (files/emails/messages) must have an established profile or profiles. That’s another common mistake – not having common datasets. Do have common datasets so you are able to carry a performance test in a reproducible manner (same shit, same data, same time, same hardware/resources).
The end result of a Performance test must be some throughput/s for a given amount of time. Pay big fucking attention to all perf tests you run. Stressing again – the Performance test establishes a baseline for the service/product which then is used in other tests. Fucking up here is not something you wish to do.
Benchmark / Stress test
That’s a private/specific case of a Performance test. If a Performance test is meant to run for 24h (an arbitrary number) then a Benchmark one is meant to run for orders of magnitude shorter. 2h, 30 minutes. Figure it out for your context.
The goal of any Benchmark / Stress test is to determine the absolute/utterly maximum stable throughput of a system, for a given (short period of) time. This test is to be run before any Performance tests.
You want to know how many successful ops/s your software can do for 30 minutes. You setup the environment, load it up with 300 ops/s for 30 minutes while monitoring when the first errors (instability) would appear. If it appears in 20 minutes, then you lower the load, say move to 200 ops/s and do the same. The goal is to find out what is the # of ops/s for a given amount of time without any errors. It’s a good practice to alter the runtime as well. If no errors for 30 minutes for 300 ops/s, then increase the ops/s. Let’s assume that at the end you’ve established the system can do 300 ops/s for 30 minutes w/o failures. Now it is time to figure out how many it can do for 1h, 2h, 12h (again arbitrary numbers). Then when you discover it can do 200 ops/s for 12h, use that as a baseline for the Perf test (or don’t, it all depends on the context).
Load / Endurance, a.k.a Reliability test
Load test is something that runs for a long time. Here the goal is not to establish the maximum throughput of a system but rather the “expected/ordinary” one. Say a Performance test showed the system can successfully do 200 ops/s for 24h. Then you’ll run your Load test with 100 ops/s for at least a week (or whatever makes sense for you). The system must endure the load and provide for 100 ops/s for 1 week without any errors. The latter implies log monitoring at all times.
Load tests are usually are complex ones (depending on the context). Say you have a software that has a cron job running every 6 hours. Or 100 cron jobs, or whatever. They must run successfully at all times during the test. Say your software allows users to use it through a mobile app, browser, client software or whatever. They must be able to do so, successfully, for the time of the test. All that means that while your whatever is processing some requests, it must also process everything else it needs to.