Scalability tests, usually have the following generic goals:
- Ensure the system throughput can scale by adding new resources
- Determine the performance factor/multiplier (if it is unknown)
- Understand when (and ideally why) the system does not scale as expected
Say you have one VM running your software. You have already run a Performance Test and determined that the the maximum, stable, throughput of your software is 100 operations per second. Fortunately the software has clustering capabilities so you add a 2nd VM and successfully see 200 ops/s. When adding a 3rd VM instead of seeing 300 ops/s, you see 250 ops/s. After 250 you start seeing failed operations. Ooops. Something’s wrong since +1 VM leads to 2x the performance but +2 VMs to 2.5x instead of 3x. Time to investigate…
What can you do? You’ll have to figure out where the bottleneck is. It could be because of the hypervisor used or the underlying Guest OS, it could be due a hardware issue (faulty HDD/SSD, not enough memory). It could be due bad software design or a known legacy limitation – 32bit software instead of 64. It could be because of weird DB query – it takes too long to execute and once the operations grow to 250 this starts to show. It could be a memory leak that leads to some service being constantly restarted or say a cluster synchronization issue which forces some operations to be retired too many times. Or it could be something else entirely – say the library you use for whatever the software is doing is not optimized, you hit its limit, not any other.
The above example is quite simple. A more complicated one would include another goal – to determine if the service responsible to automatically scale/descale (assign/reassign) resources is working as expected. Imagine this is not a cluster solution but rather a “true” SaaS. You don’t add VMs, you add resources to some existing infrastructure. Maybe the problem is there? Or maybe things are even more complex – it is not only determining the throughput but also ensuring that the rest of services are working (if any). Say the software scales 1:1 with each new VM added but some other services stops working or lags behind. Not good.
Important things to remember:
- Be hyper aware what you are exactly testing about and what are the goals
- By mega aware about the whole architecture and functionality
- Keep in mind what matters more than something else (priorities)
- Ensure you are able to use the tools at your disposal, know their limitations
- Take notes
- Establish a baseline
- Always test using THE SAME FUCKING SETUP if you can’t – determine a new baseline
- Automate; little by little but do fucking automate