Variable set analysis in load testing

If you use variable sets in load tests, we’ve just released a new reporting tool that we think you’ll find extremely useful.

A bit of background

Just in case you didn’t know, we often need to use large sets of data in load tests. They might be login credentials, search terms or product lists. This is required when user journeys involve making a purchase, placing a booking or logging into a password-protected area.

Sometimes, these credentials fail for some reason. It’s not normally intended (or expected). But it happens. And sometimes, these failures hint at something interesting. For example, perhaps products in a certain range can’t be added to basket because they all sit in the same database, and there’s a problem with that database. Or perhaps input validation on a login page is incorrectly rejecting email addresses that follow a certain format.

The needle in the haystack

It’s always been possible to pick this sort of thing up, but it’s been a bit of a laborious process before now. It’s meant trawling through test results one by one, looking for common factors. Patterns might then begin to emerge – for example, maybe a disproportionate number of failed tests involved products in a certain category. However, what we haven’t been able to do is aggregate results based on data sets. Until now.

The new reporting tool will make it much easier to understand what’s behind failed or slow tests. For example, you’ll now have much greater insight into which set of authentication details pass or fail, or which products are unavailable. This in turn will make it far easier to pinpoint the source of any problems and follow up with tests that focus on the affected data sets.

Possible applications

This is a relatively simple feature, but it opens up quite a range of possibilities. For example, you might want to verify a large number of dummy log-in details before using them in a load test, so that incorrect credentials don’t affect the results of that test. Alternatively, you could verify that every product in a database can be added to a shopping basket or shows up in search results.

Would you like to learn more?

If you’re not already an NCC Group customer, please feel free to get in touch, so we can discuss your requirements. If you’re already using our load testing portal on a SaaS basis, you might need a bit of extra training on the new reporting tool – just let your account manager know, and we’ll arrange something for you. For managed load testing customers, it’s simply a matter of understanding what the service can now do. Again, please contact your account manager if you’d like to talk through the possibilities.

Leave a Reply

Your email address will not be published. Required fields are marked *