We were pleased to help one such retailer with a series of tests that ran just days ahead of a big Black Friday promotion.
Bottlenecks in the first round of testing
The first tests (to 4,500 concurrent users) suggested that the website would most likely fall some way short.
The site was doing just fine until they hit around 2,000 concurrent users, at which point they saw large numbers of warnings (shown by the yellow line) and failures (shown by the black line). These included server errors from a critical component on the page, which meant that users were unable to complete transactions. There were also page download failures – 503 errors from requests to the root HTML document, meaning that users would have got no content at all. What’s more, since there was no custom error handling, users would have seen a generic browser error page.
These errors are represented by the yellow and black lines on the graph below (with the green line showing successful tests).
The pressure was on. Black Friday wasn’t moving, and the promotion was definitely going ahead. Failure was not an option. The retailer’s web team therefore worked hard to pinpoint the bottlenecks and implement fixes, delivering significant improvements in less than a week.
But would it be enough?
The only way to know for sure was to run the tests again.
A different picture
Fortunately, the results of the retests painted a completely different picture, and showed that the remedial work had been a complete success.
This time, less than one per cent of tests resulted in warnings, compared more than half of tests in the previous round. The more serious errors were down to just 0.33 per cent (from nearly 12 per cent).
Performance on Black Friday
Of course, the real test would be how the website coped during the actual promotion. Here, once again, the remedial work proved to have paid off. Monitoring for key pages and user journeys revealed very little variation in performance – and no outages.
Performance of a key user journey over Black Friday
A successful promotion
The Black Friday stories that grab the headlines tend to be the failures. The ones where retailers failed to test or dramatically underestimated visitor numbers.
However, for every failure, there are successes that often go unreported – retailers who plan, test, improve and retest. This team did exactly that, working day and night to deliver a website that didn’t hit the headlines – it just worked.