Website Down Monday

Hi Jim,

Our server clusters in both the US and the UK came under a sophisticated distributed denial of service (DDoS) attack late on Monday afternoon. The initial attack took both centres offline for an hour, before mitigation could be put in place by the network security team at our upstream provider (Rackspace).

Approximately an hour later, the attackers changed both their approach and the servers being targeted, resulting in a further outage of approximately one hour.

At that point, a new DDoS mitigation solution was put in place and, while the attack continued for many hours afterwards, the sites were fully available from then on.

The mitigation solution was technically “on” for 24 hours from that point, and then was set to “auto trigger” when it was clear that the attack had stopped.

While the mitigation solution was on, some legitimate TCP requests were being dropped and forced to retry. Had you visited the site during that period, which I presume you did, you would have seen that the site was fully responsive to end users, perhaps sometimes requiring the browser to automatically refresh to see pages.

This mitigation approach meant that most uptime checkers failed to properly detect the status of the sites, so they were incorrectly showing them as unavailable. They had particular trouble negotiating SSL connections. However, shoppers using regular browsers did not.

You can see from our own third party uptime checking service that there was a total of just over 2 hours of downtime: NitroSell Platform Uptime

Obviously, while we do make extensive efforts to protect the sites, it is impossible to fully protect against denial of service attacks. We did get the sites back up as quickly as we could, and we are prepared for these types of attacks in future.

Note that at no point was site security compromised, nor was any data breached; this was purely a denial of service attack.

Regards,
Donogh