DEBRIS.COMgood for a laugh, or possibly an aneurysm

One of the things I do to keep the lights on is help run a busy consumer website. The site sees awesome, unreal spikes in traffic around the holidays. (Hint: it ain’t debris.com.) Traffic doubled on the night before Valentine’s Day, from our normal daily peak of about 4 Mb/sec to over 9 Mb/sec, causing some strain on our server network — the webservers would back up under the deluge of pending connections, and finally succumb to the load and die. If we caught them in time, they’d recover; if not, we’d have to send someone to the colo facility and kickstart them. We did some of each as we learned what the limits are. This hardware had never been pushed so hard.

Traffic declined after 11pm, but began to climb again early on Valentine’s Day. By 6am most of my team were glued to their seats, watching in disbelief as the MRTG graph showed our traffic climb past 15 Mb/sec, past 20 Mb/sec, to finally peak north of 27 Mb/sec. Imagine that: the site’s traffic had grown 700%.

We brought additional hardware online to help handle the torrent, but we still had occasional problems with webserver suicides. I drove the load-balancer for much of the morning, manually pulling a machine out of the server array when I saw it approaching redline. Of course, pulling a server offline causes the load on the rest to climb, in compensation for the lost machine, so this became a sort of dance, attempting to mete the load in the face of accelerating growth in page requests.

But we pulled it off. It was exhilarating. I commented at one point that I felt, while manually load-balancing traffic for multiple websites across nine servers of vastly different capacities, like I was conducting a symphony. A co-worker responded wryly that “herding cats” was a more accurate description.