Two weeks ago, me, my two partners in crime Aaron Nemoyten and Jessica Mah launched our first official Facebook application: Serial Killer. Suffice it to say that this application is the answer and ultimate de-facto shove in the face to all the other poke apps on Facebook. Before you get all offended by its rather innocuous content, please read the TOS.

Now onto the interesting stuff — Siqi Chen recently wrote a brilliant series of articles about Facebook virality, what works, and what doesn’t on the Facebook platform. In it he made it clear that certain apps, like the one we built, are currently successful by virtue of their inherent viral features. This is certainly true of Serial Killer. Currently the app is growing linearly, at the rate of ~800 users (considering a linear growth rate, this is damn good). Part of the reason the growth is linear is by design — we want to be ready for the hockey stick — and are currently happy with a straight and predictable line. But, even at ~800 daily new users, this presents some interesting design and scaling challenges. I had to optimize things three times already. Although I won’t get into the details of this now, suffice it to say it took quite a bit of work, patience and adapting to the 50k daily page views we are serving after 2 weeks in operation.

But what if I were to tell you that today we ‘accidentally’ doubled our page views from 50k to shy of 100k (the last few didn’t quite make it in before the clock struck midnight). Here’s why all this happened today… When Aaron and I got home at around 5pm today from a calm shopping trip, I found myself staring in awe at my screen. I was watching the 1-minute load average in ‘top’ go from .3 to 1, to 2, to 3, to 4, to 6, to 8, and eventually to 23 at its peak, all in a matter of about 2 minutes. Sadly I knew what was happening. We were being hit with a DOS (Denial of Service) attack. It turned out to be a DDOS. Three or four facebook users (who are now known as the ‘asshat brigade’), all from fa-away-parts like Taiwan, Singapore, and China, hit us with a bot-generated DDOS. I scrambled while analyzing the tail of our web server log and looking at top. Bottom line, I resolved the hole we had opened in our code in about 15 minutes, and we were up and running again (under a slight lockdown). The offenders were banned from our app.

The lesson to learn from all this: even if your Facebook application is optimized and running smoothly, and you are ready for 800 new users at 50k page views per day, you never know what’s around the door. Set up those alerts on your server, and make sure to be ready to dig up your logs. When your back is sweating and you’re frantically trying to lock access while minimizing the damage to your existing data, you know what it’s like to be coding for the facebook platform =).

Alex is currently digesting a good steak, which he cooked in celebration of a successful elimination of a DDOS Attack to his Facebook Application.