Building World Class Software at uptime

So you’ve launched your new enterprise web site, and you’re so confident with your fully redundant web architecture that you figure you can provide a five-nines SLA - 99.999% uptime, or 5 minutes downtime a year. After all, what can possibly go wrong?

Welcome to SLA inversion, where a 99.999% available web service depends on a 95% available DNS server. Or a 98% available Internet connection. If your users can’t do work, it doesn’t matter why - they don’t care that your web application is still able to process transactions; they don’t appreciate the difference between application failure and DNS failure.

As it happens, this topic is close to my heart: I run a few cancer support mailing lists, and one day I decided to switch the domain name hosting provider. What I didn’t realise was that the new registrar wouldn’t let me edit the zone file until the domain transfer had completed, and that left me dead in the water for several hours: requests to the web site were going to the registrar’s default page. No matter that the web site and mailing lists were just fine; the point was that nobody could reach them.

Whenever you’re monitoring for a service level agreement, you need to follow a few important rules.

1. Measure as close as you possibly can to what end users will see. That generally means running synthetic transactions at a minimum: HTTP for web applications and test emails for email services.
2. Take time to think through all the services your users rely on, and make sure they’re monitored, implicitly or explicitly. For example, are there routers outside your control on the path between you and your users? If so, find out what SLA, if any, is in effect. You can’t provide a better target than the weakest link in the chain.
3. If there is a weak link controlled by your customer, make sure it’s explicit in the SLA contract.

Service level agreements are all about avoiding nasty surprises: taking a bit of time up front to think through how you might be caught off guard will benefit you, and more importantly, your users.