Building World Class Software at uptime

I was talking to someone recently about how to improve the quality of his software. He’s reached the point where he can see there is a problem, but he doesn’t yet know how to solve it.

Here at uptime, we have a beautifully disciplined software development process that does everything from test-driven development to continuous integration and fast feedback using Cruise Control. Our code is mostly well factored, reliable, and easy to work with, mostly because we haven’t skimped on process. The temptation to take shortcuts in the face of looming deadlines is always there, but we also know that doing that would lead to long term pain.

As I talked to my friend, I realised that you don’t have to do everything at once; in fact, it’s better to do a bit at a time. So here’s my N-step process for getting your software under control.

Step 1: Make a code-level regression test suite. Your software keeps breaking in weird ways, and you can’t possibly predict everything. But you can certainly do something. For example, if you are building a web-based application, you could check that the front page of the app has a login form on it. And once you’ve done that, you could check that a login works OK and the username is displayed on the resulting page. These tests could be as simple as using grepping the output of wget, or you could use a more sophisticated tool like Selenium. Either way, you have something that can tell you if something is badly broken. And what if you added one test a day to the suite? After three months, you’d be testing for sixty possible failure conditions every time you ran the suite.

Step 2: Automate the build process. You can almost certainly find a way to script a software build, database reset and regression test run automatically. Maybe you have to commandeer a physical machine; maybe you can set up a vmware instance. The important thing is that it can run unattended. This is called a Smoke Test because you are checking to see if anything “catches fire” when you run it.

If you’re using a language like PHP that doesn’t require compilation, you will probably want to at least do a syntax check against all your PHP files. On Unix, PHP comes with a command-line program and you can run “php -l”. Other languages generally have similar syntax for doing a syntax check on a script without running it (Perl, Python and Ruby all use the -c flag).

What you’ll probably find once you’ve done this is that (a) the suite fails more often than you’d expect, (b) there are random bugs that keep slipping through (but you can add tests for at least some of them), and (c) your team often forgets to run the tests before checking in code. Which leads to...

Step 3: Run the tests automatically. In the old days, we used to run build scripts from cron once a day, and then look at the output in the morning. Nowadays, we have continuous integration (CI) tools such as Cruise Control that wake up every few minutes, check CVS for new checkins, run a build and unit+regression test, and notify the developers that broke the build.

CI tools have a reputation for being hard to set up, but mostly because it’s hard to get a repeatable build process. If you’ve followed steps 1 and 2 already, you’ll have that working and the rest of the setup will be a breeze.

Other advice: Don’t try to do all this at once. Build a couple of tests, and get used to running them. Then start automating parts of the build and run it every day until you’re comfortable. Only once you can run a single command to build and test should you take the final step to running Cruise Control.

Labels: agile, tdd