Pete Bevin: Live From the Software Factory

Santizing HTML with regular expressions

noreply@blogger.com (Pete Bevin) — Sat, 28 Jun 2008 13:14:00 +0000

Jeff Atwood argues for the sanity of this regular expression:

var whitelist = @"</?p>|<br\s?/?>|</?b>|</?strong>|</?i>|</?em>| </?s>|</?strike>|</?blockquote>|</?sub>|</?super>| </?h(1|2|3)>|</?pre>|<hr\s?/?>|</?code>|</?ul>| </?ol>|</?li>|</a>|<a[^>]+>|<img[^>]+/?>";

This is perfectly good "dumb code"; as long as there are unit tests for the containing method that make it clear what the intent is, I'd accept this as production code. It's unlikely that the list of allowable tags will change much over time, so maintenance of the regex isn't really an issue.

That said, if the whitelist is expected to change, I'd point out that the regexp has a strong smell of code duplication: nearly all the tags listed have the same form: . I'd probably start with an array of allowable tags, and construct the regexp from those:

String constructRegex() { String[] containers = { "p", "b", "strong", "i", "em", "s", "strike", "blockquote", "sub", "super", "h1", "h2", "h3", "pre", "code", "ul", "ol", "li" }; String[] emptyTags = { "br", "hr" }; List components = new ArrayList(); for (String tag : containers) { components.add("</?" + tag + ">"); } for (String tag : emptyTags) { components.add("<" + tag + "\\s?/?>"); } components.add("</a>"); components.add("<a[^>]+>"); components.add("<img[^>]+/?>"); return StringUtils.join("|", components); }

The advantage of this is that is says things Once And Only Once. The disadvantage is that it's a bit less readable for people who are already fluent in regular expression syntax.

Where does this all leave us? That there isn't a single right answer. A lot of decisions in programming come down to taste, and knowing your audience.

I'll be saying more about this in a future post, especially as it relates to unit testing.

Not ready for Agile

noreply@blogger.com (Pete Bevin) — Thu, 26 Jun 2008 16:13:00 +0000

Some companies just aren't ready for Agile.

How to get better at programming

noreply@blogger.com (Pete Bevin) — Mon, 23 Jun 2008 14:21:00 +0000

A great article from Jeff Atwood called The Ultimate Code Kata. If you're at all interested in getting better at programming (rather than getting better at Java/C++/PHP/whatever), read and act.

Learning C is important, but you can still get a job without it

noreply@blogger.com (Pete Bevin) — Fri, 23 May 2008 19:55:00 +0000

In the second Stackoverflow podcast, Joel Spolsky argues that all developers should know C, and Jeff Atwood disagrees. Bloggers are up in arms; most of them agree with Joel, and Alastair Rankine even uses it to argue that Jeff has jumped the shark.

Here's my take: No, Virginia, you don't have to know C to be a professional programmer, but it definitely helps. You don't have to know Ruby, Java, Prolog, C# or Lisp either, but they will all teach you things that you can apply to whatever you happen to be doing for your day job.

Here's a random list of things a well rounded programmer should be comfortable with:

Macros (C- and Lisp-style) and their pros and cons.
At least two assembly languages (x86 and SPARC for preference): pipelines, caches, scheduling and calling conventions.
At least one dynamic language such as Ruby or Lisp
Data structures: hash tables, red-black trees, B-trees, graphs, linked lists.
Complexity theory: O(n) notations and how to reason about algorithmic efficiency.
NP-complete problems and methods for solving them (branch-and-bound search; simulated annealing; genetic algorithms)
Memory management: C-style (malloc/free), garbage collection (mark and sweep, and generational)
Running a business - if you've founded a startup, even if it failed, you'll have learned a lot from it.
Compilers (to machine code and to bytecode). You should definitely implement an optimizing compiler at least once. Extra points if you bootstrap it.
Efficient bytecode interpretation and Just-in-time compilation (plus specific bytecode languages for at least 2 languages).
Relational database theory and practice: data normalization, optimizers and ORM.
Threads in Java and in C; patterns for synchronization and locking.
Game development - sprites, 3D rendering; maze generation and search; game AI. A good place to start is Geoff Howland's article on GameDev.net.
Web site development: HTML, HTTP and forms of RPC such as SOAP, REST, XML-RPC.

Server-side web frameworks such as Ruby on Rails, Django, PHP and Struts, and their pros and cons.

Data presentation and techniques for communicating trends. Edward Tufte is the undisputed king of the realm.
Unix internals: inodes, memory management, disk management, kernel-space versus user-space. Ideally, you should implement a miniature operating system at least once in your life.
Linux: at least make your own distro from scratch.
Network protocols: SMTP, HTTP, and Bittorrent at minimum.

Is anything above necessary for being a professional programmer? Absolutely not. But the insights you get from learning them are important in ways you won't appreciate until you try.

Unit testing: not just for testing

noreply@blogger.com (Pete Bevin) — Fri, 23 May 2008 17:59:00 +0000

I've said it before and I'll say it again: unit testing is not a testing technique; it's a design technique.

Got that? The point of the technique is to reduce coupling between modules, which promotes code reuse and means you don't get nasty surprises when you come to make changes to it later. The fact that it also inoculates your code against future breakage is mostly incidental.

Vikar Hokstad gets it. If you can't instantiate business objects without connecting to a database, your code is too tightly coupled and it's going to drag you down.

Over the years, I've heard a lot of excuses for why code isn't unit tested, and nearly all of them boil down to "it's too hard". Michael Feathers wrote a whole book on this, which turned out to be a 450-page euphemism for "try harder then". There's always a way, and your code will be better for you spending the time to find it.

Gordon Ramsay and programmers' egos

noreply@blogger.com (Pete Bevin) — Fri, 16 May 2008 13:21:00 +0000

If you've never watched Ramsay's Kitchen Nightmares (the British version, not the Fox travesty), you should. Gordon Ramsay visits troubled restaurants and tries to turn them around - sometimes succeeding, sometimes not.

As you watch the episodes, you start to see a few common themes:

Chefs cooking for an imagined audience that doesn't really exist
Overcomplicated food
Egos getting in the way of a quality product

All of this can be translated pretty easily into software:

Programmers adding features that make the product harder to use
Software that is so over-designed that it has lost its flexibility
Beloved design patterns that don't apply to the task at hand.

Today we're getting ready to release up.time 5. I started here at uptime when it was back at version 3, and quickly realised that my first priority had to be simplifying the code base. As the product had evolved over the years, techniques that people had thought were really cool at the time were just getting in the way; if they had used the direct, obvious methods, it would have been much easier to change them with the product's needs. A few months and a lot of hair-tearing later, we'd managed to simplify most of the main areas, and by now, though I say it myself, the code base is really quite good.

Ramsay's advice for pretty much all the restaurants he visits is this: Stick to simple dishes with good, fresh ingredients. He applies this rule to small "hole in the wall" places right up to French restaurants with Michelin stars. And the software equivalent of this looks something like:

Keep the design obvious so that it can change later on;
Focus on end-user features ("ingredients"), not programming techniques.

That is: if you're building a whizzy new database library before there's an easy way for customers to get your product up and running, there's something wrong - and that something is that your ego is getting in the way. Don't let it happen to you!

Monitor This!

noreply@blogger.com (Pete Bevin) — Fri, 09 May 2008 18:58:00 +0000

In case you were wondering what the inside of one of these looks like:

There are pictures over at Royal Pingdom. Very tasty.

Perl is here to stay

noreply@blogger.com (Pete Bevin) — Sat, 12 Apr 2008 16:26:00 +0000

It's ugly, it's dated, and only its mother could love it, but Perl isn't going away. Actually, I have a soft spot for the language; it's the fastest and easiest way to do a lot of small jobs on the command line, and the library support is great.

Java 6 update 10

noreply@blogger.com (Pete Bevin) — Sat, 12 Apr 2008 16:18:00 +0000

The upcoming Java 6 update 10 isn't just a point release - it adds a lot of new features:

More modular JRE downloads: if you don't need the entire JRE feature set in your app, you can break it down to just the pieces you need.
A new Look&Feel module called Nimbus. Unlike Metal, it looks nice.
Much, much better startup performance, which has been my longest standing complaint about the platform.

More analysis at Ajaxian.

Finally, a release to get excited about!

If there are epigrams, there must be meta-epigrams

noreply@blogger.com (Pete Bevin) — Tue, 01 Apr 2008 20:43:00 +0000

In 1982, Alan Perlis distilled what he had learned from a lifetime of programming into 130 pithy quotes. TWENTY FIVE YEARS AGO. And most of them are still relevant.

Some of the best:

Syntactic sugar causes cancer of the semi-colon.
It is easier to write an incorrect program than understand a correct one.
If you have a procedure with 10 parameters, you probably missed some.
If a listener nods his head when you're explaining your program, wake him up.
Everyone can be taught to sculpt: Michelangelo would have had to be taught how not to. So it is with the great programmers.
There are two ways to write error-free programs; only the third one works.
In English every word can be verbed. Would that it were so in our programming languages.
Making something variable is easy. Controlling duration of constancy is the trick.
If there are epigrams, there must be meta-epigrams.

How to undelete files on Linux

noreply@blogger.com (Pete Bevin) — Fri, 28 Mar 2008 20:41:00 +0000

Carlo Wood explains how to undelete files on Linux ext3 file systems. The really cool thing is not that you can do it, but that Carlo decided not to believe the received wisdom that it was impossible, and went and figured out how to do it anyway.

One of my mantras is "I don't mind you complaining that it can't be done, as long as you don't bother the people who are doing it".

SLA Inversion

noreply@blogger.com (Pete Bevin) — Fri, 28 Mar 2008 15:13:00 +0000

So you’ve launched your new enterprise web site, and you’re so confident with your fully redundant web architecture that you figure you can provide a five-nines SLA - 99.999% uptime, or 5 minutes downtime a year. After all, what can possibly go wrong?

Welcome to SLA inversion, where a 99.999% available web service depends on a 95% available DNS server. Or a 98% available Internet connection. If your users can’t do work, it doesn’t matter why - they don’t care that your web application is still able to process transactions; they don’t appreciate the difference between application failure and DNS failure.

As it happens, this topic is close to my heart: I run a few cancer support mailing lists, and one day I decided to switch the domain name hosting provider. What I didn’t realise was that the new registrar wouldn’t let me edit the zone file until the domain transfer had completed, and that left me dead in the water for several hours: requests to the web site were going to the registrar’s default page. No matter that the web site and mailing lists were just fine; the point was that nobody could reach them.

Whenever you’re monitoring for a service level agreement, you need to follow a few important rules.

Measure as close as you possibly can to what end users will see. That generally means running synthetic transactions at a minimum: HTTP for web applications and test emails for email services.
Take time to think through all the services your users rely on, and make sure they’re monitored, implicitly or explicitly. For example, are there routers outside your control on the path between you and your users? If so, find out what SLA, if any, is in effect. You can’t provide a better target than the weakest link in the chain.
If there is a weak link controlled by your customer, make sure it’s explicit in the SLA contract.

Service level agreements are all about avoiding nasty surprises: taking a bit of time up front to think through how you might be caught off guard will benefit you, and more importantly, your users.

Worth 80 programmers

noreply@blogger.com (Pete Bevin) — Fri, 28 Mar 2008 15:00:00 +0000

As soon as you tell management that the changes they want will delay the release date, their first question (in most companies at least) is how many extra people you want. Brooks's Law, that adding manpower to a late project makes it later, is not well understood outside the programming field.

Steve McConnell has a great anecdote about a late project where they took all 80 developers off the project and replaced them with one highly competent guy, who finished on time. The point is not that you should do this, but that the story is believable.

A good rule if you're in a large organization and someone offers you a horde of developers to improve productivity is to take them, find out which of them are any good, and assign the rest to tasks that don't touch production code (e.g., test case automation). A good programmer is at least 10 times as productive as a mediocre one, so keep the quality of your actual production team as high as you possibly can.

Filed under "thoughts that make me happy not to work for a bank any more..."

Issues of Trust

noreply@blogger.com (Pete Bevin) — Thu, 27 Mar 2008 14:13:00 +0000

One of the big draws of open source software is that because you can examine the source code, you can, at least in theory, find bugs in the software and correct them. The idea that "with enough eyes, all bugs are shallow" reflects this. But Ken Thompson, one of the creators of Unix, gave a beautiful talk in 1984 called Reflections on Trusting Trust, which describes an ingenious method for introducing a security back door that would even work when you have all the source code available.

Let's say you wanted to modify the SSH daemon to remember the username and password of anyone who logs in, and send you the results. It would be a reasonably small change to the software to allow this, and assuming you had root access, you could easily drop in a modified version. But next time someone recompiled sshd, the change would be lost. How might you make your change more permanent?

Ken's attack uses the C compiler itself as the weak spot. First, he introduces code in the compiler to detect when it's compiling sshd and output the hacked version:

if (compiling_sshd()) {
output_hacked_sshd();
} else {
compile_normally();
}

But again, if you recompile the C compiler, the change gets lost. So let's add another check, this time for the C compiler:

if (compiling_gcc()) {
output_hacked_compiler();
} else if (compiling_sshd()) {
output_hacked_sshd();
} else {
compile_normally();
}

But here's the key idea: I can now remove this code from the compiler because the binary version will take care of it. Future compilations of gcc will re-insert the block, and so invisibly infect the binaries.

So if you could make this change on, say a Debian maintainer's machine, it could take quite a while before it gets noticed. Caveat Emptor...

Server monitoring is child's play!

noreply@blogger.com (Pete Bevin) — Wed, 13 Feb 2008 16:09:00 +0000

May I present the world's first photograph of an OLPC doing performance monitoring:

If the UI looks unfamiliar, it's because we're running a development version of the forthcoming up.time 5 codebase, which has a (IMHO much nicer) new look.

How to improve software

noreply@blogger.com (Pete Bevin) — Thu, 17 Jan 2008 19:32:00 +0000

I was talking to someone recently about how to improve the quality of his software. He’s reached the point where he can see there is a problem, but he doesn’t yet know how to solve it.

Here at uptime, we have a beautifully disciplined software development process that does everything from test-driven development to continuous integration and fast feedback using Cruise Control. Our code is mostly well factored, reliable, and easy to work with, mostly because we haven’t skimped on process. The temptation to take shortcuts in the face of looming deadlines is always there, but we also know that doing that would lead to long term pain.

As I talked to my friend, I realised that you don’t have to do everything at once; in fact, it’s better to do a bit at a time. So here’s my N-step process for getting your software under control.

Step 1: Make a code-level regression test suite. Your software keeps breaking in weird ways, and you can’t possibly predict everything. But you can certainly do something. For example, if you are building a web-based application, you could check that the front page of the app has a login form on it. And once you’ve done that, you could check that a login works OK and the username is displayed on the resulting page. These tests could be as simple as using grepping the output of wget, or you could use a more sophisticated tool like Selenium. Either way, you have something that can tell you if something is badly broken. And what if you added one test a day to the suite? After three months, you’d be testing for sixty possible failure conditions every time you ran the suite.

Step 2: Automate the build process. You can almost certainly find a way to script a software build, database reset and regression test run automatically. Maybe you have to commandeer a physical machine; maybe you can set up a vmware instance. The important thing is that it can run unattended. This is called a Smoke Test because you are checking to see if anything “catches fire” when you run it.

If you’re using a language like PHP that doesn’t require compilation, you will probably want to at least do a syntax check against all your PHP files. On Unix, PHP comes with a command-line program and you can run “php -l”. Other languages generally have similar syntax for doing a syntax check on a script without running it (Perl, Python and Ruby all use the -c flag).

What you’ll probably find once you’ve done this is that (a) the suite fails more often than you’d expect, (b) there are random bugs that keep slipping through (but you can add tests for at least some of them), and (c) your team often forgets to run the tests before checking in code. Which leads to...

Step 3: Run the tests automatically. In the old days, we used to run build scripts from cron once a day, and then look at the output in the morning. Nowadays, we have continuous integration (CI) tools such as Cruise Control that wake up every few minutes, check CVS for new checkins, run a build and unit+regression test, and notify the developers that broke the build.

CI tools have a reputation for being hard to set up, but mostly because it’s hard to get a repeatable build process. If you’ve followed steps 1 and 2 already, you’ll have that working and the rest of the setup will be a breeze.

Other advice: Don’t try to do all this at once. Build a couple of tests, and get used to running them. Then start automating parts of the build and run it every day until you’re comfortable. Only once you can run a single command to build and test should you take the final step to running Cruise Control.