Circuit Breakers

It is extremely difficult to write bug-free software.  Even if you managed to acquire the most talented developers in the world, gave them plenty of time to release, allowed them to use the best suited methodologies for the project, and had well-specified requirements, I'm confident the resulting software would still have bugs.  There are many reasons behind this unfortunate truth.  According to Fred Brooks in The Mythical Man Month, software construction is probably one of the most complex endeavors yet undertaken by human beings.

Many techniques are helpful to reduce the amount of bugs in a codebase.  Writing clean readable code is probably the easiest.  Several levels of testing (unit, integration, functional, user, load, performance, etc.) are the most obvious.  In Facts and Fallacies of Software Engineering, Robert Glass shows that code inspection (which includes pair programming) when done frequently and in small groups is also an effective way to reduce bugs.  Unfortunately, all of these techniques will only reduce the number of bugs, not guarantee the elimination of all of them.  As developers we have to accept that we will ship software with bugs.

The question is what can we do as developers to reduce the impact of bugs as much as possible given that we know they will occur in production.  Michael Nygard, in his book Release It!, introduces what he calls the Circuit Breaker pattern.  The software pattern matches very closely to the real world pattern.  No circuit in your house should draw more current than it is rated to handle.  Of course at some point, somewhere, someone will plug in enough power hungry devices to overload a circuit.  Before circuit breakers, this would cause a fire.  With circuit breakers, the circuit is broken, and the house remains safe.

This technique involves more than a simple try-catch block.  It involves keeping timeouts for interactions with external systems.  It involves limiting the total number of times an operation will be retried before alerting the user to the problem.  It involves monitoring resources such as disk space or memory and terminating non-essential actions until the resources are freed.  It involves limiting the number of rows that can return from a database query.

While every reasonable effort should be made to eliminate bugs, as a professional developer you have to admit that bugs are not 'ifs' but 'whens'.  Circuit Breakers are an excellent defense against the intractable reality of software bugs.