The Badger’s first thought on hearing of the recent global IT outage was that the cause would be a cyber-attack or a messed-up change of some kind. It turned out to be the latter, which triggered a smile because the advances in tools, techniques, processes, and professionalism over the last three decades, hasn’t made it impossible to release software that causes havoc! The outage is a reminder that there’s some fragility in the underbelly of our computerised world, that complexity always comes with risk, and that individuals and enterprises should never have all their eggs in one basket.
The Badger was reminded of his involvement in stabilising and completing a troublesome software project a few decades ago. The software was installed in the client’s premises and being extensively tested on computers that were connected to specialist equipment and other systems. The project team was focused on the defect fix, build, and release cycles needed to support the testing, which was producing a continuous stream of faults, many more than anticipated. The fix/build/release team, under enormous and intransigent internal and client pressure, struggled to make inroads to the growing defect backlog, and to produce timely update releases. Costs escalated, shortcuts were taken, release quality suffered, and the client was angry that new releases fixed some defects but introduced new ones! The team were demoralised, defensive, and disgruntled. Team spirit had evaporated, and financial incentives were making little difference.
The situation was improved by two things. The first was the injection of hard-nosed, technically-savvy, leadership that kept all stakeholders at bay and also demanded rigour, quality, and process professionalism from the team. The second was to use a house brick to rebuild team spirit! Every Friday, a team member was allocated the brick for having made a mistake of some kind. The brick sat on their desk until it was reallocated the following Friday (if someone else had made a mistake). No one wanted to be a recipient of the brick! Rigour, process diligence, and fix to release quality improved. The brick also triggered good-natured banter across the team whose morale, motivation, and cohesiveness improved dramatically. On project completion the brick was mounted on a plinth and presented – amidst raucous merriment – for posterity to the person who’d received it the most! No one ever took offence at being awarded the brick, but would that be the case today? Doubtful.
Those whose change caused the recent global IT outage were probably a team of good people operating under relentless pressure from their stakeholders. Lessons, of course, must be learned, but deriding them for a mistake that exposed a global IT fragility would be a mistake. You never know, deploying a humble house brick might help them ensure they don’t make the same mistake again…
