Mission control for coordinated software deployments

It’s nice to release software in small increments. Small deployments are more manageable, and they can be rolled back faster if there’s a problem. With good planning, even major changes can be broken down into smaller chunks and drip-fed into production. But as a project manager or team leader, now and then you come across a gnarly project where — for whatever reason — there are a bunch of interlocking changes that all have to go out together. You’re responsible for updating a dozen critical services and databases. Everyone’s relying on you to get it done. And it’s scary.

Mission Control

A technique I’ve used for dealing with these situations is to set up a Mission Control Centre for the release. The idea is simple: get empowered representatives to commit to being present for the entire span of the release.

• Empowered representatives means people from all disciplines who have contributed to the project or will be affected by its outcome. In addition to the obvious software developers and operations staff could mean people like commercial stakeholders, designers, and customer support.
• Being present is more than just being in the room — it means that when you’re in the room, you’re paying attention to what’s going on, even if you’re not “at the wheel.” You shouldn’t have to ask for a recap of the last half hour of conversation if someone suddenly needs your opinion.
• The entire span really does mean the full duration of the deployment. Block out calendars. Rearrange meetings. Even if a representative only has one part to play during the deployment, they shouldn’t wander off as soon as they’re done.

The purpose of all this is that if something goes wrong the people who can solve it are all in immediate communication with each other. You don’t have to go scrambling to find someone outside the group if you need to make a hard call. You don’t have to spend ten minutes bringing them up to speed on context. You can make decisions fast, and act on them immediately.

Setting it up

The idea may be simple, but it can be hard to arrange. People are busy. It can be hard to get them to clear an entire morning or afternoon in their calendar, let alone a whole day. Software developers are notorious for disliking meetings, and may not see the point of sitting around for hours when all they have to do is run one task in a checklist of twenty.

The first step towards success is therefore getting people on board with the idea. You can use the standard tricks of offering food and treats for everyone in the room. But don’t underestimate the power of cool code names and the very fact of calling it “Mission Control,” either! Stick up a poster on your meeting room door, and you’ll get people envious of the clearly important stuff going on inside. If you’ve got a hot designer on your team, get them to make some mission patch laptop stickers for participants.

When I talk about “being in the room”, I mean that in a broad sense. The point is to optimize communication flows, but “optimal” is different for every team. At FanDuel, our teams are spread over multiple offices in different time zones and include remote workers in home offices. If it’s possible to get everyone in a single room, try for that. If not, use a video call and get everyone to be dialled in throughout the event. If you don’t have video calling available, get everyone together in a single chat room.

The checklist

Atul Gawande’s book The Checklist Manifesto is about cross-functional teamwork in industries with a high degree of specialisation, such as construction, air travel, and medicine. The lessons apply to the software industry as well. Not every item on a checklist is a measurement to be taken, or a switch to be flipped. Some of them seem non-obvious, and perhaps even counter-intuitive. In the World Health Organization’s Surgical Safety Checklist, the first item before a surgeon makes a cut is “Confirm all team members have introduced themselves by name and role”: