Postmortems without ceremony
Last Tuesday a deploy took down a client's checkout for forty minutes. A migration that looked safe in staging locked the orders table in production because the staging dataset was 400 rows and production was 2.2 million. By the time we noticed, the client had forwarded three customer screenshots of failed payments.
We fixed it, restored the old build, and sent a note to the client. Then the three of us sat down for thirty minutes the next morning. That was the postmortem.
When a team is three people, the standard postmortem format gets in the way. We do not need a document with sections for timeline, contributing factors, action items, and lessons learned. We were all on the call. We all watched the monitoring dashboard go red. The timeline is not in dispute. What we need is to agree on what actually happened, decide what we are changing, and get back to work.
The meeting has three parts, and we keep them strict.
First, one person explains what they think went wrong, start to finish. Not what the process was supposed to be — what they actually did, in the order they did it. The other two listen and correct. This takes about ten minutes because there is no ambiguity about who did what. There are only three of us. If Ankit ran the migration, Ankit explains it.
Second, we ask one question: what would have caught this before it reached production. Not five questions. Not a root cause analysis with a five-whys exercise. One question, and we hold ourselves to honest answers. This time the answer was straightforward: the migration was tested against a dataset that did not represent production scale. We have a staging database, but it gets seeded from a small sample. The size difference was the gap.
Third, we pick one or two changes and assign them to a person with a date. Not a backlog of improvements. One or two. If we cannot narrow it to two, we have not understood the problem.
This time we picked two. Ankit would write a script that copies a production-sized snapshot into staging weekly, and I would add a check to the deploy script that warns when a migration includes an `ALTER TABLE` on a table over 100,000 rows. Both were done by Thursday.
The thing about a three-person team is that the postmortem is not really about learning. We know what happened. We were there. It is about deciding what is worth changing and actually changing it. A template does not help with that. A conversation does.
We keep a single page in our internal wiki with a list of incidents and the one or two changes that came out of each. It is not a report. It is a record of decisions we made, so that six months later when someone asks why staging has a full production snapshot, we can point to the entry. That is the only artifact.
The client does not see the postmortem. They see the fix and the note we sent. If a change we make affects them — a new staging process, a deploy script that now pauses on large migrations — we mention it in the next status update. But the postmortem itself is internal, because it is for us, and the ceremony of a formal document would be for someone else.
The thirty minutes we spent that morning were worth it. The changes we shipped that week have prevented two similar incidents since. We know because the monitoring dashboard stayed green through the last two migrations, and the deploy script paused on an `ALTER TABLE` last month when someone tried to add a column to the payments table without checking the row count first.