6. Lessons Learned
Every event has lessons learned, and these should be candidly and clearly articulated.
It usually follows that each lesson learned has an action, and a person attached to that action as a followup. It perhaps isn't worth making this a cast-iron rule as part of your process, but is a good rule-of-thumb. If we learned something, then there is probably action associated with it, if only to ensure that all others learn the point too.
A lesson and an action are different things, and this section is much easier to write once that is clear. A lesson is a statement about the world that is now true and that we did not believe, or had never looked at, the day before. An action is something a named person does by a date.
Our monitoring did not cover this and would not have caught it is a lesson. Add an alert on cache-hit ratio for authenticated routes is an action. If a sentence in this section can be done, it belongs in Next Actions. If a sentence in Next Actions cannot be done, it probably belongs here.
Our monitoring is not sufficient for this type of event.
We were monitoring the service concerned, but not at sufficient granularity or regularly enough to catch the outages when they occurred.
We learned that it takes too long to test the system in production, manually.
Nine minutes passed between the first customer report and anybody treating it as an incident, because the report was recorded as a display fault. Our fastest signal was sitting in the contact centre and we did not recognise it.
The rollback itself was quick, and it was quick because the team has insisted on keeping rollbacks quick through two years of pressure to ship. That is worth saying out loud in a document about a failure.
Not every lesson is about the failure. Some of the most useful are about how we responded to it — that the bridge took nine minutes to open, that nobody knew who could authorise a rollback, that the only person who understood the system was on leave and was telephoned at home. Incident response has its own review, described in what does good incident response look like?, and where that review finds something belonging in the permanent record, this is the section it goes in.
Write down what worked, too. In a document about a failure this can feel like self-congratulation, and it is not. An organisation that only ever records what went wrong will, over a few years and a few reorganisations, quietly dismantle the things that saved it, because nothing anywhere says they mattered. If the recovery was fast because somebody fought to keep it fast, that is a lesson, and writing it down is what protects the practice.
Before issuing, read the Lessons Learned from the last year of CoEs.
A lesson we have already written down and not acted on is a far more serious finding than a new one, and it changes what the document is about. The event is no longer the subject. The fact that we knew, and it happened anyway, is the subject, and the Next Actions that follow from that are about the process which allowed a lesson to be recorded and then ignored.
This is also the only way the promise made at the start of this method — that patterns can be identified across historical CoEs — is actually kept. Patterns are not found by having the documents. They are found by somebody reading the old ones before writing the new one.
The hardest lesson to write is the one that points at a decision somebody senior made, and it is most often a decision to defer something in favour of delivery. It is also the lesson the whole method exists to make writable. A process that cannot record it will record everything except the reason, the Next Actions will address the symptoms of a decision nobody is allowed to name, and the same event will be back inside the year.
Be careful with lessons that are really complaints. The vendor is unreliable is not a lesson unless it comes with what we now know about how unreliable, how we found out, and what we intend to do about depending on them. A lesson we cannot act on and cannot measure is a feeling, and it will crowd out the ones that matter.
If we learned nothing, was it even an outage?