4. Timeline
The timeline of a CoE describes what happened when and who did what. It should answer most questions:
When was the problem first discovered, and how?
Who noticed or reported the problem first to us?
Who engaged in the incident, and helped recover from the failure?
When was the failure resolved?
What indicators did we use to prove it was resolved?
How did we prove that the failure was resolved?
There are two durations in every incident and a good timeline states both. The first is how long the failure lasted. The second is how long it took us to know about it, and it is the more useful of the two, because it is usually the one we can do something about without touching the system that failed at all.
A timeline that begins at the first alert conceals it. If the change went out at 19:42 and the first alert came at 20:03, those twenty-one minutes are the most interesting entry in the document, and they are the only entry with nothing written against them.
The more accurate and specific the timeline the better. There are often process failures during the incident response itself which should be assessed and learned from. In many cases, a failure might have been responded to or fixed more quickly, had we known about the exact nature of the problem sooner — therefore the alerts we received, and the actions we took discovering and analysing the nature of the problem, are just as important as those around the cause of the failure itself.
The CoE timeline should be a simple list of time and event, focusing on people and their actions.
21:01 - Incident opened. First alarm tripped, engineer Bob investigates
21:02 - Bob alerts Operations team of inital impact
21:05 - Cathy comes online
21:07 - Bob begins reviewing all changes from today
21:10 - Cathy opens up conference bridge and sends initial alert email
21:14 - Configuration item changed this afternoon identified as potential issue - testing
21:17 - Testing confirms likely configuration issue as cause
21:18 - Rollback of changes Xd4DD begun
21:22 - Rollback completed in production, testing begins
21:23 - System confirmed online, testing begins
21:31 - All production tests pass and system confirmed as working again
21:35 - Conference bridge and incident closed.
After investigation, and particularly after subsequent actions taken, it is perfectly valid to expand the timeline to well before the incident, particularly if subsequent data comes to light that shows how it might have developed into failure, or could have been mitigated earlier.
A timeline is written in one timezone and it says which one. This looks like a triviality until the document is read in another office, or read back a year later across a change of clocks, and half the entries move by an hour. Where a team is spread out, UTC costs nobody anything and removes the question entirely.
Some of a timeline is observed and some of it is reconstructed, and the difference should be visible. An entry taken from a log line or a chat message is a fact. An entry reading the job began failing at around 19:40 is a conclusion, and it will be read as a fact by everybody who was not in the room. Where an entry is inferred, say so, and say what it was inferred from.
Name people, or name roles, and be consistent about which. A timeline of things that happened, with nobody doing them, cannot be checked by the people who were there, and it reads as though the system recovered itself.
Naming people is only safe in an organisation that has genuinely committed to no blame, which is one of the reasons that commitment comes first — see Implementation notes. Where it has not been made, timelines quietly become vaguer, and the first thing to be lost is the detail that made them worth writing.
Many engineering teams use chat systems (such as Slack, Teams, Discord or IRC) habitually, and particularly during incident response. Incident Response teams should think aloud in these chat forums during an incident, and the log of chat and actions taken will make an excellent start to any CoE timeline, and should be preserved as soon as possible after the incident.
The chat log is also the only record that reliably survives. Memory of an incident degrades within days, and it degrades towards a version in which the right thing was done sooner than it was. Preserve the log before anybody has had a night's sleep.
Resist tidying. The timeline of a well-run incident still contains the fifteen minutes spent looking at the wrong system, and that quarter of an hour is usually where the Lessons Learned come from. A timeline in which every step was correct is describing an incident that did not need a CoE.
Experienced CoE practitioners will begin keeping the timeline during the incident itself.