AgentOps Incident Post-Mortem Template

An AgentOps incident post-mortem explains how an incident became possible, what limited or increased its impact and which changes will prevent or reduce a repeat. It should preserve evidence and uncertainty, assign useful actions and avoid reducing a complex operating failure to the last person who touched the system.

This template is for an incident owner preparing the learning review after containment. It complements the agent recovery process rather than replacing immediate response. Within AgentOps, the post-mortem connects a production event to changes in controls, evidence, ownership and operating practice.

Set the learning question and the scope

State the affected service, incident period, known impact and review owner. Explain why the event warrants review: an incorrect external action, an uncontained uncertainty, a near miss, repeated operational effort or a failure of an important control. A review can be valuable even when intervention prevented the final consequence.

Google's SRE workbook on post-mortem culture describes a learning process that avoids blame while tracking concrete corrective work. Blameless does not mean ownerless. The review should examine the information and constraints people faced, while assigning responsibility for improving the system they operate.

Agree who needs to contribute: the operator, engineering owner, affected process owner and relevant specialists. Use a protected evidence location and avoid unnecessary personal or customer detail in the shared narrative. Preserve source records according to the applicable evidence and retention arrangements.

Illustrative incident: a shuttle notice uses a superseded schedule

Consider an internal employee-shuttle notification agent. It prepares notices from an approved schedule and submits them through the organisation's communication workflow. In this illustrative incident, a pickup time changes from 16:10 to 16:25 in schedule version V4, replacing V3.

A notice prepared after the change still uses V3. A staff member reports the discrepancy before the pickup. Operators pause the affected notice path and investigate which tasks used the superseded schedule. This is a hypothetical operating example, not a report of an actual customer incident.

The learning question is how an ineligible schedule supported a current notice and why the operating controls did not identify it before submission. The review should not assume that the model invented the time: the old time may have been supplied accurately from an unsuitable source.

Build a timeline of observed events

Illustrative incident timeline; all times use the same local clock

Time	Observed event	Evidence and limits
14:00.	V4 becomes the approved schedule; V3 is superseded.	Schedule approval record and effective version.
14:20.	A task prepares a notice containing the V3 pickup time.	Task artefact and cited source identifier.
14:28.	A staff report reaches the operating team.	Report timestamp; earlier awareness is not established.
14:31.	The affected dispatch path is paused.	Control record and verification that queued submissions are held.
14:40.	The team identifies a source-eligibility discrepancy.	Comparison of task evidence with the schedule register.
14:50.	A controlled test reproduces use of superseded material through the affected path.	Test record; broader prevalence still requires investigation.

Use timestamps from their source systems and record clock or ordering uncertainty. If the submission time is unknown, do not substitute the preparation time. Preparation, request acceptance, delivery and reading are different events; the impact account should say which of them is established.

Copyable post-mortem structure

1. Summary: what happened, the service boundary and the current state.
2. Impact: confirmed affected tasks or users, potential exposure and unresolved scope.
3. Timeline: observed events with evidence references and uncertainty.
4. Mechanism: how the failure crossed task, evidence, decision and action boundaries.
5. Contributing conditions: design, workload, access, change and coordination factors.
6. Response: detection, containment, reconciliation and recovery, including what worked.
7. Actions: owners, priority, completion evidence and follow-up decisions.
8. Learning distribution: related services that should inspect the same condition.

Write the summary after the evidence has been assembled. Keep it understandable to a business owner without losing the distinction between confirmed impact and possible exposure. An executive summary should not turn an uncertain task count into a definitive number for convenience.

Establish impact without overstating what the logs prove

Identify the cohort that used the affected source path and version. Separate notices prepared, submissions accepted, delivery confirmed where supported and recipients known to have acted. A provider acceptance record is not proof that a person read or relied on a notice.

For the shuttle example, the wrong time is confirmed in an artefact. Whether every related recipient received it requires additional evidence. Record the investigation owner and the next verification for that uncertainty. Do not state that nobody was affected merely because no complaint has arrived.

Include operational impact such as manual reconciliation, delayed notices and staff time, with explicit measurement limits. Avoid attaching a financial loss figure unless there is a defensible basis. The review can explain a consequence qualitatively when reliable quantification is unavailable.

Describe the mechanism and contributing conditions separately

A mechanism explains the path from conditions to outcome. For example, the approved-source status changed, but an affected retrieval path continued to supply V3, and the notice process did not require evidence of current eligibility before submission. Each part needs support from the investigation.

Contributing conditions may include a release that did not test superseded-source handling, an interface that omitted version status or an operating metric that counted submission as success. Do not list plausible causes as facts. Mark hypotheses and identify the test or record needed to resolve them.

A human missed check may be part of the timeline, but ask why the system depended on that check, whether the person had the right evidence and what workload or incentives affected the decision. “Be more careful” rarely specifies a control that somebody can verify.

Turn findings into actions with closure evidence

Illustrative corrective-action register

Action	Accountable role	Evidence of completion
Enforce source eligibility at the affected path.	Engineering owner.	Controlled test holds a task using superseded material and permits the valid current case.
Make source version and status inspectable during review.	Workflow owner.	A reviewer demonstrates recognising and challenging the invalid source.
Reconcile the affected task cohort.	Operations owner.	Each identified task has a supported disposition or an explicitly owned unresolved state.
Update the response procedure.	Runbook owner.	An authorised operator rehearses the revised containment and verification branches.

Choose dates and priorities from the consequence and the work involved. A deadline without a test of completion can encourage superficial closure. Keep immediate protections separate from permanent corrective work and state who accepts continued operation while that work remains open.

Review what worked, including successful intervention

The staff report may have arrived quickly, the pause may have stopped queued submissions and the evidence may have made the source discrepancy easy to identify. Record those strengths because they explain why the incident did not become larger and which controls should be preserved.

Also examine the cost of response. If containment required the original developer's private knowledge or excessive privilege, record that operating gap even when the response succeeded. The runbook and escalation design should be updated from verified observations, not from an idealised version of the incident.

Validate learning across related agents

Use the agent inventory to identify services sharing the affected retrieval component, source-status rule or notice workflow. Ask their owners to inspect the relevant condition. Do not assume the same defect exists everywhere, and do not broadcast customer-specific evidence to unrelated teams.

Add a targeted test that checks the known defect has not returned, with a current-source case and a superseded-source case. Tie the test to the change process so it remains relevant after tools or source rules evolve. The drift and regression guide explains how to keep such evidence useful.

Separate learning from the search for a single root cause

Some incidents have several necessary conditions. Removing any one might have prevented the observed outcome, but that does not make the others irrelevant. In the shuttle example, source eligibility, review visibility and dispatch verification can all matter. Explain which condition enabled the incorrect notice and which condition allowed it to progress, using evidence for each claim.

Use counterfactual questions carefully: would the proposed control have interrupted this specific sequence, and what evidence supports that answer? A generic training session may be helpful, but it is not a demonstrated interruption of a source-status defect. Prefer actions whose expected effect can be tested against the incident mechanism.

Record tradeoffs introduced by the repair. A stricter eligibility check may hold more notices when source status is unavailable. That may be the correct behaviour, but the operating owner needs a queue and response arrangement for it. Learning is incomplete if a fix prevents the original failure while creating an unowned backlog that pressures staff to bypass the new control.

Close the review without hiding unfinished work

A post-mortem document can be approved while corrective actions remain open, provided their owners, protection and follow-up are explicit. Distinguish document completion from incident recovery and from permanent action closure. These milestones answer different questions.

Discuss an AgentOps post-mortem review with the incident timeline, evidence boundaries and proposed actions. A useful review leaves the organisation able to explain what happened, what remains uncertain and how the next operating decision will be better supported.