AgentOps Runbook Template

An AgentOps runbook tells an authorised operator what to do when an agent's operating condition changes. It connects a recognisable symptom to a bounded action, the evidence needed to choose that action and a way to establish whether recovery worked. A diagram of the architecture or a list of dashboard links cannot do this alone.

This template is for service owners converting engineering knowledge into a usable operational document. Its worked example concerns association membership-renewal notifications. The example is illustrative: the agent prepares approved administrative messages and submits them through a delivery service. The runbook governs operation; it does not confer permission to change membership terms or contact policy.

Write the first screen for the person receiving the alert

Start with the service name, environment, accountable owner, runbook version and review date. State the symptom in observable terms: the confirmation queue is ageing, submission results are uncertain, or the approved recipient source has become unavailable. “The AI is behaving strangely” is too broad to identify a safe response.

Put the immediate safety boundary near the top. For this notification service, an operator may pause new submissions and investigate existing items. They may not regenerate every message, replace the recipient list or assume a timed-out request was never accepted. These limits should match actual system permissions.

Google's incident-management guidance connects prepared playbooks, exercises and clear response roles. The document below applies that preparation to agent workflows, where a model response, a requested external action and a confirmed business result can be different events.

Copyable runbook header and decision record

Runbook fields for a membership-renewal notification service

Field	What to record	Operational test
Scope	Named agent, deployment, environment and notification cohort.	Can the operator identify exactly which workload to pause?
Authority	Permitted containment actions and the owner of exceptional decisions.	Are those permissions available without borrowing another person's credentials?
Trigger	Observable alert or manually reported symptom and its evidence link.	Does the condition represent an actionable problem?
Preconditions	Required access, current deployment version and authoritative destination status.	Can the operator verify them before acting?
Procedure	Numbered actions, expected results and branch conditions.	Is the next step clear when an expected result does not appear?
Recovery check	Confirmation evidence, queue reconciliation and observation window.	Does it establish restored behaviour rather than merely a healthy process?
Escalation	Named duty role, contact route, fallback and response boundary.	Can somebody with the required decision rights be reached?
Evidence record	Incident reference, affected task identifiers, action times and unresolved items.	Can the next operator continue without reconstructing the incident?

Use references to approved access mechanisms instead of embedding passwords, tokens or private customer records. Give the operator enough context to locate restricted evidence through authorised access. A runbook copied into a wider collaboration space should not become a second store of sensitive payloads.

Worked procedure: renewal submissions with uncertain results

Assume an illustrative batch of 80 membership-renewal notices. Before the alert, the service records 61 accepted submissions, seven requests with an uncertain destination result and 12 items not yet submitted. Acceptance by the delivery service is not proof that the member received or read a message. The batch ledger must preserve these distinctions.

1. Confirm the affected scope. Match the alert to the deployment and batch. Check whether the symptom affects this cohort, other batches or a shared delivery component. Record the current agent and approved-message versions.
2. Pause new submissions using the authorised control. Confirm the pause took effect at the dispatch boundary. A stopped user interface is insufficient if a worker can still submit queued work.
3. Preserve the seven uncertain requests. Record their stable task identifiers, destination references where available and last known state. Do not reclassify them as failed solely because the caller timed out.
4. Query the authoritative destination. Reconcile each uncertain request through the supported status mechanism. Where evidence remains unavailable, retain the uncertainty and escalate according to the approved procedure.
5. Decide the disposition of the 12 unsent items. Confirm that the recipient source and message approval remain valid. A missed renewal window may require a business decision about changed wording or contact timing.
6. Resume a bounded cohort only after approval. Observe the intended confirmation evidence and queue behaviour. Expand intake only when the recovery criteria are met.

This is a procedure pattern, not a claim that a particular delivery platform supports a specific status query. Replace each generic control with the verified method for the deployed system, including access requirements and the expected response. The retry and idempotency guide explains how to design action identity before an incident makes it necessary.

Describe branches, including when to stop

A useful step has an observation and a branch. If the dispatch pause is confirmed, proceed to reconciliation. If it cannot be confirmed, escalate containment and avoid actions that increase the affected workload. If destination evidence conflicts with the local ledger, protect both records and assign a reconciliation owner.

Write explicit stop conditions for unauthorised access, missing evidence, changed message approval and a suspected privacy incident. “Try again” should never be the default branch for every error. An operator who stops under a defined condition has followed the runbook, even when the service remains unavailable.

Separate technical recovery from permission to resume the business process. Restoring a connector does not answer whether an expired notification is still appropriate. Connect those decisions to the human oversight operating model, with a named business authority and a safe waiting state.

Include the dependencies that matter to a decision

Document the recipient source, approval store, identity provider, queue, delivery service and confirmation mechanism. For each, state what happens when it is unavailable or stale. Avoid reproducing the full architecture catalogue; include the relationships needed to choose containment, escalation or recovery.

A dependency can be healthy while the combined process fails. The delivery API may respond normally even though the agent is using a superseded recipient extract. The runbook therefore needs freshness and version checks, not only availability checks. Use the agent inventory as the durable register and link to the relevant records.

Prove that recovery restores the promised behaviour

Define recovery in terms of permitted work completing with the required evidence. For the example, reconcile the original 80-item cohort into known dispositions, identify unresolved items and verify that resumed submissions use the approved message and recipient source. A green host metric alone does not establish this result.

Set an observation window appropriate to the normal confirmation delay and operational consequence. There is no universal number of minutes that makes an agent recovered. State who can close the incident, who accepts outstanding work and what must remain visible after closure.

Keep recovery evidence concise but traceable. Record what was paused, which items were reconciled, which configuration changed and how the restored process was checked. Use the broader agent recovery guide for containment and reconciliation design; this runbook should reference the chosen implementation.

Test the document with somebody who did not write it

Run an exercise in a controlled environment with realistic uncertainty. Remove a convenient evidence link, simulate a destination timeout or change the approved source version. Ask an authorised operator to explain each decision while following the document. Observe missing context, unavailable access and ambiguous branches.

Record the result as a usability test, not a performance contest. An operator should not be encouraged to improvise beyond their authority to finish faster. A useful finding might be that the procedure is correct but the designated role cannot access the confirmation ledger.

Include the tested runbook in the handover to operations. Establish an update trigger for new tools, changed authorities, different queue behaviour or an incident that invalidates a branch. Assign an owner and review evidence; a date stamp without a rehearsal does not prove usability.

Control who changes the service during the procedure

Record whether another operator, deployment pipeline or scheduled job can change the same configuration while the runbook is active. A correct recovery step can become misleading if an unrecorded release changes the queue or message version halfway through reconciliation. Nominate a coordination owner and use the existing change mechanism to make concurrent work visible.

Do not solve this by imposing an undocumented global freeze. State which changes are restricted, which containment actions remain available and who can approve an exception. If urgent repair is necessary, record the old and new versions and restart the affected verification step. The objective is a coherent account of the operating state, not administrative delay. At handover, the next operator should be able to distinguish a deliberate repair from unrelated configuration drift without relying on a private chat.

Keep routine operation and incident response connected

Not every runbook is an emergency document. Routine procedures can cover approved configuration updates, retiring an agent or reconciling a daily exception queue. Give each procedure the same essentials: scope, authority, prerequisites, actions, expected evidence and stop conditions.

Use the on-call escalation design to route urgent failures and the AgentOps service overview to organise the wider operating model. Discuss a production-agent runbook with a masked alert, the present procedure and a representative unresolved task. These materials reveal whether the gap lies in documentation, access or the controls themselves.