Production monitoring
How to Monitor AI Agents in Production
Implement agent monitoring from durable task states to quality evidence, actionable alerts and controlled-failure tests.
To monitor an AI agent in production, start with a durable record of the work it accepts and the states that work can reach. Add execution traces, quality checks, resource signals and operator responses around that record. A dashboard of model calls alone cannot tell you which business tasks are waiting, incorrectly completed or missing from observation.
This implementation guide is for an engineering lead preparing a service for continued operation. It focuses on the monitoring system and its acceptance tests. The service's exact permissions, objectives and response commitments must be agreed with its owner; they cannot be inferred from the model or observability product.
1. Define the production work you intend to observe
Write down the task entry point, eligibility rule, intended result and authoritative confirmation. Identify valid non-completion outcomes, such as an authorised hold or a request for missing information. Define what remains unresolved. These states should exist in the application or business record, not only in a dashboard calculation.
For an illustrative example, a document-review reminder agent checks an organisation's internal policy register. When a review is due, it identifies the current owner, prepares a reminder and creates an internal follow-up task after any required approval. It does not determine legal review periods. The policy owner supplies those dates and rules.
Document DR-52 has a due date, a register revision and an assigned owner. A completed reminder requires the correct document and owner, an authorised request and confirmation from the follow-up system. An archived document may be correctly excluded. A missing owner requires intervention. These distinctions form the monitoring contract before a model is instrumented.
2. Emit a small set of authoritative lifecycle events
At admission, persist the task identifier, admitted time, source revision and service configuration reference. During execution, record meaningful transitions such as source checked, review requested, action submitted, action confirmed or task escalated. Preserve individual attempts without creating a new business task for every retry.
Include the reason for a transition and the responsible component. A model's statement that it sent a reminder is not the sending system's confirmation. The action adapter should record the external reference; a reconciliation component should validate the result where the initial response is insufficient. Keep uncertain actions visible until their state is established.
Use task tracing to connect the lifecycle with technical execution. A trace helps explain how the task reached a state; the durable task record helps identify work that has no complete trace. Neither should silently overwrite the other when they disagree.
3. Build a monitoring contract with seven questions
| Operational question | Signal | Response enabled |
|---|---|---|
| Is new work entering the service? | Eligible due records compared with admitted tasks | Investigate scheduler or admission failures |
| Where is work waiting? | Task age by current state and responsible queue | Resolve the specific bottleneck |
| Did the agent use current information? | Register and owner-reference versions | Refresh or suspend affected work |
| Are reminders acceptable? | Reviewed correctness of document, owner and disposition | Correct the failing rule or behaviour |
| Are external results confirmed? | Submitted actions matched to follow-up references | Reconcile uncertain actions |
| Is the service consuming resources abnormally? | Calls, retries, latency and resource use by task type | Contain loops or capacity problems |
| Can we trust monitoring coverage? | Admitted tasks, exported events and known telemetry loss | Repair the evidence pipeline and flag blind spots |
This table is a proposed operating design for the example. Replace its signals with the authoritative events in your own systems. Resist collecting a field simply because an SDK exposes it. Each signal should help answer a named question or support an agreed investigation.
4. Instrument components and protect the telemetry path
Instrument the entry service, orchestrator, model interactions, retrieval, custom tools, queues and external confirmations. Preserve context across processes where appropriate. Use safe task references to connect later events that are outside a single trace. Test each boundary with a known event sequence rather than relying only on automatic instrumentation.
Define field handling before exporting content. Remove unnecessary identifiers, credentials and full documents; retain version references or approved excerpts when they satisfy the investigation. The telemetry design guide provides a field-level approach. A content restriction in the user interface does not remove data already captured elsewhere.
If using an OpenTelemetry Collector, its configuration model separates receivers, processors and exporters in enabled pipelines. Document the actual path and its failure behaviour. Decide what happens when export is slow or unavailable, how buffering is bounded and how loss becomes visible. Do not let an observability outage create unlimited application memory growth.
5. Establish a baseline before tuning thresholds
Observe a representative operating cycle with known task outcomes. Record ordinary variation by task type, language, source condition and review requirement where those distinctions matter. Separate the time spent executing from time waiting for approval or an external system. A single average can conceal an ageing queue.
For DR-52, measure the interval from the register's due-work selection to admission, then execution, review waiting and confirmation. A reminder can be generated quickly yet arrive too late because the admission scheduler missed it. Monitor the start of the service journey as well as the visible model activity.
Use observed distributions and business consequence to choose thresholds with the owner. Baselines describe what happened; they do not automatically define acceptable performance. SLIs and service objectives for agents explains that distinction. Avoid copying a latency target from an unrelated product demonstration.
6. Combine automated checks with reviewed quality evidence
Some properties can be checked directly: the document exists, the register revision is current, the owner reference resolves and the external task is confirmed. Other properties may require review, such as whether the reminder clearly explains an unusual exception. Keep the source of each judgement visible.
Sample routine cases for quality review while explicitly including rare or consequential conditions. A convenient sample of easy completed tasks will miss missing owners, conflicting revisions and human rejections. Record the reviewed population and uncertainty rather than presenting the sample's score as the exact quality of all production work.
Connect failed checks and reviewer findings to a controlled test set. Track the application, instruction, source and evaluator versions involved. Outcome evaluation defines the acceptance process; monitoring consumes its results and identifies where further examination is needed.
7. Design alerts around decisions
Every actionable alert needs an affected population, an evidence link, an owner and a next decision. For an admission gap, the operator checks scheduler state and reconciles due records. For an unresolved action, they inspect the receiving system before considering another submission. For wrong-owner reminders, they contain the affected branch and investigate the mapping.
Separate urgent pages from work-queue items and trend reports. A low-volume quality signal may need investigation without waking a responder immediately. A potentially unauthorised external action may require prompt containment even when infrastructure metrics look normal. Severity should reflect consequence and response urgency.
Control duplicate alerts from one incident. Group related failures without hiding distinct affected tasks. Test the route with the actual on-duty roles and access permissions. Google's monitoring guidance emphasises actionable signals; the practical test is whether the recipient can make the required decision. An agent operating runbook should document that response.
8. Prove the monitoring with controlled failures
Before relying on the service, introduce a missing owner, stale register revision, delayed approval, worker interruption, external timeout and exporter outage in an isolated environment. Specify the expected task state, monitoring signal and operator response for each. An alert arriving is only part of acceptance; the evidence must support the right action.
For the external-timeout test, deliberately preserve uncertainty until reconciliation establishes whether the follow-up exists. Confirm that monitoring does not recommend blind resubmission. For the exporter-outage test, verify that tasks remain counted and that coverage loss is visible even when detailed traces are absent.
Record the observed results and unresolved gaps. Repeat the relevant tests after changes that affect their assumptions. This is purposeful regression coverage, not a demand to rerun every possible test after every minor edit.
9. Review monitoring when the service changes
Changes to a model, instruction, source register, tool contract or review process can alter behaviour without changing the monitoring dashboard. Add release and configuration references to the evidence so operators can compare like with like. Preserve historical definitions when a state or metric meaning changes.
Use drift and regression monitoring to distinguish new input conditions from a return of a known defect. An increased escalation rate may be correct if the service now encounters more incomplete records. Investigate the reason before treating every change as degradation.
Assign a regular review of unused alerts, unowned queues and missing evidence. Microsoft's agent lifecycle guidance includes monitoring and continued improvement; the operational requirement is a named person who can change or retire a monitor when its decision purpose no longer exists.
The production handover package
Hand over the lifecycle definition, monitoring contract, data-flow map, dashboard definitions, alert routes, runbook and controlled-failure results together. Name the owner of each monitor and the process for changing it. Operators should be able to explain an unresolved task without depending on the original developer's memory.
Discuss a production monitoring plan for your AI agents with a masked task lifecycle and the external systems that confirm completion. AgentOps services can be scoped around the evidence, monitoring setup and operating handover your service needs, with support commitments agreed separately.
FAQ
Your questions, answered.
What is the minimum monitoring needed before an AI agent goes live?
At minimum, establish admitted-task records, meaningful states, confirmed outcomes, unresolved-work visibility, execution evidence, quality checks, resource signals and telemetry-health monitoring. Give actionable alerts an owner and response route. The exact depth depends on the consequences and reversibility of the agent actions; a model-call dashboard by itself is insufficient.
How do we monitor tasks that wait for human approval?
Record the review request, responsible queue, permitted reviewer role, current status and eventual disposition. Measure waiting separately from execution time and define an escalation process for ageing work. Preserve the connection to the resumed attempt. A long wait is not automatically a model failure, but it must remain visible to the service owner.
Should every agent trace be manually reviewed?
Not necessarily. Use automated checks for properties that can be established directly and a purposeful review sample for qualities requiring judgement. Include rare, rejected and consequential conditions instead of only easy completed tasks. Document sampling limits, reviewer criteria and uncertainty. Some actions may require review individually because of the service’s own authority or risk rules.
How can we choose alert thresholds without production history?
Begin with controlled tests, explicit business waiting limits and a cautious observation period. Use the resulting distributions to refine thresholds with the service owner. Keep baseline behaviour distinct from acceptable behaviour. If evidence is sparse, record that uncertainty and use review queues or targeted checks rather than presenting an arbitrary percentage as an established norm.
What happens if the observability backend is unavailable?
The behaviour should be designed and tested. Bound buffering, expose loss or delay, and decide which evidence is mandatory for the service to continue. Keep authoritative task states available through an appropriate durable mechanism. Some actions may need to pause if required evidence cannot be recorded; others may continue with a clearly identified monitoring gap.
When should production agent monitoring be retested?
Retest the checks affected by changes to instructions, models, sources, tools, state definitions, review processes or telemetry configuration. Preserve controlled-failure fixtures so the team can verify that known defects remain detectable. Do not assume an unchanged dashboard means unchanged coverage; field meanings and event boundaries can move underneath it.