What is AgentOps?

AgentOps is the continuing work of keeping AI agents useful, controlled and recoverable after people begin relying on them. It connects ownership, observation, evaluation, permissions, change decisions and incident response to the tasks an agent is allowed to perform. Its practical question is whether the agent is still doing acceptable work under current conditions, and who acts when it is not.

For a business owner or technology leader, this means looking beyond the launch demonstration. A model can remain available while the agent uses an obsolete instruction, waits indefinitely for a reviewer or reports an action that another system never completed. AgentOps services address that operating boundary according to an agreed scope.

One term describes a discipline and a named product

In this guide, AgentOps means the operating discipline. IBM describes AgentOps as an emerging set of lifecycle practices building on earlier software and machine-learning operations. It is not a single universally mandated standard or a certification that an organisation obtains by installing a dashboard.

AgentOps.ai is also a software product and project, with its own SDK and application repository. A team can use that product as part of its operating system, or use other tools. Product comparisons elsewhere in this cluster explicitly identify the vendor. Buying an observability tool does not automatically establish business ownership, permission boundaries or a recovery service.

Start with the object being operated

An agent is more than its underlying model. Its behaviour also depends on instructions, retrieved information, memory or working state, available tools, execution logic and the identity under which actions run. A change to any of these can alter the result while the model identifier remains unchanged.

The operating unit should therefore be a named agent performing a bounded class of tasks. Record the intended users, permitted inputs, allowed actions, authoritative systems and conditions for escalation. An inventory entry called “the AI assistant” is too broad if different departments give it different permissions and acceptance rules.

Define success at the business boundary. Preparing a draft, obtaining approval, submitting a change and confirming that change are different outcomes. If the agent only prepares drafts, measure whether the drafts are usable. If it performs an authorised action, its operating evidence must reach the destination state that matters.

A concrete operating example

Consider an illustrative records-management agent that prepares proposed disposal batches for a records officer. It checks document categories, the organisation's approved retention schedule and active hold flags. It may propose a batch; it cannot decide a legal hold or delete records. The example describes an internal control design, not a statement of any organisation's legal retention duties.

Batch RB-26 is prepared using schedule revision 8. Before review, a records owner changes a hold flag on one included folder. A fluent explanation and a successful model call do not make the old proposal usable. The operating process must identify the affected proposal, prevent approval against stale evidence and send the changed folder to the responsible reviewer.

Illustrative AgentOps contract for a disposal proposal agent

Operating question	Evidence to retain	Accountable role
What was the permitted task?	Proposal scope, source folders and exclusions	Records service owner
Which rules and records applied?	Schedule revision and source-state references	Records policy owner
Could the agent act?	Tools restricted to reading and drafting	Application and identity owners
Was the result acceptable?	Reviewer disposition and reasons for correction	Records officer
What happens after a hold changes?	Affected-case lookup, invalidation and review route	Operating team
Who may change the workflow?	Approved release and evaluation evidence	Named change authority

The useful operating record links the proposal to its evidence and disposition. It need not copy every document into a telemetry store. A restricted reference can support investigation while keeping document access under the authoritative system's policy.

The recurring work that AgentOps must cover

Observe execution. Capture enough information to reconstruct task progress, tool interactions, waiting states and confirmed outcomes. Record missing telemetry as an evidence gap. A dashboard showing no errors may mean that the exporter stopped, not that the service became perfect.

Evaluate usefulness. Check representative completed work against a documented standard. In the example, a proposal containing an ineligible folder is a quality problem even when every API call succeeds. Review both accepted and rejected cases; examining only complaints misses quiet errors and users who simply abandon the service.

Control actions and changes. Enforce permissions in the application and destination systems. Define which configuration changes require evaluation and approval. An instruction telling a model to respect a hold is not a substitute for a tool boundary that prevents deletion.

Recover and improve. Contain the affected behaviour, reconcile outstanding cases and restore an accepted operating state. Improvement may mean correcting source information, repairing an integration, changing a queue or retiring the agent. It does not always mean replacing the model.

These responsibilities continue across the lifecycle. Microsoft's agent lifecycle guidance connects ownership, monitoring, improvement and retirement. The implementation should fit the organisation's risk and workload rather than become an elaborate programme for every small assistant.

Separate business accountability from technical support

The service owner defines what acceptable work means and whether the task is still worth operating. The engineering owner maintains the runtime and integrations. Security and privacy owners determine applicable access and data-handling controls. Domain reviewers resolve cases that require professional or business judgement.

One person may hold several roles in a small organisation, but the decisions still need names. A supplier can monitor and recover a service under contract without acquiring authority to change the customer's records policy. Likewise, a business owner cannot make an unsupported destination action safe merely by requesting faster automation.

The human oversight operating model defines review queues and decision rights in more detail. The managed operations guide explains which duties can be contracted and which customer decisions need to remain explicit. Both help prevent a gap between “the supplier is watching it” and someone actually owning the unresolved work.

Choose measures that distinguish progress from value

For the records example, useful measures include proposals requiring correction, proposals blocked by changed source state, reviewer waiting time and the proportion of proposals with complete evidence. These tell different stories. A rising hold-related block rate may show that a control is working, while a growing unresolved queue may show that the operating capacity is inadequate.

Do not combine those states into one unexplained success percentage. Define the included tasks, time window, exclusions and acceptance rule. Keep operational availability separate from answer quality and business acceptance. A reviewer can correctly reject a proposal; that should not be relabelled as an accepted proposal to improve a chart.

Cost measures should include the resources required to achieve the chosen result. Low model spend can coexist with expensive manual correction. Start with the outcome evaluation method before deciding which score belongs on an executive dashboard.

How much operating structure does a small agent need?

A narrow read-only assistant may begin with a named owner, defined information sources, a modest evaluation set, basic monitoring, a contact route and a documented way to disable access. An agent that changes customer records or coordinates several systems needs stronger confirmation, intervention and reconciliation arrangements.

Scale the design by the consequence and reversibility of an error, the volume of tasks, the sensitivity of information and the time available to detect a problem. Counting agents alone is a weak sizing method: one high-volume agent with broad actions can demand more operating work than many bounded assistants.

Use the readiness checklist to test evidence before accepting live responsibility. The checklist should expose missing owners and recovery paths, not become a ritual in which every box is marked complete because the application starts.

Decide how the agent stops or changes mode

Continued operation is a decision, not a default entitlement. Define the conditions under which the agent becomes read-only, pauses new tasks or is retired. For the records example, loss of a current schedule reference could suspend new proposals while allowing a records officer to inspect existing ones. That is a specific degraded mode, not an unexplained outage.

A pause must also address work already in progress. Record which proposals remain valid, which need fresh evidence and which users should receive an update. Removing the user interface alone can leave background jobs running or unresolved cases hidden from reviewers. The operating owner needs an inventory of those effects.

Retirement should remove unnecessary permissions and scheduled triggers, preserve records that remain legitimately required and identify the replacement process. Otherwise a nominally retired agent can continue generating cost or maintaining access. Include the criteria for stopping in the operating agreement while the team still understands the workflow.

Start an operating review with a real case

Bring one task that was accepted, one that required intervention and one that remains uncertain. Follow each from request to final disposition. Identify the evidence available to the operator, the action they are authorised to take and any person or system needed to finish the work.

That small review often reveals whether the first priority is tracing, outcome definition, source maintenance or responsibility. The comparison of AgentOps, MLOps, LLMOps and AIOps helps place the work alongside existing teams without creating duplicate ownership.

Discuss the operating needs of your AI agents with the agent's task boundary, current owner, permitted actions and a masked example of a case that is difficult to resolve.