How to Evaluate AI Agent Outcomes

An AI agent outcome should be evaluated against the work the organisation agreed to accept, including the conditions under which it was produced. A convincing final response is insufficient when the agent selected the wrong record, ignored a required review or left an unfinished action behind. Conversely, a justified request for clarification can be the correct result.

This guide is for process owners, quality leads and engineering teams establishing an outcome contract within AgentOps. It separates business acceptance, technical completion and safe handling so that a rising dashboard score means something operationally useful.

Write the acceptance contract before choosing a score

Define the unit of work using business identity and scope. An evaluation task needs its starting state, permitted sources, allowed actions, required result and deadline where relevant. State who can accept the result and what evidence they inspect. The agent's own success flag should not decide whether its work passed.

An outcome contract for a bounded agent task

Contract element	Question to resolve	Evidence
Identity and scope	Which request, object and revision are being handled?	Authoritative identifiers and starting snapshot.
Required outcome	What must be usable when the task ends?	Destination state or approved deliverable.
Permitted route	Which data and actions may be used?	Access decision and action policy.
Uncertainty	When must the agent defer or ask?	Defined exception state and responsible owner.
Acceptance	Who checks correctness and when?	Versioned rubric and review decision.

Describe exclusions as explicitly as success. Preparing evidence is different from approving the underlying operational decision. If the task is a recommendation, acceptance must not imply that a recommended action was performed. If a task includes a write, a generated proposal and a confirmed destination change need separate checks.

A worked example: calibration evidence packs

Consider an illustrative agent that assembles calibration evidence for equipment records. It matches an equipment serial number with approved certificates, identifies missing information and prepares a review pack. It does not declare equipment fit for service or change a maintenance authorisation. Those decisions remain with the responsible technical team.

For equipment EQ-83, a certificate for EQ-38 is not acceptable evidence even when the equipment description is similar. A superseded certificate should be identified as historical. An unreadable date should trigger a defined exception, not an invented date. The review pack must preserve the original reference so a reviewer can inspect the actual document.

Suppose an illustrative evaluation includes 240 incoming requests. Forty are outside the agreed task scope and are reported separately. Among the 200 eligible requests, 160 packs are accepted without rework, 24 are accepted after correction, 10 are appropriately deferred and 6 fail. These are mutually exclusive final dispositions for that observation window.

The accepted-result rate is 184 divided by 200, or 92%. Acceptance without rework is 160 divided by 200, or 80%. If the contract recognises appropriate deferral as controlled handling, that measure is 194 divided by 200, or 97%. The figures answer different questions. None should be renamed autonomous completion, and the 40 excluded requests must remain visible.

Check the result independently of the agent's narrative

Read the destination or deliverable through an independent test path. Confirm the correct equipment identity, included certificate versions, unresolved fields and review status. Where possible, compare structured fields deterministically. Do not ask the same model to certify its own unsupported statement that the task is complete.

Keep evidence time relevant. A record inspected after another user repaired it does not prove that the agent originally produced the correct state. Capture the result at the defined acceptance boundary, then record later corrections separately. For tasks whose value emerges later, maintain provisional and mature outcome views instead of silently backfilling yesterday's score.

Anthropic's agent-evaluation guidance distinguishes tasks, repeated trials and graders, and describes evaluation of both execution and outcomes. The practical implication here is to use more than one evidence source: exact checks for identifiers, qualified review for domain meaning and operational confirmation for completed work.

Use hard gates and graded quality for different purposes

An average cannot compensate for a prohibited disclosure or an unauthorised action. Define non-compensable failures separately from quality dimensions such as clarity, completeness or reviewer effort. A beautifully written pack with the wrong serial number should fail identity checks rather than lose a few presentation points.

Evaluation dimensions for the calibration-pack example

Dimension	Suitable check	Interpretation
Equipment identity	Exact authoritative identifier match.	A mismatch blocks acceptance.
Document relevance	Certificate scope and version review.	Relevant evidence must support the stated equipment.
Missing information	Expected exception compared with output.	Correct deferral can be successful handling.
Reviewer usability	Anchored rubric and timed review sample.	Useful quality may vary without granting authority.
Action boundary	Tool and destination evidence.	No fitness-to-operate decision is permitted.

Use model-based judges only for dimensions they can assess reliably after calibration. A language model may help screen explanations for missing qualifications, but it should not replace authoritative identity checks or independently required professional judgement.

Build a representative case set and protect its meaning

Sample the actual operating population: Arabic certificates, English certificates, mixed-script serial numbers, scans, revised documents, duplicate filenames and genuinely missing evidence. Add difficult cases deliberately, but report their results separately from a production-weighted estimate. An adversarial suite is useful without being a prevalence survey.

Group related documents and request variants when splitting development and held-out evaluation data. Near-identical certificates in both sets can make generalisation look stronger than it is. Preserve a restricted reference set, record who changed expected results and explain whether a change corrects the test or relaxes the requirement.

Include tasks where a reviewer disagrees with the original label. Reference answers are operational artefacts with limitations, not unquestionable truth. A challenge process should retain both interpretations, seek the appropriate domain decision and update the case deliberately. Do not tune the agent repeatedly against unresolved labels.

Repeat trials without manufacturing extra business outcomes

Repeated trials reveal variation in tool choice, evidence selection and disposition. Keep the business task identifier distinct from the trial identifier. Ten successful runs of one easy request do not create ten independent customer outcomes. Report per-case consistency alongside aggregate trial results.

Use isolated or simulated destinations when repeated trials could create real side effects. If a replay uses captured tool responses, label that limitation: it tests behaviour against those responses, not the current live system. The drift and regression process should preserve this distinction when comparing releases.

A useful review packet shows the expected outcome, actual outcome, material evidence and the first consequential deviation. Keep irrelevant model chatter out of the review queue. Task tracing can connect the result to its attempts without requiring every reviewer to read an entire conversation history.

Measure the work transferred to people

Acceptance after correction is valuable only when its cost is understood. Record reviewer minutes, the kind of repair and whether the agent made the task easier or merely moved work into a new interface. Distinguish substantive corrections from harmless formatting edits and from decisions that the workflow always required a person to make.

Review queues also change outcomes. A correct pack waiting beyond its useful deadline may fail the service expectation even if its content is sound. Link the acceptance record with service-level measures and maintain a separate reason for lateness. Otherwise quality and timeliness failures become impossible to diagnose.

Use the same accepted-task definition when computing cost per accepted task. Changing the numerator's cost scope or the denominator's acceptance policy can produce an apparent improvement without any operating change. Version those definitions and show a comparison on a consistent basis.

Make uncertainty part of the acceptance report

Show how many cases support each conclusion and how they were selected. A result based on a small convenience sample should not carry the same decision weight as a representative review of the intended workload. Separate directly verified outcomes, sampled estimates and cases for which evidence is still missing.

For consequential failures, preserve the individual event even when the overall rate looks strong. A single equipment-identity mismatch can justify a focused control correction without claiming that every packet is unreliable. Conversely, observing no mismatch in a limited sample does not establish that the risk is impossible. The report should state the scope of what was tested and the operational consequence of the remaining uncertainty.

Give the decision owner a clear choice: accept the current bounded scope, collect further evidence, revise a control or withhold expansion. Specify the evidence that would change that decision. This turns evaluation from a recurring score-production exercise into a practical mechanism for deciding which work the service can responsibly handle.

Turn evaluation findings into an operating decision

For each failed dimension, identify a response: correct a source mapping, narrow the supported scope, change the review interface, improve instructions or suspend an action. Do not default to replacing the model when the evidence points to an identity join or an unavailable source. Assign a measurable acceptance test to the proposed fix.

Recheck the relevant cases after a change and retain a broader suite for unintended effects. Sample production outcomes under the permitted data-handling rules, then compare observed failures with test coverage. The resulting feedback loop should improve both the agent and the definition of what useful work means.

If the underlying task boundary still needs engineering, Trusted Data's enterprise agent-development guide provides related implementation context. For operating evaluation, discuss an AgentOps outcome assessment with a masked task example, its acceptance owner and the evidence that currently distinguishes accepted, deferred and failed work.