AI Agent Testing and Evaluation

AI agent evaluation should establish whether the system completes its supported task, respects constraints and leaves external systems in the correct state. Grading the final answer alone is insufficient when an agent retrieves restricted information, calls tools, changes records or pauses for approval.

This guide gives product owners, quality leads and engineers a practical evaluation design. The worked example is an illustrative service information assistant that explains published application requirements from approved sources. It can identify missing information and route unresolved questions; it does not make the authority's final eligibility decision. Every dataset and count below is a proposed test design, not a report of completed testing.

Define the task, trial and outcome

A task is an evaluation case with initial conditions and expected behaviour. A trial is one execution of that case. The trace records relevant model and tool activity, while the outcome includes what the user received and what changed in the environment.

Anthropic's evaluation guidance distinguishes tasks, repeated trials, execution traces and outcomes, and discusses different grading methods. The distinction is useful because an acceptable-looking response can conceal an incorrect action. See the primary evaluation discussion.

Write the acceptance contract before inspecting the latest output. Otherwise, reviewers may unconsciously redefine success to match the demonstration. A case should state the authorised user, available sources, relevant source versions, initial system state, expected disposition and prohibited behaviours.

Start from failures that matter to the workflow

For the service information assistant, important failures include citing an inapplicable document, treating a draft as published policy, inventing a missing requirement, exposing a restricted source and claiming that an application was accepted. Polished prose is secondary to those obligations.

Also include useful uncertainty. If the correct source is unavailable, a clear limitation and approved handoff can be the right outcome. If the user omits the context needed to select a policy, a targeted question can be correct. Do not score every clarification or escalation as a failure simply because the system did not answer immediately.

Separate critical failures from quality gradations. An unauthorised disclosure should not disappear inside an average that also rewards tone and formatting. Define release-blocking conditions with the responsible business and security owners, proportionate to the intended use.

A worked evaluation set

Suppose the team proposes 120 distinct tasks, balanced as 60 English and 60 Arabic tasks. The categories below describe the primary challenge; language is a separate dimension. The numbers are illustrative and must be adapted to the real workload and consequences.

Illustrative service information evaluation set

Primary challenge	Tasks	Expected evidence
Routine supported questions	40	Correct applicable facts with supporting sources
Missing user context	30	Targeted clarification without unsupported determination
Conflicting or superseded sources	20	Correct version selection or explicit unresolved conflict
Restricted-data requests	20	Enforced access boundary and appropriate response
Unavailable source service	10	Truthful limitation and defined fallback

Running each task 3 times produces 120 × 3 = 360 trials. Repetition helps inspect variability, but 360 executions of 120 tasks are not 360 independent business scenarios. Preserve the task grouping when interpreting results and do not present repeated runs as broader coverage than they provide.

Balance languages within meaningful categories where feasible. If Arabic cases are mostly easy questions while English cases contain the difficult exceptions, the language comparison is misleading. Similarly, a deliberately difficult challenge set should not be presented as a direct estimate of ordinary production failure rates.

Keep development cases separate from release evidence

Use development cases to diagnose and improve the system. Maintain a held-out assessment set that is not repeatedly exposed during tuning. When a held-out case becomes part of routine debugging, recognise that it no longer offers the same independence and refresh the assessment appropriately.

Document how cases were selected. Production-derived samples, expert-designed edge cases and synthetic variations serve different purposes. A synthetic case can test a specific control, but its existence does not prove how often that condition occurs in customer traffic.

Protect sensitive data in the evaluation environment. Preserve the structural relationships needed for the test while removing unnecessary personal or commercial details. Verify that redaction does not accidentally remove the ambiguity or permission boundary the case is intended to test.

Use the right grader for each obligation

Use deterministic checks where the condition is exact: schema validity, required fields, allowed tool names, identity scope and destination state. A language-model grader should not be the sole authority for whether a prohibited tool was called or a transaction was duplicated.

Use a clear semantic rubric for meaning: factual support, preservation of conditions, completeness and whether clarification addresses the actual missing information. Provide examples of acceptable and unacceptable interpretations. Allow different wording when the meaning and behaviour are correct.

Use qualified human review for consequential domain judgement and language nuances that automated grading has not demonstrated it can assess. An automated judge can assist with scale, but it needs calibration against relevant human decisions and periodic checks when the evaluated system changes.

A proposed answer rubric

• Evidence: Each consequential factual claim is supported by an applicable approved source.
• Scope: The answer distinguishes information from a final eligibility or approval decision.
• Uncertainty: Missing or conflicting evidence remains visible.
• Language: Conditions, negation, dates and identifiers preserve their intended meaning.
• Usefulness: The user receives an actionable next step without an unsupported promise.

Record the reason for a failed grade, not just a score. “Wrong policy version” points towards retrieval or source metadata; “correct source, altered exception” points towards interpretation or generation. Those distinctions make evaluation a debugging tool rather than a leaderboard.

Inspect the trajectory without prescribing one exact path

Some cases permit several valid tool sequences. Grade the required constraints and outcome rather than forcing the agent to imitate one reference trace. At the same time, inspect unauthorised access, unnecessary repeated calls and missing approval transitions even when the final answer is correct.

For write-capable agents, inspect the receiving system. A final message claiming completion does not prove that the correct record exists. Test duplicate-event handling, timeouts after submission, cancellation and resumption. The orchestration guide explains the states those tests must examine.

Use simulated tools for fast, controlled cases where appropriate, then verify the relevant contracts against the intended integration environment. A simulator that always returns clean results can hide the very errors that production introduces. Keep the simulator's limitations visible in the evidence report.

Report denominators that match the question

Task success should specify which assigned tasks are counted and what disposition qualifies as success. Report answer quality among answered cases separately from coverage across all assigned cases. Otherwise, a system can appear more accurate simply by declining difficult work.

Report critical failures as their own counts and case descriptions. Also report clarification, escalation, abandonment and unresolved technical outcomes where relevant. A correct refusal of restricted content is a successful control outcome, while an unnecessary refusal of supported work is a usability failure.

Measure review effort, latency and resource use alongside quality. Include retries and failed attempts. A design that improves answer quality while substantially increasing human correction or waiting may require a different operating boundary. The ROI framework connects these operational measures to a later business assessment.

Compare releases under controlled conditions

Record the model configuration, instructions, tool schemas, retrieval settings, source snapshot and application version. Keep relevant conditions comparable when assessing a change. If both the source corpus and model changed, avoid attributing every difference to the model alone.

Compare against a useful baseline: the current manual process, a fixed workflow or the previous release. Use cases that match the question being asked. An agent-versus-agent comparison does not answer whether an ordinary search interface would meet the business need.

Inspect regressions by category. A change may improve routine answers while weakening Arabic clarification or restricted-source handling. The Arabic-native guide details language-specific checks, and guardrails covers enforcement tests.

Calibrate review before trusting aggregate scores

Have reviewers independently assess a common sample, then discuss disagreements using the rubric and sources. Distinguish uncertainty in the case from inconsistency in grading. Revise ambiguous criteria before treating small score differences as meaningful.

Where practical, conceal the candidate identity during comparative review so the evaluator focuses on the output rather than a preferred vendor or model. Preserve the order and context needed to understand a conversation, but avoid unnecessary cues that bias judgement.

A perfect result on a finite suite does not establish that failures are impossible. State the tested scope, sample limitations and unresolved categories. The release decision should combine the evidence with the consequence of failure and the operating controls, rather than presenting a test score as a guarantee.

Continue evaluation after release

Production monitoring can identify new failure patterns, changing source quality and unexpected user behaviour. Turn confirmed incidents into regression cases after appropriate data handling. Keep new assessment cases separate where needed so the test set does not become only a memory of previously fixed mistakes.

Online availability and good latency do not prove task quality. Use targeted review and outcome evidence alongside telemetry. The monitoring guide explains that operating layer.

Open for AI can scope agent development with an evaluation plan tied to the actual workflow. To discuss acceptance evidence for your agent, bring representative inputs, examples of unacceptable outcomes and the person authorised to decide what correct completion means. That is the foundation of a useful test suite.