How to Evaluate an AI Agent Before You Buy

A supplier demonstration can show that an agent performs a prepared scenario. It cannot, on its own, show that the agent will handle your ordinary work, respect your permissions or recover when one of your systems fails. A purchasing decision needs evidence tied to the workload you intend to buy.

The aim is not to embarrass suppliers with impossible puzzles. Give them a clear task, fair access to representative development examples and the same evaluation conditions. Then reserve a separate set of cases and inspect the business outcomes. This guide provides an original buyer-led exercise for a supplier-document exception assistant.

Write the task and the forbidden outcomes first

In the illustrative use case, the assistant reviews a supplier’s document submission against an approved checklist, identifies missing or conflicting evidence and prepares a follow-up case. It may read authorised records and draft a request. It may not approve the supplier, alter bank details, waive a requirement or send a message to an unverified recipient.

Define what a correct result contains: the right supplier identity, applicable checklist version, item-level evidence, unresolved questions and a correctly routed case. A fluent paragraph is insufficient if it refers to the wrong entity or treats an expired document as current. State the conditions under which clarification or handoff is the correct outcome.

Record the current baseline too. How long does a person take, which errors occur and what volume is eligible? Without a baseline, a good-looking result cannot establish whether the purchase improves the service. The requirements checklist helps turn these expectations into inspectable requirements.

Ask what you are actually evaluating

Identify the exact product or proposed implementation, model configuration, tools, source set, environment and permitted actions. Ask which capabilities are available now, which are configured for the exercise and which require future development. A mock interface can be useful for discussing a process, but it should not be presented as proof of a working integration.

Request a diagram of the data and action paths, including the system of record and any external processing. Identify your own responsibilities: source approval, test access, reviewer availability and internal security decisions. Keep a written list of assumptions so a later success is not attributed to a capability that the buyer supplied manually.

The NIST AI RMF Playbook’s Measure guidance links evaluation methods to the purpose, context and relevant risks of the system. In a buying exercise, that means the supplier’s general benchmark is supplementary evidence; your task and operating conditions remain the main test.

Build a buyer-owned case pack

The following 60-case pack is a proposed starting exercise, not a statistically validated benchmark or a required sample size. Use appropriately approved redacted records or realistic synthetic cases. Keep development examples separate from the final purchasing set, and document how the set represents your workload.

Illustrative 60-case purchasing exercise

Case family	Count	What it reveals
Ordinary supported submissions	30	Correct identity, checklist application, evidence extraction and case preparation
Missing or ambiguous information	10	Useful clarification, preserved uncertainty and appropriate handoff
Tool and source failures	10	Timeout handling, stale sources, partial results and duplicate-effect prevention
Permission and hostile-content cases	10	Access boundaries, untrusted document instructions and unauthorised actions

For this bilingual example, allocate 30 cases to Arabic, including relevant mixed-language inputs, and 30 to English across the four families. This balanced exercise is an assumption for inspection, not a claim about real traffic proportions. Use the actual language and document distribution for the operational forecast, and add cases where a rare error would have a serious consequence.

Each case needs an expected outcome, required evidence, permissible alternative and severity definition. Some cases can have several acceptable wordings but only one correct supplier identity. Have business reviewers agree the rubric before scoring; otherwise disagreement about policy may be mistaken for model error.

Keep the final cases separate from tuning

Suppliers need examples to configure and understand the task. They should not receive the final answers to the same cases used to claim independent performance. Set a configuration freeze for the purchasing run and record any change made afterwards.

If a supplier repairs a failure, retesting that case demonstrates the repair. It does not make the original case unseen again. Keep the original result, inspect related regression cases and use additional held-out evidence where needed. Apply the same rules to all suppliers so one is not rewarded for receiving more tuning opportunities.

For variable behaviour, repeat selected cases under the agreed conditions and report the variation. Do not choose the best of several attempts without disclosing the selection. A one-off successful answer is weaker evidence than consistent handling of the same consequential boundary.

Inspect effects, not just answers

For each task, ask the evaluator to see the source evidence and the resulting business record. If the assistant says it created a follow-up, verify the case and its recipient. If it says access was denied, verify that protected content did not enter the output or an unintended tool request. If it reports a timeout, inspect whether an action nevertheless occurred downstream.

Use a controlled test environment for consequential operations. A procurement exercise should not need uncontrolled production writes to prove that the design can recognise a missing certificate. Define who may run the exercise and how evidence is captured and handled.

The audit-trail guide explains the request-to-effect chain. A supplier should be able to show enough of that chain for evaluation without exposing secrets, unrelated customer data or hidden internal model reasoning.

Use a scorecard that preserves serious failures

Report task acceptance by case family and language, with counts as well as percentages. Separate correct completion, correct clarification, correct handoff, incorrect result and unauthorised action. A handoff can be the right behaviour but still leave work with your team, so do not hide it inside an undifferentiated success rate.

Agree non-negotiable gates before the run. For example, an unauthorised disclosure or prohibited business action may require remediation and a new decision regardless of the overall score. Zero observed incidents in a small pack does not prove that the risk is absent in production. Record the remaining uncertainty and the additional assurance required.

For a narrower quality measure, use a clear denominator. “28 of 30 ordinary cases accepted” is interpretable. “93% accurate” without the population, rubric or failure consequences is not. Do not compare one supplier’s document-level metric with another’s field-level metric as though they measure the same thing.

Measure the work that returns to your staff

Have a representative employee review the result using the intended interface. Record handling time, corrections, systems opened and whether the evidence is sufficient to decide. A response that saves drafting time but requires a long verification process may produce little net value.

Test the handoff too. Can the receiving team understand the missing item, source conflict and requested next step without re-reading the entire conversation? Can it reject or correct a proposal? The oversight model covers capacity and decision quality, while the ROI framework converts measured net effort into an explicit value hypothesis.

Request cost and ownership evidence with the result

Record technical consumption, attempts, retries, processing time and human effort for the test. Ask which costs are included in the quoted price and which are estimates or pass-through charges. Extrapolate carefully: the test’s deliberate failure mix may not match normal traffic, and peak capacity may introduce costs that a small run does not reveal.

Request the operational owner, support boundary, model and connector change process, handover artifacts and exit arrangements. A successful proof that depends on a single supplier engineer manually intervening is not yet a sustainable service. Ask the engineer to make those interventions visible rather than presenting them as autonomous behaviour.

Use the TCO model to compare the full responsibility over a consistent horizon. Keep purchase acceptance distinct from a long-term operating guarantee; the contract must describe both if both are being bought.

Turn the evidence into one of four decisions

• Proceed within the tested boundary: the task is supported, material gates are satisfied and operating responsibilities are acceptable.
• Proceed with a narrower scope: a useful subset works, while unsupported cases remain in an explicit manual path.
• Run a targeted investigation: a specific unresolved dependency could change the decision and has a bounded way to test it.
• Do not purchase this approach: the remaining gap, risk, cost or retained workload makes the proposal unsuitable.

Write the decision with the tested release, case pack, results, exclusions, owner and next review trigger. A limited exercise should lead to a limited claim. The RFP template provides a place to specify this evidence before suppliers prepare their proposals.

For an Open for AI agent project, bring a representative redacted task and the outcome your team would accept. Discuss a scoped proof and its acceptance evidence before committing to broader authority or rollout.