AI Agent Pilot Not Delivering Value: Diagnostic

If an AI agent pilot is already running but the business cannot see value, stop treating “better answers” as the only remedy. The loss may occur before the model receives an eligible task, after a correct answer reaches a reviewer or during the manual work needed to recover exceptions. Diagnose the whole path from incoming work to a useful business outcome.

This guide is for pilot sponsors, product owners and operations managers deciding whether to fix, narrow, extend or stop an existing deployment. It provides a measured funnel and a worked cost example. All figures below are illustrative assumptions, not Open for AI customer results or a forecast for your project.

Freeze the question before changing the system

Write the value hypothesis in one sentence: for which eligible work, compared with which current process, should the agent improve which outcome? Specify the period and the cost boundary. A pilot designed to reduce preparation time cannot be judged solely by chat satisfaction; a pilot intended to improve quality cannot be evaluated only on speed.

Preserve a stable observation period and version record. If prompts, models, source sets and approval rules change every day, you may be unable to explain which configuration produced the result. Urgent defects still need correction, but record the change and separate affected cohorts rather than blending all runs into one score.

Collect the current process baseline using comparable work. Include the time spent checking, correcting and handing off, not just the fastest manual step. If the pilot receives harder cases than the baseline, adjust the comparison or state the limitation. An apparent decline can be a case-mix difference; an apparent gain can be selection of easy work.

Reconstruct the value funnel

Use a stable case identifier to connect intake, agent execution, review, downstream use and outcome. Define each transition with observable evidence. A generated answer proves production of an output. A reviewer’s acceptance proves acceptance. A completed action in the receiving process is stronger evidence of use.

Illustrative monthly pilot funnel for document preparation

Stage	Cases	Meaning
Incoming work	1,000	All requests in the observed monthly queue
Eligible and attempted	700	Requests within the defined scope and available inputs
Technically correct output	560	Output meets the task-specific evidence and accuracy criteria
Accepted by reviewer	420	Receiving employee accepts the output for the workflow
Actually used	360	Accepted output replaces the defined manual preparation step

In this example, technical correctness is 560 ÷ 700 = 80% of attempted cases. End-to-end use is 360 ÷ 1,000 = 36% of incoming work. Neither figure should replace the other. The 300 ineligible cases remain part of the queue’s operating reality, while the reasons for their exclusion determine whether expanding scope is sensible.

The 140 correct outputs that reviewers do not accept need investigation. They may be incomplete for the receiving task despite passing a narrow technical test, or reviewers may lack confidence or a usable interface. The further 60 accepted outputs that are not used point to a different gap, such as duplicate entry, timing or a broken handoff.

Look at the path, not only the final answer

Google Cloud’s agent evaluation documentation distinguishes final-response evaluation from evaluation of the tool-call trajectory. Both can help locate technical failures. Business value additionally requires evidence of review, use and the resulting workload or outcome.

For each failed case, identify the first material breakdown: wrong eligibility, missing source, wrong retrieval, incorrect reasoning, tool rejection, incomplete output, review rejection or unused result. Record secondary problems separately. Counting every symptom as an independent failure can make one source issue look like several unrelated model defects.

Sample cases from every funnel exit, including successful ones. A case that appears successful may still contain hidden manual repair. Ask the employee to show what they did after receiving the output. Observed work is more informative than an adoption survey alone.

Calculate the work that remains

For the illustrative month, assume each of the 360 used outputs removes 7 minutes of the previous manual preparation work. Gross capacity released is 360 × 7 = 2,520 minutes, or 42 hours. That is the attractive headline, but it excludes new work introduced by the pilot.

Assume every one of the 700 attempted cases requires 1.5 additional minutes of review, totalling 1,050 minutes. The 140 technically incorrect cases also require 6 additional minutes of exception recovery each, totalling 840 minutes. These recovery minutes are additional to both ordinary manual processing and the review already counted. Defining them this way avoids double counting.

Net capacity released is therefore 2,520 − 1,050 − 840 = 630 minutes, or 10.5 hours. The original manual work that remains on unused or failed cases has not vanished; the calculation only credits the step actually replaced and subtracts new effort. Use your own time observations rather than assuming the agent’s execution time equals staff time saved.

Illustrative monthly exception-economics worksheet

Component	Calculation	Result
Manual work displaced	360 used cases × 7 minutes	2,520 minutes
Additional review	700 attempts × 1.5 minutes	−1,050 minutes
Additional recovery	140 incorrect cases × 6 minutes	−840 minutes
Net capacity	2,520 − 1,050 − 840	630 minutes = 10.5 hours
Assumed capacity value	10.5 hours × SAR 120 per hour	SAR 1,260
Assumed monthly operation	Model, infrastructure and incremental support	−SAR 2,100
Difference before implementation cost	SAR 1,260 − SAR 2,100	−SAR 840

The SAR 120 hourly value is an assumption for economic analysis, not a claim that payroll falls when capacity is released. If the team cannot use those hours productively, realised value may be lower. Implementation cost and other excluded costs would also need consideration. The ROI framework separates these value categories, and the total cost of ownership guide covers the wider cost boundary.

Choose the repair by its position in the funnel

Low eligibility: inspect why work is out of scope. If most requests lack a required identifier, repair intake or the source process. Adding another model does not create the missing data. If the eligible volume is inherently small, the original business case may need to narrow or stop.

Low technical correctness: replay representative failures and distinguish source, retrieval, calculation and tool problems. Use the evaluation framework to test the specific repair. A more expensive model should earn its cost on the failing cases rather than be adopted as a general expression of confidence.

Low reviewer acceptance: compare the acceptance rubric with the actual downstream task. Reviewers may need source links, a missing field or a different output structure. They may also be rejecting correct outputs for a valid policy reason the technical test omitted. Investigate before treating every rejection as resistance to change.

Low use after acceptance: examine timing, placement and handoff. Does the output arrive after the employee has already done the work? Must it be copied into another system? Does the receiving queue have an owner? This is often a workflow problem even when the agent itself performs well.

Test a repair against the break-even requirement

In the example, 60 accepted outputs were not used. If a handoff repair caused all 60 to replace the same 7-minute step, without increasing other costs, it would add 420 minutes, or 7 hours. At the assumed SAR 120 per hour, that adds SAR 840 of capacity value.

The repair would bring the illustrative monthly difference from −SAR 840 to zero before implementation cost. It would not yet establish an attractive return. This sensitivity calculation tells the sponsor what a successful repair must achieve and prevents a modest adoption improvement from being presented as proof of a strong business case.

Test the repair on a defined cohort with unchanged outcome definitions. Measure any new review or support effort it introduces. If only part of the 60 cases becomes usable, update the calculation with the observed result. Do not assume that every accepted-but-unused case is recoverable; some may no longer be needed.

Inspect differences hidden by the average

Break results down by task family, language, source quality and consequential exception type. Arabic scanned documents may behave differently from structured English forms; short cases may have little manual time available to save. Use sufficiently sized groups and report uncertainty rather than interpreting every small difference as a stable effect.

Review latency as experienced by the employee. A short model response time can coexist with a long queue or repeated approval delay. Track the entire cycle and the user’s active effort separately. Check whether the pilot adds interruptions that are absent from a simple stopwatch estimate.

Use monitoring to connect drift, source outages and integration failures to the affected cases. Keep the diagnostic data at a level that supports investigation without unnecessarily exposing sensitive business records.

Make the next investment conditional on evidence

Write a decision record with the measured bottleneck, proposed repair, owner, cost, evaluation period and success threshold. Extend the pilot only when the extension answers a specific unresolved question. A longer pilot without a new test design can simply accumulate more of the same uncertainty.

If the blockage is unclear organisational ownership or an unapproved production path, use the broader Saudi project-failure diagnosis. This page assumes enough operation exists to measure a funnel. If that foundation is absent, resolve the delivery decision before interpreting weak usage as model failure.

For custom agent development and workflow review, bring evidence of the actual operating path. Request information about diagnosing an underperforming pilot with a redacted case log, baseline timings, review and exception effort, operating costs and a sample of accepted outputs that were never used. Those materials support a concrete fix-or-stop decision.