Pilot-to-production gap
AI Agent Pilot vs Production: Operating Gap
Expose hidden pilot repairs and build repeatable operating capacity with an intervention ledger and receiver-led workload exercise.
A successful agent pilot establishes that a task can work under the conditions of that pilot. Production requires something different: a repeatable service that can handle its declared workload, exceptions, changes and absences with named operating responsibility. The gap often sits in work performed around the agent rather than in the model itself.
This guide is for sponsors deciding whether an agent is ready to leave a closely supported experiment. It explains how to expose hidden assistance, define the operating capacity needed and test transferability. The aim within AgentOps is to turn a promising result into an explicit service, including the human work that should remain.
Find the work that made the pilot look effortless
Ask who prepared the inputs, repaired records, selected examples, reset failed runs, interpreted ambiguous outputs and contacted system owners. These activities may have been reasonable during experimentation. They become a production problem when the plan assumes they will disappear without a design or an owner.
Separate intended human decisions from hidden engineering assistance. A qualified reviewer accepting an uncertain result can be part of the approved operating model. A developer silently repairing filenames before each run is an undeclared dependency if production operators cannot perform or replace that work.
Google's SRE workbook discussion of toil provides a useful lens for identifying repetitive operational work. Do not import a staffing percentage as a universal rule. Measure the actual assistance surrounding this service and decide which work should be engineered away, bounded or explicitly staffed.
Illustrative pilot: laboratory-equipment service-record indexing
Consider an administrative agent that indexes historical service documents for laboratory equipment. It extracts document identifiers, dates and equipment references into draft records for review. It does not determine calibration validity, equipment safety, clinical suitability or whether a device may return to service.
The pilot uses 24 selected documents. Six documents receive undeclared developer help before their records are accepted. Across those six documents, the developer performs four filename repairs, three manual equipment-record associations and two reruns. Some documents receive more than one intervention, so nine intervention events are not nine separate affected documents.
The sponsor sees accepted records and a short demonstration. Operations needs to know which results depended on repair, whether the ambiguity can be detected automatically and who will handle it after transfer. The pilot may still be valuable, but its output alone does not establish unattended operating capacity.
| Hidden work | Observed events | Production decision |
|---|---|---|
| Filename repair. | Four corrections across the selected cohort. | Validate input naming or create an owned exception path. |
| Equipment-record association. | Three manual associations. | Define evidence and qualified review; do not infer safety status. |
| Rerun after repair. | Two reruns. | Preserve task identity and verify whether any destination action already occurred. |
| Final record acceptance. | Review of the draft result. | Keep intended acceptance separate from undeclared engineering rescue. |
The counts describe this illustrative selected cohort. They are not a measured market benchmark or a reliable estimate of the failure rate for all future documents. The immediate lesson is the existence and type of assistance, not a population percentage.
Translate the demonstration into an operating contract
Write the eligible inputs, supported document conditions, output definition, review requirement and excluded decisions. Specify the systems that supply equipment identities and the owner of ambiguous associations. A document may be indexed successfully while still lacking evidence required for a separate maintenance decision.
Define task states that survive interruption: received, awaiting input correction, draft prepared, awaiting review, accepted and unresolved where applicable. Do not collapse them into one completion flag. The outcome evaluation guide helps distinguish an accepted administrative record from a model response that merely looks plausible.
State the operating calendar and what happens when a required reviewer or source is unavailable. Production needs a safe waiting state and an accountable next action. “The developer will sort it out” is a temporary arrangement only when that work and its availability are explicitly included.
Test a representative workload without undeclared rescue
Choose a controlled evaluation set that includes the variation expected in scope: naming differences, scans of differing quality, incomplete identifiers and documents that should be held. Use appropriate access and protect sensitive details. Explain how the cases were selected and what the sample cannot establish.
Ask the receiving operating roles to process the cohort using only the documented controls and intended review steps. Developers can observe and protect the exercise, but any extra intervention should be logged rather than silently incorporated into the result. The purpose is to learn where the proposed operating model stops.
Compare the resulting dispositions and effort with the pilot's claimed behaviour. A lower completion count may be more honest and safer if previously hidden ambiguity now reaches review. Investigate the cause rather than pressuring operators to reproduce the demonstration's appearance.
Build capacity from the real work mix
Capacity depends on arrival patterns, routine processing, exception complexity, reviewer availability, source delays and recovery work. Measure each separately enough to identify the bottleneck. Increasing model throughput will not resolve a queue waiting for qualified equipment-record association.
Use observed effort as an input to planning, with explicit assumptions and uncertainty. A small curated pilot may underrepresent damaged documents, repeated submissions or peak arrival periods. Test sensitivity to a higher exception workload rather than treating the pilot average as a guaranteed production rate.
Decide how overload is handled. Options include limiting intake, prioritising agreed categories, extending a declared service window or adding qualified capacity. Expanding the agent's authority to clear the queue changes the operating contract and requires a separate decision.
Expose support dependencies and key-person risk
List the knowledge held by the developer: source quirks, undocumented mappings, diagnostic queries, repair steps and contacts. Convert it into verified procedures or named specialist responsibilities. Do not assume that a recorded walkthrough makes every tacit judgement transferable.
Run an exercise in which the original developer is unavailable for routine decisions. The receiving team should identify the state, locate evidence, use the runbook and reach the right domain owner. Record where the process waits and whether that wait is acceptable.
Use the handover process to demonstrate transfer, and the on-call design to cover urgent conditions that cannot wait for normal support. The desired result is a service that knows its limits and owners, not one that appears independent because assistance is unrecorded.
Decide which gaps need engineering and which need an operating choice
Repeated filename repairs may justify an input validator or an upstream standard. Ambiguous equipment identity may require human review because the available evidence genuinely cannot support a unique association. A missing escalation contact is an ownership gap. These problems should not all enter the backlog as “improve the prompt”.
For each gap, record the observed condition, consequence, proposed treatment, owner and proof of closure. A technical change closes the gap only when the receiving team can use it and the resulting behaviour is verified. A staffed process closes it only when capacity and authority are available under the declared service conditions.
The readiness checklist can then decide acceptance, restriction or hold. A narrowed launch may be sensible if unsupported document types are detectably excluded and the remaining service is useful. An exclusion that nobody can recognise at intake is not a reliable operating boundary.
Test changes in the source relationships
A pilot often uses a fixed collection of documents and equipment records. Production may receive a revised equipment register, renamed files or a document that refers to an older identifier. Test how the service discovers that change and whether it preserves the distinction between historical reference and current association.
In the indexing example, a filename that resembles an equipment code is a clue, not definitive evidence of identity. If the document body and the approved register disagree, the agent should follow the defined exception path. A developer's remembered mapping must not become an invisible production rule. Record the evidence that supports any manual association and keep it separate from a decision about equipment condition.
Also test whether corrections change previously prepared drafts. State which records are reopened, which remain accepted under their original evidence and who decides when a later source revision matters. This prevents a static pilot success from concealing the ongoing work of maintaining associations as the source estate changes. It also makes clear which responsibilities remain with the owner of the equipment register.
Keep the production evidence comparable to the pilot
Retain the pilot's task definitions, selection method and intervention ledger. When production results differ, compare the workload and operating conditions before concluding that the model deteriorated. A broader input population or a missing developer repair can explain a change without any model update.
Report intended human review, extra repair work and unresolved tasks separately. If an operator spends time correcting the evidence, include that work in the service's operating account. A pilot-to-production comparison that counts only model calls can make a labour-intensive service appear more autonomous than it is.
Discuss the operating gap in an agent pilot with the pilot cohort, intervention history, proposed production workload and receiving-team responsibilities. These materials support a concrete decision about what must change before the experiment becomes a repeatable service.
FAQ
Your questions, answered.
Why does a successful agent pilot struggle in production?
The production workload, operating hours, input variation and support arrangement may differ from the pilot. Hidden preparation and developer repairs can also disappear at transfer. Compare the actual conditions and intervention history before blaming the model. Production readiness requires a repeatable operating contract, named exception ownership and evidence that the receiving team can sustain the service.
How do we identify hidden support in an agent pilot?
Record preparation, corrections, reruns, manual associations, interpretation and escalation performed around each task. Distinguish intervention events from affected tasks because one task may receive several actions. Separate intended human review from undeclared engineering rescue. The ledger should explain what assistance occurred, why it was needed and who could perform it under the proposed production arrangement.
Is human review evidence that the pilot failed?
No. Human review can be an intentional and valuable part of the approved service. The problem is assuming that necessary work will disappear or omitting it from capacity and ownership. Define the reviewer decision, evidence and availability, then test the complete human–agent process. Greater autonomy is not automatically the correct production objective.
Can we use the pilot exception rate to forecast production staffing?
Use it cautiously as one input, with the selection method and uncertainty visible. A curated pilot may exclude difficult inputs, peaks or repeated submissions, so its observed rate may not represent the production population. Test a representative workload and examine sensitivity to exception volume and duration before committing capacity or service expectations.
What should a pilot-to-production exercise remove?
Remove undeclared assistance while preserving the intended controls and human review. Let the receiving roles use documented procedures and actual authorised access. Developers may observe and protect the exercise, but log extra help explicitly. The purpose is to expose where the operating model stops, not to force every task through without a legitimate review or hold.
When is a narrower production launch reasonable?
When the supported scope is useful, excluded inputs can be recognised and the restriction is enforceable. Define the receiving owner, exception path and evidence needed before expansion. A narrow launch is not credible if unsupported cases enter indistinguishably or if operators must improvise hidden repairs. Use a readiness decision to accept, restrict or hold the service.