An experiment with a decision
AI Proof of Concept Design
Design an AI PoC with a testable hypothesis, representative cases, credible baseline and decision rules that make positive and negative results useful.
An AI proof of concept should reduce a decision-critical uncertainty. A polished demonstration can show that a system sometimes works, while leaving unanswered whether it performs the intended task better than the current process, at an acceptable cost and within the required boundaries. Good experimental design makes a negative result useful as well as a positive one.
This guide is for a sponsor, product owner or evaluation lead designing a bounded PoC. Its output is an experiment brief, a defensible comparison and a decision about the next commitment. It is not a substitute for a full production acceptance review.
Choose one uncertainty worth testing
The Australian Digital Transformation Agency describes an AI PoC as a focused experiment concerning feasibility and potential value. Use that framing to avoid testing an entire transformation programme at once. The detailed experiment below is an original planning method.
Ask what result would change the decision. Can the system interpret the relevant Arabic documents? Can users verify its output quickly enough? Is a required data source accessible under the intended arrangement? If none of the possible results would alter the sponsor's plan, the activity is a demonstration rather than a useful experiment.
Resolve obvious prerequisites first through the readiness assessment. If the business purpose, lawful access or basic operating option is still unclear, use a feasibility study to decide whether an experiment is worth funding.
Write an experiment brief before configuring the solution
| Field | Decision to record |
|---|---|
| Question | The uncertainty and the business decision it affects |
| Task and population | Users, document types, languages and cases the result will represent |
| Hypothesis | Expected improvement and quality or safety conditions that must also hold |
| Comparators | Current process and any credible simpler option |
| Evaluation material | Authorised examples, reference answers and held-back cases |
| Measures | Quality, review effort, total task time, cost and consequential errors |
| Boundaries | Allowed information, integrations, users and actions |
| Decision rules | Proceed, revise, stop or remain inconclusive, with reasons |
| Resources and authority | Budget limit, time boundary, owners and approval route |
Define success in a sentence that includes constraints: “Reduce verified extraction effort while meeting the agreed accuracy requirement and making every uncertain date visible to a reviewer.” This prevents the team from declaring success on speed while ignoring important errors.
Do not move the criteria after seeing results without recording the change and its reason. A justified change may require a new test. Preserve the original question so reviewers can distinguish learning from moving the goalposts.
Construct an evaluation set that represents the decision
Collect authorised material covering ordinary cases and relevant difficult cases. For a document task, differences might include language, scan quality, layout, missing fields and conflicting versions. Record which conditions are represented and which are excluded. A convenient folder of clean documents may be useful for initial setup but insufficient for the intended conclusion.
Create reference answers through qualified domain review. Where experts disagree, resolve the rule or label the case as ambiguous. The system cannot be assessed fairly against an answer key whose meaning is unsettled. Preserve the basis for material labels so later reviewers can inspect them.
Separate material used to improve the solution from material used for the final assessment. If the team repeatedly tunes against every evaluation case, the final result no longer provides the same evidence about unfamiliar work. Keep suitable held-back cases and record any exposure or reuse.
Synthetic examples can help test specified edge conditions, while historical material can reveal realistic variation. State what each contributes. Do not claim representativeness merely because an example looks plausible, and do not use confidential records without the appropriate approved arrangement.
Compare the complete task with a credible baseline
Measure the current process using the same definition of completion. If the AI result includes review, include review in the baseline comparison as well. Record preparation, correction and exception time where they are part of the work. Otherwise the comparison may reward shifting effort to an unmeasured person.
Consider a simpler alternative, such as improved source structure or deterministic extraction, when it plausibly solves the task. The experiment should inform whether AI is the appropriate approach. It should not be designed so that only the proposed technology is allowed to win.
Control obvious sources of distortion. If users perform the same case twice, learning may make the second attempt faster. Use comparable cases or an appropriate order arrangement and record the limitation. If one group contains only experts, do not generalise its review effort to new users without further evidence.
A worked example: extracting equipment warranty dates
A fictional maintenance business tests whether an assistant can extract warranty expiry information from equipment documents. It uses eighty authorised cases with reviewed reference answers. The current process averages four minutes per case. The assisted process averages one minute to prepare a result and one and a half minutes for verification.
The illustrative total is 80 × 4 = 320 minutes for the baseline and 80 × 2.5 = 200 minutes for assisted work, a difference of 120 minutes across the test. This is task-time evidence under the tested conditions, not an annual saving or a headcount reduction.
Quality changes the decision. Suppose the baseline produces seventy-eight correct accepted records and the assisted process seventy-seven. One assisted error confuses an installation date with an expiry date, a failure the team had classified as consequential. Faster processing does not override that condition.
The next decision could be to revise how uncertain dates are presented, restrict the supported document types or stop the approach. Retesting should use suitable fresh cases and preserve the original failure. If the sponsor changes the question from full extraction to draft assistance, record the narrower hypothesis rather than claiming the original one passed.
Record costs and uncertainty without overclaiming
Track the resources used by the experiment separately from an estimate of future operating costs. Prototype support may be unusually intensive, while production may introduce costs absent from the test. Identify the assumptions that require further evidence.
Use the ROI framework when translating observed task outcomes into a business case. Extrapolate only after considering volume, adoption, review capacity and exceptions. A small PoC should make assumptions visible rather than create a precise-looking return from uncertain inputs.
Where several suppliers participate, give them comparable task definitions, permitted preparation and evaluation conditions. The vendor scorecard helps organise the resulting comparison. Complete proportionate supplier diligence before sharing information or granting access.
Finish with a decision, including an inconclusive one
Report the hypothesis, configuration, sample, comparator, measures, failures, exclusions and actual resource use. State whether the evidence supports proceeding, revising, stopping or conducting a different test. Inconclusive is a legitimate result when the evidence cannot distinguish the options.
Limit further experimentation to a clear remaining uncertainty. Repeatedly extending a PoC without a decision rule can become an unfunded production service or an endless search for favourable examples. A time or budget boundary should prompt a conscious choice about the value of additional evidence.
If the result supports a pilot, define the new operational question and use the pilot-to-production playbook for the later transition. Do not reuse an experimental success statement as a blanket release approval.
Open for AI's agentic AI consulting can help define boundaries, evaluation and delivery sequence. Bring a redacted task description, representative examples and the decision you need to make to discuss an AI proof of concept that can change the decision.
FAQ
Your questions, answered.
How many examples should an AI proof of concept test?
Choose enough relevant variation to answer the defined question, and state what the sample cannot establish. Cover consequential edge cases as well as ordinary work. A fixed number copied from another project is not a design method. If a statistical conclusion is needed, plan the sampling and analysis with suitable expertise.
Can we run an AI PoC without production data?
Yes, when authorised synthetic, public or appropriately prepared material can answer the chosen question. Explain the limits: such material may not represent real document quality, user variation or operating exceptions. A later stage may need additional approved evidence before the organisation can make a production decision.
What makes a fair baseline for an AI experiment?
Use the current process or a credible alternative with the same definition of completed, quality-checked work. Include preparation, review and correction effort consistently. Record differences in user expertise and case selection. Comparing an unreviewed AI draft with a fully verified human result would answer the wrong question.
Should success criteria be changed when the PoC misses them?
Only through a recorded decision explaining what was learned and why the new question is legitimate. Preserve the original result. A narrower use may be worth testing, but it should not be presented as proof that the original hypothesis passed. Materially changed criteria may require fresh evaluation cases.
How long should an AI PoC run?
Set a time and budget boundary from the evidence required, access constraints and available reviewers. Define what must be learned before extending it. There is no universal duration. An extension should resolve a specific remaining uncertainty, not merely provide more time to find a favourable demonstration.
Is a failed AI proof of concept wasted money?
A well-designed negative result can prevent a larger unsuitable commitment or identify a narrower useful approach. Its value depends on whether the question mattered and the evidence is trustworthy. Document the reason for stopping or changing direction so the organisation does not repeat the same experiment without learning.