AI Audit Sampling Approach

An AI audit sample should support a stated decision about a defined population. It might estimate how often an assistant misroutes enquiries, examine whether an approval control operated, or deliberately expose a rare failure. These objectives need different selection methods. A large collection of convenient examples does not become representative because it contains many rows, and a carefully designed challenge set does not establish the frequency of failures in ordinary use.

This guide is for internal audit leads, quality managers and technical assessors who need to explain why particular cases were selected and what the results allow them to conclude. The proposed approach separates population estimation, control testing and targeted investigation. Agree those objectives through AI audit scoping before selecting records; the sample cannot repair an undefined engagement boundary.

Start with the assertion, the unit and the observation period

Write the assertion in testable language. “The assistant works reliably” is too broad. “For enquiries admitted during the agreed month, the final routing destination matched the independently adjudicated destination” identifies an outcome that can be checked. Specify whether the destination is the AI suggestion, the employee's accepted choice or the queue actually receiving the enquiry. Those are different events, and testing one does not establish the others.

Next define the sampling unit. A customer enquiry may generate several model calls, retries, employee edits and routing attempts. If the claim concerns the customer's outcome, use the unique enquiry as the primary unit and preserve its event sequence. If the claim concerns approval enforcement, the relevant unit may be an attempted restricted action. Do not divide the number of affected customers by the number of model calls; that denominator can make repeated failures appear smaller.

Set a mature observation window. An enquiry created near the cut-off may still be awaiting review, so its outcome is pending rather than correct or incorrect. Record the cut-off, the permitted completion interval and the treatment of unresolved cases. A historical sample describes the tested period and versions. It does not automatically represent a new model, a changed queue structure or a season with different demand.

Build a sampling frame you can reconcile

The sampling frame is the list from which selection occurs. Obtain it from a source that captures the stated population, then reconcile counts against an independent operational total where possible. Check duplicates, missing identifiers, cancelled tasks, failed ingestion and records removed by retention rules. An export containing only successful responses cannot support an estimate about all admitted enquiries, even when the random selection within that export is impeccable.

Keep a frame manifest with extraction time, query conditions, source system, version, record count and relevant exclusions. Record the language, channel and release identifiers needed for stratification without collecting unnecessary personal information. Preserve a protected copy or a reproducible reference according to the agreed evidence rules. The later reviewer should be able to see what was eligible before the auditor knew the selected outcomes.

Test the frame itself on a small reconciliation exercise. For example, trace records from a queue ledger into the export and trace export rows back to the ledger. Investigate whether joined tables multiply enquiries with several attachments. A perfect match on total counts is useful but insufficient: omitted records and duplicates can offset each other. Resolve material discrepancies or explicitly narrow the population claim rather than silently treating the extract as complete.

Use a selection method that matches the question

Proposed sampling choices and their permitted interpretations

Audit question	Selection approach	Interpretation and limit
How frequently did a defined error occur?	Probability selection from a reconciled population, with documented inclusion probabilities	Can support an estimate and uncertainty analysis under the stated design; excludes cases missing from the frame.
Does the control handle unusual but important conditions?	Purposeful challenge cases chosen for specific failure mechanisms	Shows what happened in those conditions; does not estimate operational prevalence.
Was a control used across releases or branches?	Structured coverage selection across the relevant contexts	Supports statements about the tested contexts; representation is not automatically statistical.
What caused a known incident?	Incident sequence and related-case investigation	Helps distinguish explanations; incident selection is biased towards known problems.
Did a correction address the original defect?	Original-case replay plus new cases selected for the corrected mechanism	Supports bounded retesting conclusions, with separate checks for recurrence and side effects.

Combining these approaches is often sensible, provided their results remain distinguishable. Report a probability sample and a targeted challenge set as separate evidence streams. Explain which decision each stream informs. If a challenge reveals a severe defect, the response may be immediate containment even when the probability sample contains no instances. Severity and frequency answer different questions.

Worked example: bilingual lost-property enquiry routing

Consider a hypothetical service centre whose assistant routes lost-property enquiries to the responsible team. The eligible monthly population contains 8,000 unique, completed enquiries: 6,000 in English and 2,000 in Arabic. The auditor wants both an overall routing-error estimate and enough Arabic cases to examine language-specific behaviour. The design therefore selects 200 enquiries randomly within each language group. These figures illustrate a method; they are not a recommended sample size or an observed customer result.

Illustrative unequal allocation and population weighting

Language stratum	Population	Random sample	Confirmed routing errors	Sample error proportion
English	6,000	200	4	2%
Arabic	2,000	200	16	8%
Total	8,000	400	20	The pooled 5% is not the population-weighted estimate.

The population weights are 75% English and 25% Arabic. Multiplying each stratum's error proportion by its population weight and adding the results gives an overall point estimate of 3.5%. Simply dividing 20 errors by 400 sampled enquiries gives 5%, which overweights Arabic enquiries because they were deliberately sampled at a higher fraction. Neither number by itself supplies an uncertainty interval. That requires a calculation appropriate to the stratified design and its assumptions.

The language difference also needs interpretation. It might reflect different enquiry complexity, transliteration, routing categories, source data quality or model behaviour. The sample establishes observed errors against the agreed adjudication rule; it does not establish a causal explanation. Examine relevant case characteristics before claiming that language alone caused the gap. Fairness assessment addresses the further questions of affected groups, meaningful comparisons and consequential differences.

The team additionally runs 36 deliberately difficult cases involving mixed-language descriptions and ambiguous location references, finding 14 failures. Those 36 cases are reported separately. Adding them to the random sample would corrupt the overall prevalence estimate because their selection probabilities do not match the operational design. Their value is diagnostic: they identify mechanisms that ordinary sampling may encounter too infrequently to explain.

Choose sample size from the decision, not a familiar number

A request for “100 records per system” skips the central questions. What error rate matters for the decision? How precise must the estimate be? Is the objective to detect at least one instance of a specified defect, compare subgroups, or test whether an operating control was consistently performed? How heterogeneous is the population? Are outcomes sufficiently independent? What resources are required for reliable human adjudication? A larger sample with weak labels can provide a precise-looking answer to the wrong question.

For statistical estimation, ask a suitably competent analyst to document the confidence method, anticipated proportion, desired precision, finite population treatment, stratification and any clustering. Preserve the assumptions before seeing the final result. If the audit instead uses judgemental selection, describe why the selected cases are informative and avoid statistical confidence language. The budget may constrain the conclusion; it should not be disguised as a statistical justification.

Rare harms often require a different strategy. A random sample may provide useful ordinary-use evidence while having little chance of encountering a highly consequential edge case. Pair it with bounded challenge tests and evidence about preventive controls. Do not promise that a larger random sample alone will establish safety. The coverage and limitations record should show important scenarios that were not tested and the reasons for that omission.

What zero failures can and cannot mean

Suppose a separate illustrative test observes zero failures in 60 independent trials, each with the same unknown failure probability. Under that simple binomial model, the one-sided 95% upper confidence bound is one minus 0.05 raised to the power of one divided by 60: approximately 4.87%. This is a conditional statistical statement, not proof that the system's failure rate is zero. It is also not a probability that the fixed, unknown parameter lies below the bound.

NIST's exact binomial confidence-interval reference explains exact interval methods and why normal approximations can be unsuitable when failures are few. The simple calculation above assumes a particular model. It is not automatically appropriate for finite-population sampling without replacement, clustered customer records, adaptively selected cases or trials with different probabilities. A statistically trained reviewer should select the method that matches the actual design.

Repeatedly submitting the same enquiry can reveal output variability, but it does not create 60 independent customer situations. Likewise, cases sharing a source document, branch procedure or reviewer may have correlated outcomes. Record both the number of distinct cases and the number of runs. Explain whether the inference concerns variation across tasks, variation within a task, or both. Avoid a single confidence figure that conceals these different sources of uncertainty.

Protect the result from selection and judgement bias

Generate the random selection after the population frame and eligibility rules are frozen. Retain the selection code or method, random seed where applicable, frame identifier and any software version needed to reproduce the draw. Randomness does not compensate for a biased frame, but recording the draw helps detect selective substitutions. Access to outcome labels before sampling should be controlled where practical, particularly when the provider supplies the records.

Define the error rubric independently of the system's preferred answer. For the lost-property case, two plausible routing destinations may both be acceptable when the enquiry lacks sufficient detail. The rubric should specify when clarification is required, when multiple destinations are valid and how uncertain cases are adjudicated. Use qualified reviewers for Arabic and mixed-language records; machine translation alone can remove the ambiguity that the system actually faced.

Resolve disagreements through a documented adjudication process. Keep the original labels, the disputed reason and the final determination rather than overwriting the history. Review a subset of apparently correct cases as well as failures, because an assessor may miss subtle errors. If an automated judge assists, establish its suitability against human-reviewed examples and keep its errors visible. Evaluation integrity covers contamination, judge validity and other reasons a reported score may be misleading.

Handle missing records and changing systems explicitly

A selected record that cannot be retrieved is not a pass. First distinguish an ineligible item, an unavailable item, a pending outcome and an unreadable source. Use the replacement rule agreed before testing; do not choose an easier neighbour merely to maintain the target count. Report original selections, valid tested units, exclusions, replacements and unresolved items separately. If missingness may relate to failures, explain the resulting bias rather than presenting a clean percentage.

A release during fieldwork can divide the population into materially different groups. Preserve the old release's results and decide whether the new version needs a separate sample. Mixing both versions may be appropriate for an explicitly weighted period-wide estimate, but it is weak evidence for the new release alone. Record changes to prompts, retrieval sources, routing rules and employee procedures as well as the named model: these can alter outcomes without a model change.

When an early result leads to expanded testing, document the reason and distinguish the original design from the extension. Adaptive investigation is useful, yet conventional fixed-sample statistical claims may no longer apply. An honest report can present an initial probability estimate and a subsequent diagnostic investigation without merging their denominators or pretending that the full procedure was planned from the outset.

Separate discovery, confirmation and estimation

A short pilot can reveal that the planned test cannot be performed consistently. Perhaps reviewers disagree about which queue should receive an enquiry, or the export lacks the final destination. Use that pilot to refine the rubric and extraction process before freezing the main design. Do not quietly retain favourable pilot observations while dropping inconvenient ones. Decide in advance whether pilot cases are excluded from the estimation sample, and retain the reason in the workpaper.

Discovery also differs from confirmation. A case found through staff complaints is useful for understanding a suspected defect. Replaying it after the rules are clarified can confirm the observation under controlled conditions. Neither step makes that complaint representative of the population. A subsequent probability sample answers the different question of how often the defined defect appears in eligible operational records. Keeping these stages separate lets the audit learn during fieldwork without presenting learning-driven selection as a prespecified statistical experiment.

For the hypothetical lost-property process, reviewers might initially disagree about enquiries mentioning both a station and a vehicle. They should resolve the routing criterion with the accountable process owner, preserving the ambiguity and the date of clarification. If the published operating procedure was itself unclear during the tested period, that may support a separate process-design finding. Retrospectively inventing a clearer rule and treating every earlier departure as employee error would misstate what the evidence shows.

Agree how the decision changes under plausible interpretations of unresolved cases. If several records could reasonably be classified either way, present a sensitivity analysis or an explicit unresolved category rather than selecting the label that produces the preferred headline. A sensitivity analysis is not a substitute for better evidence; it shows whether the decision is robust to the remaining uncertainty. Explain which denominator each displayed result uses.

Finally, distinguish a stopping rule from a convenience stop. A protocol may specify that testing stops and escalates when a particular serious condition appears, because continuing would create unnecessary exposure. Document that decision and its implications for any planned estimate. Stopping because the observed percentage has become favourable can invalidate the intended inference. The audit manager should approve changes to the plan, and the report should describe material changes that affect interpretation.

A sampling plan that can survive review

Reusable sampling-plan fields with concrete completion rules

Field	What to record	Review question
Assertion and unit	Final routing destination per unique completed enquiry	Does the evidence identify the actual outcome rather than only a suggestion?
Population and frame	Period, eligibility, source query, reconciliation and exclusions	Could unsuccessful or deleted tasks be missing?
Selection and allocation	Probability or purposeful method, strata, inclusion probabilities and reproducibility information	Can the selection support the claimed inference?
Adjudication	Rubric, reviewer competence, disagreement process and treatment of uncertainty	Is an ambiguous case being forced into a convenient label?
Analysis	Numerators, denominators, weights, uncertainty method and missing outcomes	Are targeted tests kept separate from prevalence estimates?
Limits and decisions	Untested contexts, release boundaries and escalation conditions	Would a reader overgeneralise beyond what was observed?

Before finalising, have a reviewer reconstruct one selected record from the frame through selection, evidence, adjudication and reported result. This modest exercise often exposes a missing join, an unexplained exclusion or a denominator that changed during analysis. Store the reconstruction with the audit working papers. Reproducibility concerns the reasoning and evidence trail; a nondeterministic model need not reproduce an identical sentence for the original observation to remain documented.

For a Saudi organisation, useful scoping inputs include the business process, languages, relevant channels, deployment versions, available operational counts and the decisions the audit must support. Do not send unmasked personal enquiry records in an initial request. OpenForAI's AI auditing service can be discussed as a bounded evidence-led assessment, with the sampling approach and reporting limits agreed for the engagement. Discuss the population and sampling needs for your AI audit.