AI Audit Retesting

AI audit retesting asks whether a specific finding has been addressed under defined conditions. It is not a replay designed to produce a green result, and it is not automatically a full new audit. A credible retest connects the original criterion and failure mechanism to the corrected version, new evidence and an explicit closure decision. It also checks whether the correction introduced important adjacent problems.

This guide is for audit managers, control owners and technical reviewers deciding whether to close an AI finding. A hypothetical employee travel-expense coding assistant provides a worked example. The assistant proposes internal expense categories for review; the example does not provide tax advice or validate actual employee claims. The proposed protocol separates implementation evidence, behavioural verification and evidence of operation after deployment.

Define exactly what is being retested

Start with the original finding, criterion, affected scope and observed condition. Identify which part of the response is claimed to resolve it. A model upgrade, new instruction set, revised mapping table and additional human review have different implications. The retest must examine the relevant change and the pathway it is supposed to interrupt. Testing a new model alone may miss an unchanged integration defect.

Agree the closure criteria before reviewing the new results. State the outcomes required, the conditions under which they must hold and the evidence needed. If the original finding concerned an approval bypass, content accuracy tests do not close it. If it concerned semantic misclassification, a valid output schema does not establish correct meaning. Keep the procedure aligned with the actual departure.

Review the affected deployment inventory. The correction may exist in one environment while other instances still use the old configuration. Identify the tested version and the scope to which a successful result can apply. Partial deployment may justify partial verification with remaining exposure recorded, rather than a single closed label covering every site or workflow.

NIST SP 800-53A Rev. 5 is a primary reference for structured security and privacy control assessment methods. This guide's AI retest protocol is an original practical proposal. The method and evidence should be selected for the specific control, rather than assuming that one generic test sequence establishes every kind of effectiveness.

Distinguish four kinds of closure evidence

Evidence types in a proposed AI retest

Evidence type	Question answered	Limit
Implementation evidence	Was the claimed change made and deployed to the identified scope?	A deployment record does not prove that the behaviour is correct.
Original-case verification	Does the corrected version address the known examples?	Known cases may have influenced the fix and cannot establish generalisation alone.
New-case and adjacent testing	Does the intended mechanism work on relevant unseen variations without important side effects?	Purposeful cases do not automatically estimate operational prevalence.
Operating evidence	Does the revised control function in the actual workflow over an appropriate observation window?	The conclusion remains bounded by period, population and available records.

Not every finding needs the same balance. A documentation correction may be verified by examining an approved, applicable record and its communication. An operating-control failure usually needs more than a document. A permission change may need configuration inspection and authorised boundary tests. Explain why the selected evidence is sufficient for this finding and what remains outside the verification scope.

Worked example: expense coding that passes the old cases

In the hypothetical audit, the assistant misclassified certain travel-related receipts under the organisation's approved internal category rules. The original finding was supported by 12 identified examples. Management changes the category mapping and instruction version, then reports that all 12 examples now produce the expected category. That is useful evidence about the known cases, but it does not yet establish that the underlying distinction is handled reliably.

The reviewer prepares 48 new, purposeful cases covering the same distinction with different wording, mixed Arabic and English descriptions, itemised charges and incomplete supporting context. Three new cases are incorrectly coded. The 48 cases were selected to exercise the mechanism, not sampled randomly from ordinary expense submissions. The reviewer therefore reports three exceptions in this challenge set, not a population error estimate or a general accuracy percentage.

The decision should reflect the agreed criteria. If the required outcome was correct coding or explicit referral when the source did not support a category, the three exceptions may mean the finding is not ready for verified closure. The original 12 successes do not cancel the new failures. Investigate whether the correction overfitted the examples, whether the criterion was misunderstood or whether a different mechanism remains.

Illustrative retest record and its interpretation

Test group	Result	Permitted conclusion
12 original defect cases	All produce the agreed category or required referral after correction	The known examples were addressed under the tested version.
48 new purposeful variations	3 exceptions against the agreed rule	The mechanism still fails in identified conditions; prevalence is not estimated.
Deployment check	Corrected mapping and instructions identified in the test environment	The tested configuration is known; production reach needs its own evidence.
Operating follow-up	Not yet performed in this illustrative stage	No claim of sustained production effectiveness is available.

These numbers are illustrative, not a recommended retest size. The appropriate design depends on the finding, consequence, diversity of conditions and decision. Record the selected case identifiers, why each group exists and how results were adjudicated. Use the sampling guide when the retest also intends to estimate operational frequency; keep that analysis separate from purposeful challenge testing.

Protect the new cases from contamination

Original examples are often shared with implementers so they can understand and correct the issue. That is legitimate, but it makes them weak evidence of performance on unseen cases. Preserve a separate set of new cases where feasible and control who can see them before testing. If developers have already used the cases during tuning, disclose that fact rather than presenting them as an independent holdout.

Create variations that test the same underlying distinction, not cosmetic rewordings alone. In the expense example, change the presence of supporting context, the arrangement of itemised charges and the language of the description while keeping the approved coding rule explicit. Avoid introducing a new accounting interpretation that the original criterion never addressed. A domain owner should resolve genuinely ambiguous category rules.

Use competent adjudication independent of the model's preferred answer. If an automated judge assists, verify its suitability and inspect consequential disagreements. Keep the source, expected outcome rationale and final adjudication linked. Evaluation integrity covers leakage, judge validity and other reasons a favourable test result can be misleading.

Design regression checks around plausible side effects

A correction can improve the target cases while damaging adjacent behaviour. A broader category rule may misclassify previously correct receipts; a stricter referral condition may send nearly everything to manual review; an output-format change may break downstream validation. Identify plausible effects from the actual change, rather than assembling an enormous unrelated test suite merely to demonstrate effort.

Include representative adjacent cases that should remain unchanged and cases that should now be referred. Explain why they are relevant. Regression tests check whether known or anticipated problems appear under tested conditions; the tests themselves do not prevent recurrence. Prevention depends on the design, release and operating controls that use the results appropriately.

Do not average away a serious regression. A new exception affecting a consequential boundary may block closure even when most other cases improve. Compare results against the agreed criteria and the severity of the affected pathway. Where the retest discovers a genuinely new issue, record its relationship to the original finding and decide whether it requires a separate record in the findings register.

Account for nondeterministic outputs and repeated runs

Some AI systems can produce different outputs for the same input. Decide whether repeated runs are needed to examine that variability. Record the number of distinct cases and the number of runs per case separately. Repeating one case many times does not create the same evidence as testing many different cases, and correlated runs should not be treated as independent observations without justification.

Preserve the relevant model or service identifiers, instructions, source references and generation settings available to the assessor. If the provider can change the service without exposing a precise underlying version, record the observed service and time boundary. A retest result remains useful within that limitation; inventing a stable version identifier would make it less trustworthy.

Set the repetition and stopping rules before examining the outcome. Re-running until the system produces an acceptable answer selects a favourable result and conceals variability. Retain all relevant runs and explain any technical failures or exclusions. A timeout, missing response and incorrect category are different outcomes and should not be collapsed without a defined rule.

Verify the correction in the real operating context

A controlled test can show the corrected behaviour under specified conditions. Operating evidence asks whether the control is deployed, used and maintained in the actual process. For the expense assistant, examine whether uncertain outputs reach the review queue, whether reviewers can access the source and whether their decisions are recorded before the proposed category is accepted. A training slide does not establish these events.

Choose the observation window according to the control's frequency, relevant variation and consequences. A universal number of days is not justified by this guide. A rarely used workflow may need more elapsed time or a different evidence strategy than a high-volume process. Record the population available during the window, missing outcomes and changes that occurred. Do not infer sustained operation from a period with no eligible cases.

Compare the test and production environments. Permissions, integration paths, source quality and user behaviour may differ. Where the correction was tested only in a sandbox, state what additional evidence is needed before concluding about production. If a production test is authorised, use suitable controlled procedures and avoid creating real financial or other consequential actions merely to prove the interface works.

Use a closure decision with more than two possible outcomes

Proposed retest dispositions

Disposition	When it fits	Required record
Verified corrected	Agreed criteria are met by adequate evidence within the stated scope	Tested versions, results, reviewer and remaining limits.
Partially verified	Some deployments or criteria are supported while others remain unresolved	Exact verified boundary and outstanding work.
Further correction required	Retest exceptions show that the required outcome is not met	Exceptions, revised diagnosis and next action.
Inconclusive	Evidence or conditions do not permit a defensible judgement	Missing evidence, decision implications and follow-up owner.
Alternative management disposition	Authorised retirement or risk acceptance changes the response	Separate decision evidence; no false claim of technical correction.

A failed retest is information, not a reason to suppress the result. Retain the version, cases and observations, then update the remediation plan. An inconclusive retest is different from a failed control: the evidence may be unavailable or the environment may not reflect the relevant conditions. Both can prevent closure, but they require different next steps.

Risk acceptance belongs to management's authorised decision process. It should not be recorded as a successful retest. Likewise, retirement may remove exposure without demonstrating that the original behaviour was repaired. The register and report should preserve those distinctions so a later reactivation does not inherit a misleading “fixed” status.

Make the review sufficiently separate and competent

The implementer should provide the change explanation and evidence, but should not be the only person deciding whether the audit finding is resolved. Arrange competent review with appropriate separation for the engagement. Where resources require overlapping roles, disclose the conflict and define safeguards. Independence and self-review considerations matter because a technically knowledgeable person may still be reviewing their own work.

Review competence must match the question. Expense-category meaning may require the responsible process expert; Arabic source interpretation requires suitable language competence; statistical inference may require an analyst. A reviewer need not personally possess every skill, but must obtain and assess the required specialist input. A generic approval signature cannot replace those judgements.

Preserve the retest record in the working papers: original criterion, change claimed, scope, selected cases, procedure, observations, analysis, review notes and disposition. A later reviewer should understand why closure was granted or withheld without relying on the memory of the engineer who implemented the change.

Investigate a retest exception before changing the acceptance rule

An exception can arise because the correction is inadequate, the test fixture is wrong, the criterion is ambiguous or the environment differs from the intended scope. These explanations lead to different actions. Preserve the observed result first, then investigate the relevant evidence. Do not relabel a failure as acceptable simply because it appears after the team has announced completion. Equally, do not insist that a genuine test-data error is a system defect.

For the expense example, one of the three new exceptions might contain an itemised charge whose category depends on context absent from the source. If the agreed rule requires referral in that situation and the assistant confidently assigns a category, the exception remains meaningful. If the expected label was created from information the assistant was never authorised to receive, the test oracle may be wrong. Record the adjudication and its effect on the result, including the original label.

Original exception-resolution record for a retest

Record element	Completion requirement
Observed discrepancy	Preserve input, output, expected outcome and the rule used to judge it.
Competing explanations	Identify correction failure, fixture error, criterion ambiguity or environment mismatch where credible.
Adjudication evidence	Obtain qualified source interpretation and configuration evidence relevant to the disputed point.
Decision	Retain the exception, correct the test record or revise the scope with an explicit reason.
Effect on closure	State whether the remaining evidence meets the original or formally revised criteria.

If the criterion genuinely needs revision, approve the change through the engagement process and explain why. Determine which earlier cases must be re-evaluated. A changed rule can alter both failures and apparent successes; reviewing only the inconvenient failures creates another selection bias. Keep the earlier result and the revised analysis linked so a reviewer can distinguish a legitimate clarification from moving the goalposts.

Prepare the next retest from the updated diagnosis. Some cases should remain as regression checks, while additional unseen cases examine the revised mechanism. Do not keep calling the same disclosed test set unseen after successive development cycles. Maintain a record of when cases became available to implementers and which evidence still provides an independent challenge.

Finally, consider whether repeated unsuccessful retests indicate a broader planning problem. The team may lack an agreed criterion, sufficient source data or a feasible control design. In that situation, another identical test cycle is unlikely to resolve the issue. Escalate the underlying decision: revise the approach, constrain the use or obtain the missing evidence. Retesting should clarify the control position, not become a ritual that continues until everyone is tired enough to accept closure.

Link closure to future review triggers

Verified correction is bounded by the tested conditions. Record changes that should trigger reconsideration, such as a new model service, revised category rules, altered source formats or expansion to a new workflow. The trigger does not mean every small change requires a full audit; it identifies when the basis of the closure decision may no longer hold.

Keep recurrence monitoring distinct from the retest conclusion. A later alert can identify a possible return of the issue, but it needs investigation against the criterion and affected version. Preserve the original closure evidence and link the new observation rather than rewriting history. This makes it possible to distinguish a failed correction, a new mechanism and an unrelated event.

For a scoped AI audit follow-up, bring the original finding, remediation plan, change identifiers and available verification evidence. Discuss retesting and closure criteria for your AI audit findings.