AI Audit Coverage: Tested vs Not Tested

AI audit coverage describes the questions, components, populations and conditions actually examined, together with the evidence and limitations behind each conclusion. It is the bridge between an agreed audit scope and the work completed. A scope document says what the engagement intended to address; a coverage record shows what the auditor could establish and what remains untested or unresolved.

This guide is for audit sponsors, system owners and report readers who need to understand the reach of an AI assessment. The hypothetical example is a multilingual research-grant document checklist generator. It turns approved programme instructions into an Arabic or English checklist for administrative use. It does not assess applicants, rank proposals or award funding. The coverage question concerns the generator's behaviour and evidence, not the fairness of funding decisions.

A useful AI auditing engagement makes its limits visible alongside findings. “No issue found” is meaningful only with the tested version, paths, evidence and conditions attached. It should never be used to convert an untested feature into an implied clean result.

Separate scope, coverage and results

Scope defines the intended system boundary, criteria and work. Coverage records the portion actually examined and how it was examined. Results describe observations and conclusions within that coverage. These concepts are related but not interchangeable. A fully completed scope can contain failed controls; an apparently clean set of results can cover only a small part of the intended work.

The scope agreement guide addresses how the engagement boundary is set. This page addresses the later evidence map. If access restrictions or system changes prevent planned work, preserve the original intent and record the deviation. Quietly rewriting the scope at the end can hide the very limitation the reader needs to understand.

Define the unit of coverage. It might be a claim, control, workflow, model version, language, document class or access path. A count of test cases is not a substitute for that definition. Twenty similar tests of one easy path may leave a consequential boundary unexamined, while a smaller set can address several distinct questions with stronger evidence.

Start from decisions the report is meant to support

Ask what the sponsor will do with the report. A decision to continue a limited internal checklist service requires different evidence from a decision to expose it to applicants or connect it to an automated submission system. The coverage record should show which intended reliance is supported, not merely list technical components that were convenient to inspect.

For the research-grant example, relevant questions include whether current programme requirements are preserved, whether Arabic and English outputs retain the same conditions, whether withdrawn instructions stop influencing new checklists and whether the interface makes unresolved source conflicts visible. Applicant scoring and funding allocation remain outside this system's purpose.

Identify claims that depend on other services. A checklist may rely on document parsing, identity management or a supplier model. Testing the final output can provide useful evidence, but it may not establish the internal operation of those dependencies. Name the boundary and the evidence available rather than assigning the whole dependency a blanket covered status.

Record the method as well as the subject

Reading a policy, interviewing an owner and observing a controlled test provide different evidence. A design review can establish that a process is documented or a control is configured. It cannot by itself establish that the process operated consistently during a period. Coverage should therefore say how the question was examined and which conclusion that method can support.

NIST SP 800-53A provides a tailorable methodology for security and privacy control assessment. It is useful background on assessment planning, not a universal AI audit checklist. The original coverage record below applies the general discipline of matching evidence to the question without claiming that every AI topic is a NIST security control.

Attach evidence references that a reviewer can follow. A row saying “tested” should lead to the case, configuration, expected result, observed result and adjudication. The working-paper guide explains how to maintain that link. An evidence reference that points only to a meeting invitation or an unversioned slide may not support the conclusion stated.

Use distinct statuses for distinct evidence conditions

“Not tested” should not be a catch-all for every limitation. Separate work that was outside the agreed scope, planned but not executed, executed but inconclusive, and reviewed only for design. Also distinguish a criterion that does not apply from one whose applicability has not been established. These statuses lead to different decisions.

Original coverage status dictionary with required explanation

Status	Meaning	Required accompanying information
Tested with a conclusive result	Evidence supports a stated observation against the criterion.	Met or unmet result, evidence, version and conditions.
Tested but inconclusive	A procedure ran, but the available evidence cannot settle the question.	Missing or conflicting evidence and the next resolving step.
Design reviewed only	Documentation or configuration was examined without sufficient operating evidence.	The design conclusion and the period behaviour not established.
Planned but not executed	Agreed work did not take place.	Reason, affected reliance and owner of the next decision.
Outside scope	The engagement did not undertake this question.	Boundary and potential relationship to the intended use.
Not applicable	The criterion does not apply under an evidenced condition.	Rationale and evidence establishing that condition.

A failed control can be fully covered: the test produced a clear adverse result. Conversely, a blocked test has no passing result to report. Keep coverage and findings in separate fields so management can see both the amount of work and the consequences of what was found.

Build a coverage map around the actual service

The example below is hypothetical. Its rows represent different questions rather than equally weighted units of risk. It deliberately includes a design-only conclusion, an access limitation and a feature outside scope. The map is useful because it makes those differences explicit, not because it produces a single percentage.

Illustrative coverage map for a research-grant checklist generator

Question	Population or condition	Method and status	Bounded conclusion
Current requirements preserved	Selected approved programme instructions in the tested release.	Controlled comparison; conclusive.	Report the specific preserved and omitted requirements observed.
Arabic-English meaning retained	Paired checklist cases reviewed by qualified bilingual reviewers.	Controlled comparison; conclusive.	Applies to the paired case set, not every possible programme.
Scanned appendix extraction	Supported scanned attachments in the intended scope.	Planned but not executed because test fixtures were unavailable.	No conclusion about those attachments.
Withdrawal procedure	Documented process for replacing programme instructions.	Design reviewed only.	Procedure exists; propagation behaviour was not established.
Supplier model change detection	Provider-controlled deployment changes.	Evidence incomplete; inconclusive.	Available notices do not establish every change affecting the tested period.
Funding allocation	Decisions made outside the checklist generator.	Outside scope.	No conclusion about applicant ranking or awards.

Do not describe the whole service as audited without the qualifications that materially affect reliance. The map may support continued limited use for reviewed digital instructions while showing that scanned attachments need further work. The accountable owner decides the operational response; the audit provides the evidence and limits behind that choice.

Explain the dimensions hidden inside a test count

Coverage has several dimensions: function, language, data class, identity, lifecycle state, environment, release and time window. A test may cover one combination but not another. Arabic output from a current digital document differs from Arabic output from an old scanned appendix. Listing both “Arabic” and “scans” somewhere in the plan does not prove their interaction was tested.

Identify material interactions based on the service and risk. Exhaustively testing every combination may be infeasible, so explain the selection rationale and the combinations deliberately left unexamined. A risk-based matrix is more useful than an unqualified statement that all features were covered because each feature appeared at least once.

Keep the distinction between breadth and depth. Breadth concerns the range of questions or conditions examined. Depth concerns how strongly a particular question was tested, including repetition, variation and operating evidence. Neither should be inferred from the raw number of screenshots or log records collected.

Reconcile planned, executed and conclusive cases

Suppose a hypothetical test plan contains 42 cases. Thirty-seven are executed, of which 33 produce conclusive results and four remain inconclusive. Five planned cases are not run. Among the 33 conclusive cases, 30 meet the criterion and three do not. These counts should reconcile before any summary is presented.

Execution is 37 divided by 42, approximately 88.1%, if the plan's case count is the chosen measure. The rate of met criteria among conclusive cases is 30 divided by 33, approximately 90.9%. Neither is a percentage of system safety or a complete measure of audit coverage. The first describes activity against the plan; the second describes results within the conclusive subset.

The four inconclusive cases and five unexecuted cases need individual explanations. Their importance depends on the questions they address. One untested access boundary may matter more than several completed formatting checks. Report the materiality of the gap without inventing a mathematical risk weight merely to produce an attractive overall score.

Keep sample coverage separate from population assurance

A test can examine every selected sample item while covering only a subset of the operating population. State the population, selection mechanism and unit. The sampling guide addresses how that design affects generalisation. Do not use “100% tested” to imply that every production output was examined when it refers only to a sample.

Targeted challenge cases can establish a failure under chosen conditions. They do not estimate prevalence across all use. A representative sample can support a different type of inference but may contain too few rare conditions to examine a critical mechanism. A coverage map can show both evidence types without treating them as interchangeable.

Also distinguish observation windows. A test performed on one day can establish behaviour at that time, while a period review can examine operation across a defined interval. A long engagement duration does not automatically create long-period operating coverage. The report should state the dates of evidence, not just the start and end of the consulting project.

Handle evidence restrictions openly

Restricted supplier access, unavailable historical records or incomplete logging can prevent a planned conclusion. Record the specific evidence that was unavailable, why it matters and what alternative evidence was considered. The vendor-controlled systems guide addresses strategies for restricted-access reviews. Alternative evidence should be assessed for relevance rather than accepted simply because it is available.

Do not equate missing evidence with proof that the control failed. The correct conclusion may be that operation could not be established. Equally, do not treat a management assertion as a substitute for the missing test without stating the change in evidential strength. Readers need to know whether they are relying on observation, documentation or assertion.

Where access becomes available later, update the coverage record through a traceable version. Preserve the original limitation and the new evidence that resolves it. A later successful test should not rewrite what was known when an earlier decision was made.

Account for changes during the engagement

An AI service can change while testing is underway. Record which cases ran against which release, source set or policy version. If a material change occurs, assess whether earlier evidence still answers the question. Some findings may remain relevant; other results may require retesting or partitioned reporting.

A fixed report date does not mean every component was assessed at that date. State the relevant cut-offs and any changes after them. If the provider can change an underlying model without a stable identifier, disclose how the audit bounded the observable deployment and what reproducibility remains possible.

Changes in use also matter. The checklist generator may begin as an internal aid and later be presented directly to applicants. That can introduce new reliance, language and accessibility questions even without a software change. The coverage record should follow the actual use proposed, not remain attached to the original project description.

Make exclusions understandable to nontechnical readers

An exclusion should name the practical consequence. “OCR excluded” is less useful than explaining that the audit does not establish whether requirements in scanned appendices reach the checklist. “Vendor internals unavailable” should identify the claims that remain unresolved. Translate technical boundaries into the decisions the report reader may otherwise overextend.

Place material limits beside the relevant conclusion and in the report summary. A distant appendix may not prevent a reader from interpreting “no material findings” too broadly. The report structure guide explains how findings, evidence and limitations should support the overall interpretation.

Avoid defensive wording that obscures the useful result. State positively what the evidence establishes, then identify the remaining boundary precisely. The aim is a decision-quality account, not a long list of caveats detached from the actual tests.

Use coverage gaps to plan the next responsible action

For each material gap, identify the missing procedure or evidence, the owner and the decision it would resolve. The next step may be a controlled fixture set, a supplier review session, a period of operating records or a narrower service promise. Not every gap requires a full repeat of the engagement.

Prioritise by consequence and reliance. If management plans to enable scanned attachments, the unexecuted appendix tests become directly relevant to that decision. If the feature remains disabled and that restriction is enforced, the same gap may be acceptable within a narrower current use. Record the basis for acceptance and the trigger for reassessment.

Use retesting and closure evidence when a gap is resolved or a finding corrected. Coverage completion and remediation closure are different events: running a previously blocked test may reveal a failure. The coverage record should show both the new evidence and the resulting finding rather than treating execution alone as a positive outcome.

Translate the map into a conclusion people can use

An illustrative conclusion might state: the review examined the listed digital programme instructions and paired Arabic-English checklists in the named release; the report describes the observed omissions; scanned appendices were not tested; withdrawal operation and complete supplier-change visibility were not established. That wording gives the reader a usable boundary without implying that funding decisions or every programme were assessed.

Ask a nontechnical decision-maker to explain what they believe the conclusion permits. If they infer that every source format or later release is covered, revise the presentation. This is a check of communication, not a request to soften adverse results. The evidence map and summary should convey the same scope of reliance even when the summary is much shorter.

When several reports are combined, preserve their different periods, criteria and access limits. Two narrow reviews do not automatically form a complete assessment of the shared service. Identify the overlap and the questions left between them. Reusing another review's evidence requires establishing its relevance to the current configuration and question, rather than copying its status label.

Keep management acceptance visible without changing the evidence status. A sponsor can decide to proceed with a documented limitation, but that does not turn an unexecuted test into a completed one. The coverage map should continue to show the gap, the accepted use boundary and the event that would require further review.

Maintain the coverage record as part of the audit evidence

Assign ownership for keeping the map consistent with workpapers and the issued report. Review the final map for orphaned claims, unsupported statuses and contradictions. Every material conclusion should lead to evidence, and every material limitation should appear where a reader can understand its effect.

Prepare a draft map early enough to expose access and fixture gaps while they can still be addressed. For scoping, bring the system inventory, intended reliance, planned test dimensions and available evidence. Discuss an AI audit with explicit tested and untested coverage to make the final result useful without implying conclusions the work cannot support.