AI Audit Working Papers

AI audit working papers should let a competent reviewer reconstruct what was tested, what evidence was available, how it was interpreted and why the conclusion follows. A folder of screenshots cannot do that on its own. Neither can a polished report that links to a dashboard whose contents change every day. The working paper is the reasoning record connecting a specific audit question to a bounded, reviewable answer.

This guide is for audit managers, technical assessors and evidence owners building that record for an AI system. It uses a hypothetical assistant that extracts observations from equipment-inspection notes. The focus is the documentation of an examination, not the engineering of the assistant. Use the scope agreement to establish the engagement boundary and evidence-handling procedures to control collection, access and retention.

What belongs in a working paper?

Start with the question, the applicable criterion and the reason the procedure can answer it. “Review extraction accuracy” is an activity label, not an adequate objective. A better objective is to determine whether the selected outputs preserved explicitly negated inspection observations and whether uncertain cases entered the prescribed review queue. That wording identifies both the content judgement and the workflow evidence needed. It also prevents a correct-looking output from being treated as proof that review occurred.

Record the system boundary precisely: model or service identifier, instruction version, input-processing version, deployment environment, relevant source period and any configuration that could influence the result. Include the control owner, tester and reviewer by actual role or identifier. If a supplier cannot expose an underlying model version, record the available service identifier and the limitation. Do not invent a more precise version reference to make the file look complete.

Then preserve the procedure actually performed, its inputs, observations, exceptions, analysis and conclusion. A planned procedure and an executed procedure may differ. Keep both where the difference matters. Record why a test was changed, who approved the change and whether it affects interpretation. The aim is a faithful record of fieldwork, including uncertainty and unsuccessful attempts, rather than a retrospective story in which every planned step worked perfectly.

The IIA's Global Internal Audit Standards address engagement documentation and the support for results. The original structure below is a practical AI-specific proposal, not a reproduction of the standards or a claim that this guide establishes conformance. Professional requirements applying to a particular audit function still need to be identified for that engagement.

Design a file structure around evidence relationships

A useful system has stable identifiers for objectives, procedures, evidence items, observations and findings. An observation is what the assessor recorded; a finding is the assessed departure from a criterion. Several observations may support one finding, and one evidence item may inform several procedures. Use explicit relationships instead of copying the same attachment into many folders. Copies tend to drift, and a later redaction may leave an older unredacted version elsewhere.

Keep the workpaper index separate from the evidence repository. The index describes the item and points to its controlled location; it need not expose personal information or access credentials. A reviewer who can read the index should not automatically receive access to every raw source. The evidence owner can grant narrowly scoped access when the review requires it. Record inaccessible items as access limitations rather than hiding them behind broken links.

Proposed workpaper index for the inspection-note assessment

Identifier	Purpose	Linked material	Review status
OBJ-04	Preservation of negated observations and required escalation	Approved process rule CR-02 and scope reference	Criterion confirmed before testing
WP-04A	Content comparison against inspection notes	Selected-case manifest and redacted source pairs	Technical and language review completed
WP-04B	Trace uncertain outputs into the review queue	Event records and queue disposition evidence	One inaccessible interval remains disclosed
OBS-17	One output reversed an explicit negation	Source, generated output and reviewer adjudication	Observation confirmed; cause unresolved
F-03	Assessed departure requiring management response	OBS-17, applicable rule and consequence analysis	Draft finding under factual review

These identifiers are illustrative. Their value comes from stable meaning, not a particular naming scheme. Avoid embedding a severity label or owner name in the permanent identifier, since those may change. Maintain an index revision and a change log so the final report can be associated with the exact workpaper set on which it relied.

Build a worked test record, not a retrospective summary

Suppose the hypothetical inspection note states that a casing has surface marks but no crack was observed. The assistant's structured output records a crack. That discrepancy is an observation. Before calling it a model defect, confirm that the stored input is the same text sent to the service, that preprocessing did not remove the negation, and that the displayed result was not edited after generation. The workpaper should preserve this distinction between the observed error and its possible cause.

Annotated specimen: one negation-preservation procedure

Workpaper field	Illustrative entry	Why the detail matters
Objective	Determine whether explicit negative observations remain negative in the extracted record.	Restricts the claim to a defined semantic property.
Criterion	Approved extraction rule CR-02: preserve source meaning, including explicit negation; uncertainty or contradictory source wording requires review.	Identifies an internal rule; it is not presented as a regulatory requirement.
Source and context	Masked inspection note IN-147; source revision 3; output-processing build 12.	Distinguishes the observed version from later edits.
Procedure performed	Compare source wording, submitted input and extracted field; inspect related review-queue events.	Tests the content and the handling of uncertainty separately.
Observation	The output asserted a crack where the inspected source explicitly negated it.	States the discrepancy without attributing cause prematurely.
Supporting evidence	Controlled references to the source pair, request record and queue evidence.	Allows reconstruction without placing personal details in the index.
Conclusion and limitation	One confirmed content exception; causal attribution requires further examination.	A single case does not establish population prevalence or a system-wide failure rate.

Record what the assessor actually saw, including the time and source of the observation. If a live demonstration differs from the preserved output, keep both and explain the different conditions. Do not overwrite the original evidence with a successful replay. Nondeterministic behaviour, configuration changes and human edits are possible explanations to examine, not reasons to erase the initial record.

Make computational analysis reviewable

When scripts calculate error rates or compare extracted fields, preserve the code version, input manifest, execution environment and output table. A notebook that runs only on the author's machine is a weak handover. Record dependencies relevant to the calculation and provide instructions for reproducing the analysis in an authorised environment. Do not include production credentials in a notebook or command transcript. Secure access should be arranged separately.

Make denominators explicit in both the code and the narrative. Distinguish selected notes, readable notes, completed outputs, reviewed exceptions and unresolved cases. If a join removes records lacking an output, that transformation can bias a result towards successful processing. A reviewer should see the number of rows entering and leaving important transformations and the reason for exclusions. This is analytical lineage: where data originated and how it changed before becoming a reported number.

Preserve enough intermediate results to investigate discrepancies without retaining every unnecessary copy. For example, keep a case-level classification table with stable masked identifiers, adjudication status and references to restricted evidence. That supports recalculation while limiting duplication of raw notes. Apply the retention and access rules agreed for the engagement; computational convenience does not justify indefinite retention of sensitive material.

A successful rerun verifies aspects of the calculation, not every premise. The source extract may still be incomplete, the labels may be wrong or the criterion may be unsuitable. Link the analysis to the sampling design and the evidence-quality assessment. Keep statistical interpretation separate from mechanical calculation so a reviewer can challenge either without losing the underlying observations.

Document interviews and demonstrations with appropriate weight

An interview can explain how a control is intended to operate, identify relevant records and reveal inconsistencies. It does not by itself establish that the control operated throughout the audit period. Record who spoke, their role, the date, the question, the substance of the response and any corroborating evidence requested. Distinguish an interviewee's assertion from an auditor's observation. A signed note confirms the recorded account, not necessarily the truth of every statement.

A demonstration similarly shows behaviour under particular conditions. Record the environment, selected case, preparer, permissions and whether the sequence was rehearsed. If the system owner chose the example, say so. A demonstration can be useful for understanding the workflow, while an independently selected operational record provides different evidence. Do not convert “the owner demonstrated an approval screen” into “all restricted actions required approval”.

Where explanations conflict, preserve the competing accounts and identify the evidence needed to resolve them. The inspection supervisor may say that employees always review uncertain outputs while queue records show no review event for a selected case. Possible explanations include missing logs, review outside the system or a missed control. The workpaper should investigate those alternatives. It should not automatically prefer the more senior speaker or the more convenient account.

Keep facts, criteria, causes and consequences distinct

A clear analysis separates four elements. The criterion states what should happen. The observation states what was evidenced. The consequence explains why the departure matters in the actual workflow. The cause describes why it occurred only where the evidence supports that explanation. A proposed cause can be recorded as a hypothesis with a verification step. Treating a hypothesis as an established fact can lead management to implement a correction that never addresses the real mechanism.

For the inspection-note example, an incorrect extracted field might influence a maintenance decision, but the audit should establish whether the output was merely a draft, whether a competent reviewer corrected it and whether any downstream action occurred. A content error and an executed operational error have different consequences. Conversely, a reviewer correcting the sampled error does not prove that review is consistently effective. The workpaper needs evidence for each claimed protective step.

Use the findings register to track assessed issues without forcing every workpaper exception into a separate finding. Several observations may share a control weakness; other observations may need distinct owners and remedies. Preserve the mapping and the grouping rationale. The report should be concise, but the underlying record must not lose a materially different exception merely because it complicates the narrative.

Review the argument as well as the file

Administrative review checks that fields are completed, evidence references resolve and approvals are recorded. Technical review asks whether the procedure can support the conclusion. Language review asks whether the source meaning and the adjudication were understood correctly. These reviews may be performed by different competent people. A tick in a “reviewed” column should identify the review performed, the reviewer and any unresolved conditions.

A practical reviewer can work backwards from one reported statement. Identify the criterion, the linked workpaper, the selected records, the observed outcomes and the analysis producing the statement. Then work forwards from an exception to see whether it was reported, grouped, resolved or excluded with a reason. This two-direction check helps detect both unsupported claims and inconvenient evidence that disappeared between fieldwork and reporting.

Reviewer challenges that improve a working-paper file

Challenge	Evidence of an adequate answer
Could another assessor identify the exact tested version?	Available version identifiers, timestamps, configuration references and disclosed supplier gaps.
Could the result be explained by a preprocessing change?	Traceable source-to-submitted-input comparison or a recorded limitation.
Did the conclusion exceed the sample?	Population, selection and inference limits match the reported wording.
Were contrary observations retained?	Exception history, disputed interpretations and disposition reasons remain available.
Can the calculation be reconstructed?	Controlled inputs, code or calculation steps, denominator rules and saved outputs.
Was review substantive?	Review questions, responses, resulting changes and unresolved matters are recorded.

Review notes should be resolved visibly. Keep the question, response and resulting change; mark an unresolved disagreement rather than deleting it. If a reviewer identifies a material weakness after the report draft is prepared, update the analysis and assess whether the report conclusion must change. Formatting completion is not evidence that the reasoning is sound.

Manage corrections without rewriting history

Errors in working papers should be corrected, but corrections need a trace. Record the earlier value, corrected value, reason, author and time for material changes. If a finding count changes because two cases were duplicates, preserve the reconciliation. If a translation is corrected, identify which judgements were affected. A clean final file can still retain a controlled revision history; transparency does not require presenting every internal draft to every recipient.

Set a documented completion point for the engagement file and a process for later additions. New evidence received after reporting should be labelled with its receipt date and assessed for its effect on the issued conclusion. Do not backdate a newly created record to make it appear available during fieldwork. Management may legitimately improve documentation after a gap is found, but that new document is evidence of a later action, not proof of historical operation.

Retention periods and access permissions should follow the engagement's applicable obligations and organisational policy. There is no single universal retention period established by this guide. Distinguish the working-paper record needed to support the audit from temporary extracts and redundant files. A lawful preservation requirement may change ordinary disposal arrangements; the responsible legal and records owners should resolve that question rather than an assessor improvising a rule.

Preserve evidence that challenges the preferred explanation

A useful workpaper should be capable of changing the assessor's mind. In the inspection-note example, a correct submitted input and an incorrect generated field support a different investigation from an input where the negation disappeared during text extraction. If both possibilities were considered, record the evidence that ruled one out. Otherwise the final file may look persuasive while concealing an untested assumption about the component responsible.

Also record observations that reduce the assessed consequence. If the reviewer corrected the field before anyone used it, that fact matters. Preserve the correction event and its timing rather than dismissing it because the audit is focused on errors. Equally, do not treat the correction as proof that all similar errors would be caught. The file should support a measured assessment of this case while identifying the separate evidence required to assess control consistency.

When evidence remains contradictory, state the unresolved alternatives and the implication for the conclusion. A reviewer can accept a bounded conclusion with an explicit limitation more readily than a confident statement whose missing premise is hidden. Allocate follow-up ownership where necessary, but do not convert an outstanding request into evidence received. The workpaper's status should describe the investigation as it stands.

What a completed handover should contain

The receiving reviewer should obtain a usable index, approved scope and criteria, executed procedures, controlled evidence references, analysis, exceptions, conclusions, review history and unresolved limitations. Access arrangements should be tested before handover. A repository that only the departing author can open is not a completed transfer. At the same time, avoid distributing unrestricted copies merely to solve an access problem; grant the minimum authorised access required.

Check that the report's structure and conclusions correspond to the final workpaper versions. Every material report claim should have support, and every material unresolved issue should have a visible disposition. Record where the evidence cannot support a requested conclusion. That limitation is part of the deliverable, not an administrative embarrassment to be hidden in the file.

For an initial AI audit scoping discussion, prepare a masked workpaper example, the current evidence index and a description of the decisions the audit must support. This allows the assessment method and documentation expectations to be discussed without exposing inspection notes unnecessarily. Discuss a reviewable working-paper approach for your AI audit.