How to Scope an AI Audit

Scope an AI audit by defining the decision it must support, the system and workflow being examined, the criteria for assessment and the evidence needed to answer the agreed questions. “Audit our AI” is an expression of concern, not an executable scope. Without a boundary, the assessor may examine the model while management assumes that source data, human approvals, external communications and operational outcomes have all been covered.

This guide is for audit sponsors, technology leaders and procurement teams commissioning a bounded AI assessment. It uses a hypothetical assistant that drafts product-recall notices from approved source material. The example concerns the control of drafting and release; it does not provide product-safety advice or establish the adequacy of a real recall. The proposed scoping records help identify what will be tested, what will remain unknown and who can authorise changes.

Begin with the decision that needs evidence

Ask what someone will do differently after receiving the report. They may decide whether to begin a controlled pilot, expand use to another language, renew a supplier, retain a human approval step or investigate an incident. Each decision creates different evidence needs. A review of documented design may inform a pilot plan, but it cannot establish that controls operated reliably over a period that has not yet occurred.

Write two or three concrete audit questions before naming frameworks or tools. For the recall-notice assistant, one question might ask whether selected drafts preserve mandatory information from the approved source. Another might ask whether a draft can be released without the required approval. The first needs a content comparison and authoritative source; the second needs permission, workflow and execution evidence. Treating both as “accuracy testing” would miss a material boundary.

Identify the intended reader and required reporting form. A technical findings report, an internal audit communication and a formal assurance engagement have different expectations. Agree the form before procurement buys an ambiguous “AI audit certificate”. Describe the actual engagement offered and the conclusion it can support. A technical assessment should not be represented as certification or an unrestricted legal compliance opinion.

Finally, establish who owns the decision and who owns the evidence. The sponsor may control the budget but lack authority to permit testing of a supplier service or access personal records. Scoping must identify those separate permissions. A commercial purchase order is not a substitute for the technical and data-access authorisations needed to perform the work responsibly.

Map the complete workflow before choosing the test boundary

For the hypothetical drafting assistant, map the approved source notice, document extraction, retrieval, instructions, model response, editing interface, approval event and distribution system. Mark where a human can change the text and where a system can initiate an external action. The audit need not examine every component in depth, but it needs enough understanding to avoid an accidental gap between the tested output and the real business outcome.

Distinguish component ownership from audit inclusion. A third-party model may be within the functional boundary even though its internal training process is inaccessible. A distribution service may be outside the commissioned assessment but materially affect the decision. Record both facts. “Out of scope” should identify a deliberate boundary, not imply that the excluded component is irrelevant or risk-free.

Illustrative boundary map for a recall-notice drafting assessment

Workflow element	Proposed audit treatment	Evidence needed	Unresolved implication
Approved source notice	Check identity, revision and authority for selected cases	Controlled source reference and approval record	The audit does not independently determine product-recall obligations.
Extraction and retrieval	Trace source content presented to the model	Selected source-to-input records and configuration	Unavailable intermediate text limits causal diagnosis.
Draft generation	Test preservation of agreed required information	Versioned inputs, outputs and adjudication rubric	Selected tests do not establish universal correctness.
Human approval	Examine permissions and selected operating evidence	Approval events, role configuration and change history	A screen demonstration alone does not establish period-wide operation.
External distribution	Exclude live sending; inspect authorised non-production evidence if agreed	Test environment and bounded execution records	No conclusion on every production delivery outcome.

The boundary map should identify interfaces even where the adjoining component is excluded. Otherwise an assessor may verify that a draft is marked “awaiting approval” while missing a separate export path that allows distribution. The correct response is not automatically to expand the audit to every system. It is to decide whether the interface must be included, examined through alternative evidence or disclosed as a limitation affecting the requested conclusion.

Agree criteria that can be observed and challenged

Criteria define what acceptable behaviour means for the engagement. Start with applicable obligations and approved organisational requirements, then identify any additional internal criteria needed for the specific use case. Record the source, owner, version and applicability rationale. Separate a binding requirement from voluntary guidance and from a proposed internal benchmark. The assessor should not present a preferred architecture as a mandatory external rule.

For the drafting example, a criterion could require preservation of identified product references and required contact instructions from an approved source. Define how omissions, contradictions and justified paraphrases will be judged. A criterion such as “no hallucinations” is too vague unless the engagement specifies what counts as an unsupported statement, which source governs and how ambiguous cases are resolved.

Discuss tolerances before observing results. Some departures may require immediate escalation; others may support a measured quality assessment. The appropriate thresholds depend on the actual decision and consequences. Any numerical threshold introduced for the engagement should be labelled as an agreed assumption or internal requirement, not attributed to a regulator without a verified basis. Record who has authority to approve it.

NIST SP 800-53A Rev. 5 provides a customisable approach to assessing security and privacy controls and developing assessment plans. Its existence does not make every procedure applicable to every Saudi AI system. Use relevant primary guidance as a reference, then document the selected criteria and methods for the actual engagement. The scope should explain the selection rather than merely list framework names.

Specify evidence access before committing to a conclusion

Prepare an evidence feasibility check while the scope is still negotiable. Can the organisation identify the tested release? Can it obtain the original inputs and outputs? Are approval events recorded? Can the assessor select cases independently? Are retention windows long enough to cover the proposed period? A report requirement that depends on unavailable evidence creates a predictable limitation, not an issue to discover on the last day of fieldwork.

Separate direct access, supervised inspection, supplier-provided extracts and management assertions. Each may be useful, but they provide different opportunities to test authenticity and completeness. A supplier's report may cover a shared platform while excluding the customer's configuration. Supplier evidence review helps assess that relationship. Record whether alternative evidence can answer the same question or only a narrower one.

Where the supplier controls access, agree an escalation route and a fallback conclusion. A restricted-access audit may still examine observable behaviour, contractual controls and customer-side records. It should not pretend to inspect hidden internals. Use a strategy for vendor-controlled systems to define feasible procedures and the limitations that remain. Commercial confidentiality can justify controlled access arrangements; it does not automatically make an unsupported claim acceptable.

State the evidence owners, permitted locations, transfer method and access duration. Minimise personal and confidential material to what the procedures require. For initial scoping, masked examples and structural descriptions are usually more appropriate than a bulk production export. If a procedure requires sensitive evidence, obtain the relevant authorisation and handling arrangements before collection rather than relying on informal email circulation.

Set populations, versions and test environments

Define the population in terms of actual tasks or events. For example, all eligible draft notices created under specified versions during a stated period. Identify cancelled drafts, incomplete cases and records whose final outcome is not yet available. Explain whether the assessment concerns the AI draft, the approved notice or the distributed message. A count of model calls does not automatically represent a count of business cases.

Choose a selection approach appropriate to the objective. Probability sampling, structured coverage and targeted challenge testing answer different questions. The scope need not contain every final test case, but it should state the intended inference and the basis for determining sample effort. The sampling approach should preserve that distinction, especially where Arabic and English cases have different volumes or operational complexity.

Identify the environments in which tests may occur and the actions permitted there. A non-production environment may allow safer inspection, but it may differ from production permissions, integrations or data. Record those differences and their effect on the conclusion. Testing a sandbox cannot establish a production control unless the relationship between the environments is adequately evidenced.

Define treatment of changes during fieldwork. Material changes to instructions, sources, permissions or the model may require separate test groups or a revised scope. A release freeze can simplify interpretation but may be impractical; an agreed change-notification process can be more realistic. Preserve the tested version identifiers and decide who can authorise retesting or additional work when the boundary changes.

Write rules of engagement for active testing

Active testing needs explicit authorisation, permitted targets, accounts, environments and techniques. Identify prohibited actions, rate or workload limits where relevant, stop conditions and the contact who can suspend testing. The recall example should not send real notices or alter approved source documents during an audit exercise unless a separately authorised procedure specifically permits that action. Synthetic or masked fixtures can test many boundaries without exposing live recipients.

Plan what happens when testing reveals a potentially serious issue. Establish immediate notification, evidence preservation and containment responsibilities without requiring the assessor to keep testing until the sample is complete. Distinguish a test-induced event from a previously existing incident. Record what occurred and involve the responsible incident process where appropriate. The audit scope should make this operational handoff possible before a problem arises.

Technical depth should match the agreed questions and available competence. An AI security assessment may need specialised boundary tests; a content-preservation review needs domain and language expertise. Neither can be substituted for the other merely because both involve a language model. Name required competence and reviewer roles so the work is not reduced to running a generic scanner and attaching its output.

Use a scope agreement with decision-ready fields

Original scope-agreement fields and completion prompts

Field	What a usable entry must establish
Decision and audience	The management decision, intended readers and reporting form.
Subject and boundary	Use case, system components, interfaces, languages, environments and exclusions.
Period and versions	Operating window, evidence cut-off, release identifiers and change treatment.
Criteria	Requirement sources, applicability, approved internal criteria and adjudication rules.
Procedures and inference	Design review, operating tests, sample approach and intended limits of generalisation.
Evidence feasibility	Owners, availability, independence of selection and alternative procedures.
Authorisation and handling	Permitted actions, test accounts, data access, storage, transfer and stop conditions.
Outputs and follow-up	Report, findings record, evidence index, management response and retest expectations.
Change control	Who approves scope changes, their effect on effort and the resulting reporting limitations.

Do not leave a critical field as “to be confirmed” while promising a conclusion that depends on it. A staged engagement can resolve access and criteria first, then confirm the executable test plan. That is a useful commercial arrangement when uncertainty is real. The initial stage should have a defined output, such as a reconciled boundary and evidence feasibility decision, rather than an open-ended discovery label.

Agree deliverable ownership and access. The sponsor may receive the report, while detailed working papers or supplier-restricted evidence have different distribution terms. Clarify those terms before fieldwork so the buyer knows what can be used for governance, procurement or later retesting. Avoid promising unrestricted ownership of evidence that another party lawfully controls.

Control scope changes without suppressing important findings

New facts can justify new work. If a draft-only assistant has an undocumented send capability, the original boundary may no longer support the intended conclusion. Record the discovery, assess its significance and propose a change, alternative procedure or explicit limitation. A fixed budget should not force the assessor to ignore a material interface; equally, every interesting technical detail does not justify expanding the engagement.

Use a change record that describes the new question, evidence required, effort and calendar implications, authorisation and effect on the report. Distinguish optional improvement advice from work necessary to answer the original question. This helps the sponsor make an informed decision and prevents an assessment from becoming an unbounded implementation project. Keep the original scope version so the evolution remains visible.

Reconcile the completed work against the agreed objectives before reporting. The coverage record should identify completed, partially completed and untested objectives with reasons. A limitation may be an appropriate final result when evidence cannot be obtained. It is not acceptable to replace the untested objective with a different, easier procedure and retain the original conclusion wording.

Resolve conflicting expectations before the engagement starts

Different stakeholders can use the same words for different objectives. Procurement may want evidence supporting supplier acceptance, operations may want a list of defects to correct, and the board may want a judgement about continued use. These needs can coexist, but the scope must show which are included and what each requires. A technically sound defect review may still disappoint the buyer if the expected governance question was never commissioned.

Use the hypothetical recall assistant to test the agreement. Ask each stakeholder whether the proposed report should answer whether drafts preserve required information, whether a release approval was recorded, whether every recipient received the notice, or whether the recall itself meets all applicable obligations. These are distinct questions involving different systems and expertise. The scope may include the first two, exclude delivery outcomes and refer legal applicability to the responsible qualified owner. Recording that division prevents an inaccurate all-purpose conclusion.

Also define the boundary between assessment and implementation advice. An assessor can explain a control gap and the outcome a correction must achieve. Designing and deploying the replacement control is additional work with different responsibilities and possible self-review considerations. If both services are contemplated, state how later review will be arranged. Do not let an apparently small recommendation silently become responsibility for operating the customer's system.

Confirm the acceptance conditions for the audit deliverable itself. Completion might require the agreed procedures, traceable findings, a coverage reconciliation and a management discussion; it should not depend on the system receiving a favourable result. A buyer should be able to accept a well-supported report that identifies serious issues or insufficient evidence. Making payment or acceptance conditional on a “pass” creates an expectation inconsistent with an evidence-led assessment.

Record these decisions before fieldwork so every participant can distinguish agreed work from later requests.

What to bring to a scoping discussion

Prepare a short workflow description, the decision to be supported, a component and interface list, available version information, languages, approximate task volumes and an evidence availability summary. Include known restrictions and changes rather than trying to present a perfectly documented system. A truthful gap list helps determine feasible work. Mask examples and avoid sending credentials or personal records through an initial enquiry.

Ask the assessor to explain what each proposed procedure can establish, what it cannot establish and what evidence would change that boundary. Request clarity on review competence, authorisation, handling, reporting and follow-up. Those answers are more useful than the largest possible list of framework acronyms. The working-paper approach should make the eventual reasoning reviewable, not merely produce an attractive report.

OpenForAI's AI auditing service is a starting point for a scoped evidence-led technical assessment. The appropriate scope depends on the actual workflow, available evidence and decision. Discuss the boundaries and evidence needs of your AI audit.