AI Audit and Assurance

An AI assurance review should make a claim more trustworthy, or show precisely why it cannot yet be trusted. The claim might be that a knowledge assistant respects document permissions, that an agent cannot exceed a transaction limit, or that a service meets agreed quality criteria for its intended users. “We audited the AI” says little unless the scope, criteria, evidence and limitations are visible.

This guide is for an audit committee sponsor, internal auditor, risk leader or service owner commissioning a review. It explains how to frame an inspectable engagement and use its findings. It does not treat a technical assessment, a statutory audit and a certification as interchangeable services.

Choose the question the review must answer

Start with a claim-and-criteria statement. For example: “For the configured employee policy assistant, users must receive only information they are authorised to access, across the agreed document sources and user roles.” Specify the version, environment, population and period. If the service is pre-release, state that the review concerns readiness evidence rather than demonstrated operation over time.

Separate three useful review purposes. A design review asks whether the proposed controls address the identified risks. An implementation review checks whether the configured service contains those controls. An operating review examines whether they worked during a defined period. A project can pass the first and fail the second; a successful demonstration cannot establish the third.

Agree the intended audience and decision: release approval, remediation prioritisation, supplier selection or assurance to leadership. This determines the work needed and prevents a narrow report being reused for a much broader claim. Material board approval decisions should state what evidence directors actually require.

Build a claim-to-evidence plan

Example assurance plan for an employee policy assistant

Claim	Criteria	Evidence and procedure	Limit to disclose
Access boundaries hold	Permitted roles receive only permitted source material	Role mapping, configuration inspection and negative access tests	Coverage of roles, sources and environments
Answers are supported	Material statements match the approved policy version	Blind review of a defined question set against source documents	Questions and policy versions not tested
Escalation works	Unresolved questions reach an authorised support route	Observed handover exercise and case follow-up	Peak-demand performance if not exercised
Changes are controlled	Material changes receive the required review before release	Sample change records and deployed-version reconciliation	Period and releases covered

The NIST AI RMF Measure guidance emphasises suitable measurement methods and documenting risks that cannot be measured. Use that discipline to distinguish tested evidence from assumptions. The table is an original engagement aid, not a prescribed NIST audit procedure.

For each claim, identify evidence supplied by the team, evidence independently obtained and evidence unavailable to the reviewer. A screenshot, written policy and observed test answer different questions. Assess their relevance and reliability rather than counting documents.

Define independence before appointing the reviewer

Ask who designed, built, approved and operates the assessed system. If the same team reviews its own work, describe the exercise as an internal review and disclose the relationship. It can still improve quality, but its limitations matter to the audience.

For stronger independent challenge, separate the reviewer from the decisions under examination and give them access to the necessary evidence and reporting channel. Agree how disagreements will be recorded and who receives unresolved findings. A reviewer whose report can be edited by the delivery owner without trace has a different position from one reporting directly to the commissioning authority.

Open for AI provides both advisory and delivery services. Buyers should agree conflict disclosures and separation of responsibilities where relevant; independence should never be assumed from the word “consulting”. If the buyer needs a formal assurance opinion or certification, confirm the provider's qualifications, mandate and applicable engagement terms for that specific service.

Scope sampling around the risks you need to challenge

A test set should include ordinary cases, boundary cases and deliberate attempts to violate the claim. For the policy assistant, ordinary questions may establish useful answers while cross-role requests challenge access restrictions. Include supported Arabic and English use where relevant to the service, rather than assuming one language's results transfer unchanged.

Record how cases were selected. A risk-focused sample can find important weaknesses but should not be presented as an unbiased estimate of population failure rates. A random sample may estimate a defined rate, yet miss a rare severe scenario. Choose the method that answers the review question and make the limitation explicit.

Prevent the service team from seeing every evaluation answer in advance. Keep a suitable separation between tuning examples and review cases. Record configuration changes during testing. If the system is altered after a failure, distinguish the original result, remediation and retest instead of overwriting the evidence.

Write findings that lead to a decision

A useful finding has a condition, criterion, evidence, consequence and recommendation. It should also identify the affected scope and the owner's response. Avoid describing a hypothetical scenario as an observed failure, or treating an observation as proof of a root cause that has not been investigated.

In an illustrative review, two of eighteen deliberately constructed access tests return extracts from restricted payroll policies to an unauthorised role. The reviewer can state that the access claim failed in those tested conditions. They cannot infer the frequency across every employee question from that targeted sample.

The finding should identify the test cases and configurations, describe the information exposed and require correction and retesting before the affected access path is accepted. It may recommend temporarily removing that source while the issue is investigated. The final risk decision belongs to the authorised owner, with any applicable requirements respected; the auditor does not become the service operator.

Link findings to the AI risk register where they change risk understanding or treatment. Keep routine enhancement suggestions separate so the reader can see which findings affect the release decision.

Use supplier reports without stretching their meaning

Read the report's entity, product, region, period, control scope, exceptions and customer responsibilities. A supplier's assurance over infrastructure may not cover your retrieval permissions, business rules or user experience. Ask what additional evidence the deployment requires.

During vendor due diligence, record which claims the supplier's report supports and which remain unverified. Where NCA controls apply, have the relevant owner map the evidence to the correct requirements. Do not translate a general report into a blanket Saudi compliance statement.

For an acquisition, separate a service assessment from the broader investment questions in AI due diligence for M&A. A working control today does not establish ownership of intellectual property, sustainable economics or the transferability of a supplier agreement.

Agree how findings close and reports remain usable

Before fieldwork, define what closes a finding. For a failed access control, a rewritten policy is insufficient: the relevant configuration and tests need review. Record the fix, new evidence, retest result and person authorising closure. If the scope has been reduced instead, say so explicitly.

Report unresolved limitations prominently. “Unable to obtain evidence” differs from “control failed” and from “control not applicable”. Each has a distinct implication for the commissioning decision. Include an executive conclusion that a reader can trace to the findings without relying on unexplained colour ratings.

At production handover, preserve the assessed baseline and define the changes that require further review. A report is dated evidence about a defined arrangement, not a permanent guarantee. New sources, permissions or material operating changes can invalidate its assumptions.

Commission the review with a clear evidence request

Prepare a brief naming the service, intended claim, release stage, audience, available evidence and required independence. Ask the provider to state procedures, access needs, deliverables, exclusions and how disagreements will be handled. Compare proposals on whether they answer the decision, rather than on the size of the promised checklist.

Open for AI's agentic AI consulting can help define system boundaries and evaluation requirements. Bring a redacted assurance question, architecture and sample evidence to discuss a properly scoped AI review. Agree the specific review role and any independence requirements before commissioning work.