AI Audit Maturity Model

An AI audit maturity model should assess the organisation's ability to perform credible, repeatable and useful audits of AI systems. It should not confuse that capability with the accuracy of a model or the absence of findings. A mature audit programme can discover serious weaknesses precisely because it asks better questions and obtains stronger evidence. A polished dashboard with few findings may instead reflect narrow coverage or weak access.

This guide proposes an original internal maturity model for audit executives, risk leaders and programme owners. Its levels are not NIST or ISO levels, a regulatory rating or a certification scheme. The model assesses the audit programme, not every AI system it reviews. A hypothetical cross-branch donation-receipt classification programme illustrates the assessment without making claims about charity regulation, tax treatment or real organisations.

Define the programme being assessed

Start with the organisational boundary: which teams conduct AI audits, which systems they cover, which decisions they support and which reporting arrangements apply. A central audit function, a technical review team and a supplier assurance programme may have different responsibilities. The maturity assessment should identify the capability being judged rather than assign one label to the entire organisation because one team has an advanced tool.

In the hypothetical example, an organisation uses assistants to classify donation-receipt documents for internal routing at several branches. The audit programme examines source handling, classification, review and change controls. It does not decide the legal or tax status of a donation. Local workflows and document formats differ, so a central procedure must be examined alongside branch-specific practice.

State the purpose of the maturity review. It might guide investment in specialist review, improve evidence access, strengthen follow-up or decide how much reliance to place on a shared audit process. The result should lead to specific capability improvements. A target such as “reach the highest level” is less useful than identifying the decisions the programme currently cannot support and the reasons why.

NIST's AI Risk Management Framework is intended for voluntary use in managing AI risks. It is a separate reference from this original audit-programme maturity proposal. A review informed by the AI RMF does not create NIST certification, and the levels below should never be represented as an official NIST maturity scale.

Use levels as evidence descriptions, not prestige labels

Original proposed levels for an AI audit programme

Level	Capability description	Evidence needed
1: Reactive	Reviews depend heavily on individual effort and are often triggered by incidents or urgent requests.	Examples of actual reviews, with visible gaps in scope, records or follow-up.
2: Repeatable	Core procedures, ownership and records are defined and used across relevant engagements.	More than one completed engagement showing consistent use, not merely a newly written manual.
3: Reviewable	Methods, evidence and conclusions can be challenged and reconstructed by competent reviewers.	Reviewed workpapers, explicit limitations, suitable specialist input and evidence-based closure decisions.
4: Adaptive	The programme changes its methods and coverage in response to system changes, review findings and observed outcomes.	Documented learning decisions, implemented method changes and evidence that those changes improve relevant capability.

Use “not assessed” when the evidence is insufficient to assign a level. It is not level zero and should not automatically be interpreted as the lowest capability. An unknown state needs an evidence request or a scope limitation. Distinguish it from evidence that a capability is absent. Otherwise the assessment rewards teams that make confident claims while penalising those that disclose uncertainty honestly.

Do not assume that a higher level is always the immediate priority for every dimension. A small programme may first need reliable scoping and evidence handling before investing in automated continuous testing. A critical self-review conflict may require attention even when other dimensions are advanced. The appropriate improvement sequence depends on the decisions, exposure and constraints of the organisation.

Assess distinct capabilities separately

The proposed model uses seven dimensions: coverage planning, criteria and methods, evidence quality, competence, review and independence, remediation follow-up, and learning from change. Keep the profile visible rather than collapsing it into a single average. Strong automation cannot compensate for an inability to obtain source evidence, and excellent documentation cannot compensate for a material conflict in who verifies their own work.

Original capability dimensions and assessment questions

Dimension	Question to examine	Evidence beyond policy
Coverage planning	Does the programme identify material AI uses, shared controls and local differences?	Inventory-to-audit mapping, prioritisation decisions and documented exclusions.
Criteria and methods	Are questions, criteria and procedures matched to each use case?	Executed plans showing why procedures support the intended conclusions.
Evidence quality	Can the programme obtain, preserve and evaluate sufficient relevant evidence?	Reconciled sources, controlled references, limitations and reconstructable analysis.
Competence	Are technical, statistical, language and domain skills available when needed?	Actual specialist involvement and reviewed judgements, not training attendance alone.
Review and independence	Can conclusions be challenged without unmanaged self-review or commercial pressure?	Conflict records, review questions, responses and appropriate separation.
Remediation follow-up	Are findings resolved through evidence rather than owner assertion?	Action records, retests, accepted-risk decisions and recurrence investigation.
Learning from change	Do new systems, incidents and review results change the audit approach?	Method revisions, coverage changes and evaluation of their effects.

These dimensions are proposed internal categories, not an exhaustive universal standard. Add or refine dimensions where the programme's mandate requires them. Preserve definitions so a later assessment can distinguish real progress from a change in the scoring language. If the model itself changes, record how earlier profiles remain comparable or why they do not.

Collect evidence from completed work

Inspect a purposeful selection of completed engagements, including different systems, teams and outcomes. Include at least some difficult cases: restricted access, disputed findings, unsuccessful retests or material changes during fieldwork. A showcase engagement may demonstrate what the team can do under favourable conditions without showing what it normally does. State the selection rationale and limits.

Interview programme owners and practitioners, but corroborate their accounts with records. A claim that all AI audits use a consistent method should be tested against actual plans and workpapers. A claim that all material findings are independently verified should be examined through closure records. The maturity review is itself an assessment requiring evidence; it should not be a self-reported survey dressed as a definitive rating.

For each dimension, record the supported level, evidence confidence, important variation and the next improvement. Avoid forcing a level where evidence is missing. A programme can have a well-defined method that is used inconsistently across branches. The assessment should describe that variation rather than assigning the best branch's capability to the whole organisation.

Worked profile: cross-branch receipt classification audits

Consider the hypothetical programme with three branches. The central team has a common audit plan, but branch practices differ in source preservation, Arabic review and closure evidence. The following profile is illustrative. It does not describe actual performance, and its level assignments are not benchmark statistics for Saudi organisations or the nonprofit sector.

Illustrative programme profile with evidence-based improvement priorities

Dimension	Supported level	Observed basis	Next improvement
Coverage planning	2: Repeatable	All three branches are listed, but local manual export paths are not consistently included.	Reconcile workflow interfaces before the next coverage decision.
Criteria and methods	2: Repeatable	Common procedures exist, with inconsistent explanation of why selected cases answer the audit question.	Add case-selection rationale and inference limits.
Evidence quality	1: Reactive	One branch preserves source-output pairs; others rely on current screenshots after records change.	Establish authorised capture and version references across the assessed scope.
Competence	2: Repeatable	Arabic review is available, but specialist involvement is not consistently recorded.	Document who reviewed consequential interpretation and with what remit.
Review and independence	3: Reviewable	Selected engagements show substantive challenge by reviewers separate from implementation.	Check that this arrangement also covers supplier-dependent work.
Remediation follow-up	1: Reactive	Some findings are marked closed on owner confirmation without retest evidence.	Define closure dispositions and evidence requirements.
Learning from change	Not assessed	Insufficient records to determine whether lessons changed the method.	Locate decision history before assigning a level.

The priority is not to average these values into a score. Evidence quality and closure practice limit the programme's ability to support decisions even where review arrangements are stronger. Addressing those gaps may be more valuable than adding new tools. “Not assessed” remains an evidence limitation, and should not be converted into a numeric value merely to make a chart complete.

Define progression with observable conditions

For coverage planning, progression might mean moving from an informal system list to a reconciled view of material uses and interfaces, then to reviewable prioritisation and change-triggered updates. Evidence should show actual decisions, including why some systems were deferred. A comprehensive inventory is useful, but it does not establish that audit coverage is adequate unless the programme explains how priorities were selected.

For evidence quality, progression might mean consistent source identification, controlled collection, reproducible analysis and active challenge of completeness. Inspect whether a reviewer can reconstruct selected findings through working papers. More stored data is not necessarily better maturity; unnecessary copies can increase risk while adding little support. The capability concerns appropriate evidence and sound interpretation.

For follow-up, progression might mean moving from informal owner updates to recorded actions, explicit closure criteria and evidence-based retesting. At a more adaptive stage, the programme examines recurrence and changes its methods when closure failures reveal a weakness. A dashboard showing all actions complete does not demonstrate that this learning occurs.

Keep programme maturity separate from system risk

A capable audit team may assess a high-risk or poorly controlled system. Its ability to identify and communicate that problem can be evidence of programme strength. Conversely, a low-risk system with few observed exceptions does not prove that the audit process is mature. Evaluate the method, evidence and decision usefulness independently of whether the result is commercially comfortable.

Do not compare branches solely by the number of findings. Broader coverage and better detection can increase findings. Limited access or narrow testing can reduce them. Interpret counts alongside scope, evidence and severity. A branch that reports a difficult limitation honestly may demonstrate better judgement than one that issues an unsupported positive conclusion.

Separate maturity from readiness for a particular engagement. A programme with strong long-term capability may still lack access to a specific supplier's records. A newly formed team may prepare well for one bounded audit without having repeatable programme-wide practices. Both statements can be true. The maturity profile should describe the organisational capability actually evidenced.

Assess independence and competence as operating practices

A policy stating that reviewers are independent is not enough. Examine who performed the work, who implemented the system, who challenged the conclusion and what happened when management disagreed. Self-review and independence require engagement-specific consideration. A small team can use proportionate safeguards, but should not claim separation that its actual roles do not support.

Training records show attendance or completion, not necessarily the ability to evaluate a difficult AI finding. Examine how the programme obtains specialist input and reviews consequential judgements. In the receipt-classification example, competent Arabic interpretation and understanding of the internal routing categories matter. The maturity assessment should not invent professional credentials or assume that general AI familiarity covers every required skill.

Look for resilience when individuals leave or workloads rise. If a single specialist holds the entire method in memory, the programme may perform well temporarily but remain fragile. Documented procedures, reviewed examples and planned access to expertise can improve repeatability. The objective is not to eliminate professional judgement, but to make its basis reviewable and available when needed.

Build an improvement plan from the limiting capabilities

Choose a small number of improvements that address the most consequential capability gaps. For the hypothetical programme, the first priority could be preserving source-output evidence and replacing assertion-based closure. Assign owners, dependencies and completion evidence. Avoid a broad initiative called “improve AI governance” without a specific change to how audits are performed or decisions supported.

Define success through use in actual work. Publishing a new evidence procedure is an implementation milestone. Demonstrating that several relevant engagements use it correctly, with reviewer challenge and disclosed limits, provides stronger support for progression. Set an appropriate review window rather than claiming maturity immediately after a document is approved.

Use clear engagement scoping to improve the relationship between questions and methods. For larger organisations, enterprise audit coverage helps distinguish shared controls from local deployment differences. These improvements should be connected to the programme's real constraints, such as supplier access or specialist capacity, rather than copied from a generic transformation roadmap.

Use automation where it strengthens evidence and review

Automated collection, checks and reporting can improve consistency, but they do not automatically confer a higher maturity level. The programme needs to understand what a tool measures, its failure modes and how exceptions are reviewed. A continuous feed of weak indicators can produce more data without improving audit conclusions.

Continuous AI auditing requires more than operational monitoring. Assess whether the programme independently examines the signals, changes coverage when needed and follows issues to a supported disposition. Automated checks may support those activities, while human judgement remains necessary for scope, criteria, significance and limitations.

Evaluate the audit tools themselves proportionately. If an AI assistant drafts findings or classifies evidence, verify the resulting statements and source references. A mature programme should not accept unsupported content merely because it came from a tool intended to improve audit efficiency. Record material tool limitations and ensure reviewers can challenge the output.

Use a progression decision card

A progression decision should explain what changed in practice and why the evidence supports a different capability description. For the hypothetical programme, introducing a source-preservation procedure is not enough to move the evidence dimension upward. The reviewer needs to see its authorised use in relevant branch engagements, the resulting evidence links and how missing or restricted records were handled. A failed implementation attempt should remain visible as part of the learning record.

Original progression decision card for evidence quality

Field	Illustrative completion
Capability gap	Changing source records prevent reconstruction of selected findings.
Improvement introduced	Authorised source capture, version references and controlled derivatives.
Operating evidence	Completed engagements from the relevant branches showing use and reviewer challenge.
Contrary evidence	Record cases where capture failed or a source remained unavailable.
Decision	Assign the supported level only for the evidenced scope and preserve remaining variation.
Next review trigger	Material change in source systems, branch coverage or evidence access.

Record who assessed the evidence and whether they designed the improvement. Where the same team owns both activities, arrange suitable challenge and disclose the overlap. The maturity review should apply the same care about self-review that it expects from ordinary audits. An improvement sponsor's enthusiasm is not independent evidence that the capability is operating consistently.

Do not promise a level increase on a fixed date. The calendar can schedule a review, but the evidence determines the outcome. If the programme has not yet completed relevant work, report implementation progress and defer the capability judgement. This preserves a useful distinction between a plan, a deployed process and demonstrated repeatability.

Reassess without turning the model into a competition

Repeat the maturity review when the programme changes materially or after agreed improvements have produced enough operating evidence. Preserve the earlier profile, method version and scope. Explain whether an apparent change reflects stronger capability, broader assessment or different evidence. A newly discovered gap may lower a rating while improving the organisation's understanding.

Present the profile to leadership with the supported level, confidence, limiting evidence and proposed decision for each dimension. Avoid an unsupported claim that the organisation is “world-class” or certified mature. The useful result is a clear account of what the programme can currently substantiate and what will improve its ability to support decisions.

For a scoped AI audit or programme review, bring the mandate, selected completed engagements and current follow-up records. Discuss an evidence-based review of your AI audit programme's maturity.