AI Audit under SAMA

An AI audit under SAMA should begin with the regulated entity, the activity and the applicable requirements. “SAMA compliant AI” is too broad to be a useful conclusion. A bank's call-quality screening service, a fraud-detection model and a model used for regulatory capital have different purposes, consequences and evidence needs. The audit must establish which obligations apply before deciding what to test or how to describe the result.

This guide helps Saudi bank compliance teams, cybersecurity leaders, internal audit functions and supplier owners build that applicability and evidence map. Its worked example is a hypothetical supplier-operated service that screens customer calls for quality-review indicators. The service prioritises calls for authorised human review; it does not decide a customer's rights, approve financial transactions or independently determine employee disciplinary action.

Replace the broad compliance label with an auditable question

A useful question could be: “For the named bank, service and review period, were the applicable supplier cybersecurity requirements reflected in the arrangement and supported by operating evidence?” Another could ask whether the quality-review workflow preserves the required human decision and evidence trail. These questions are narrower than a universal compliance claim, but they can produce decisions management can act on.

Define the legal entity and its regulatory status, the activity performed, the affected records and people, the service providers and the review period. Identify whether the system supports a regulated activity directly or provides an internal operational function. That distinction informs the assessment; it does not automatically remove an internal tool from cybersecurity, privacy or outsourcing considerations.

Record who owns interpretation of the applicable requirements. The compliance function should identify relevant current instruments and any entity-specific directions, while technical and business specialists explain the actual service. The auditor evaluates the evidence against agreed criteria. If an interpretation remains unresolved, show it in the scope rather than allowing a technical checklist to decide it silently.

A sales proposal, vendor certification or internal model label should not determine the legal perimeter. An externally supplied tool can still be configured by the bank, and an internal application can depend on several external services. The arrangement needs to be understood as it operates, including support, administration and subcontracted processing, before its controls can be assessed.

Build a requirements register with sources and owners

SAMA's Cyber Security Framework section 3.4.1 addresses contract and vendor cybersecurity requirements, including review and audit rights. Its outsourcing section 3.4.2 includes SAMA approval before material outsourcing. The bank must determine the applicable arrangement and current requirements; these references do not classify every AI purchase as material outsourcing.

Create one register entry per relevant obligation or approved internal criterion. Include the authoritative source, version or effective date, applicability rationale, accountable owner, expected evidence and planned procedure. Keep mandatory requirements distinct from internal risk choices and supplier promises. An internal threshold can be stricter than a general requirement, but the report should explain where it came from.

Original applicability register for a bank call-quality screening service

Question	Responsible decision	Evidence to connect	Common audit error
Which entity and activity are covered?	Compliance confirms the regulatory perimeter	Entity record, service description and current applicability memorandum	Applying a rule from another institution type without checking scope
How is the supplier arrangement classified?	Responsible outsourcing and compliance owners assess it	Materiality decision, approvals where applicable and signed service scope	Treating all subscriptions as identical arrangements
Which security responsibilities remain local?	Cybersecurity and service owners allocate controls	Responsibility matrix, configuration and relevant supplier evidence	Assuming the supplier's report covers customer permissions
What may the AI output influence?	Business and risk owners approve permitted use	Review policy, queue rules and human disposition records	Auditing a recommendation while ignoring the downstream action
What additional processing rules apply?	Privacy and relevant specialists document their assessment	Data-flow and processing decisions with linked evidence	Using a cybersecurity result as a complete privacy conclusion

The register should be reviewable before fieldwork. If a source is superseded, retain the historical version where it explains the period examined and use the current version for current obligations. Do not replace the criteria halfway through a test without documenting the effect. This is especially important when comparing historical operating evidence with a recently improved supplier contract.

Keep specialist frameworks within their actual scope

SAMA's Counter-Fraud Framework section 3.8 specifically addresses auditability of AI and machine learning used in that context. That is relevant when the scoped activity is counter-fraud. It is not evidence that the same provision governs every call-quality or marketing model. The applicability register should preserve that activity boundary.

Likewise, bank credit-model validation can involve specialised requirements and statistical evidence. A review of supplier access controls cannot substitute for validating a model's estimates, and a model-validation report does not establish every operational security control. Use model validation for banks for that distinct question and AI auditing for banking for the broader operating workflow.

Do not infer a universal annual AI audit requirement from a frequency attached to another activity, report or control. Determine the exact required review, the relevant system and the stated frequency. Where the bank chooses additional risk-based reviews, label them as its own programme. Clear attribution makes the audit easier to defend and prevents unnecessary work created by overbroad interpretations.

Trace the supplier service from call intake to review outcome

In the hypothetical arrangement, approved call recordings enter a bank-controlled intake process. A supplier service creates a transcript and identifies possible quality-review issues. Selected calls enter a reviewer queue, and an authorised employee examines the source evidence before recording a disposition. The audit should map each transition, including unsuccessful processing, low-confidence results, reprocessing and manual overrides.

Separate the screening result from the final review. An AI flag is not a confirmed breach of procedure. A human decision is not automatically reliable merely because a name appears in the workflow. The reviewer needs access to the relevant source, the applicable quality criterion and a way to correct or reject the flag. The system should preserve enough evidence to determine what information was available at the decision point.

Inspect where the result is used after review. A dashboard might aggregate flags into team scores, or another workflow might import an unreviewed indicator. Those downstream uses can change the consequence of an error. If the engagement excludes them, say so; otherwise include the relevant interfaces and decision owners. Auditing only the supplier's output can miss the point where a tentative signal becomes an operational fact.

For Arabic and English calls, record the supported language conditions, transcription process and reviewer competence. Mixed-language speech, names, background noise and incomplete recordings can affect the evidence. These are proposed test dimensions, not claims that a particular vendor fails them. The audit should examine the actual population and avoid generalising from a small polished demonstration.

Worked test: a supplier update changes the meaning of a flag

Assume the bank's approved quality rule distinguishes an omitted disclosure from a disclosure made later in the same call. A hypothetical supplier update begins flagging both situations identically. The audit question is whether the bank identifies the change, evaluates it against the approved rule and prevents unreviewed flags from being treated as confirmed findings. This tests the interface between vendor change management and bank oversight.

1. Obtain the approved rule version and the bank's acceptance criteria for the screening function.
2. Create authorised synthetic call transcripts representing an omission, a later disclosure and an ambiguous incomplete record.
3. Record the supplier model or service version, customer configuration and input-processing settings.
4. Run the permitted tests and capture the flag, explanation and evidence presented to the reviewer.
5. Trace the cases through the human queue and any downstream reporting that is within scope.
6. Compare the observed behaviour with the supplier change notice, bank impact assessment and approval evidence.

If the supplier does not expose an exact model identifier, document the strongest version evidence available, such as a dated release identifier and configuration export. Do not invent reproducibility. The report may establish behaviour observed on a particular date without proving that the same hidden model served every earlier request. That limitation matters when deciding how far to extend historical review.

A detected change can produce several findings. The screening rule may be wrong, the supplier notice may be insufficient, or the bank may have received the notice without assessing it. These are different control failures. Preserve the evidence for each and avoid attributing the cause to the model simply because the visible symptom appeared in an AI output.

Assess supplier evidence for relevance and customer responsibility

Request the service description, contractual cybersecurity commitments, audit-access terms, support model, incident arrangements and evidence of the relevant controls. For third-party assurance material, examine its period, service boundary, exclusions and customer responsibilities. A report covering one hosting service may not cover the transcription subcontractor or the customer-specific screening configuration.

Distinguish availability of a document from sufficiency of evidence. A supplier may provide a current policy while operating evidence is unavailable. A witnessed demonstration may show one control works in a test environment without establishing its operation in production. Record the evidence class and the conclusion it can support. Supplier evidence assessment provides a more detailed method for this judgement.

For a bank-owned control, obtain bank evidence. If the supplier requires customers to restrict administrative roles, inspect the actual role assignments and review records. If the bank must approve a new processing route, find the specific decision and configuration. Inherited controls and retained responsibilities should meet at an explicit boundary so neither party assumes the other performed the same task.

Where the supplier declines access, determine whether another form of evidence can answer the question. A supervised export or independent report may be sufficient for one objective but inadequate for another. State the unresolved question and its consequence. Do not claim a clean result by removing every inaccessible component from the scope after the engagement has begun.

Examine operating evidence across the agreed period

Identify populations such as supplier releases, access changes, screening exceptions, completed reviews and incident notifications. Reconcile them to reliable records before sampling. A list supplied by the service owner may exclude failed or abandoned cases, which can be important to the control question. Request the selection logic and account for missing, pending and duplicate items.

In an illustrative quarter with nine supplier releases, an auditor might examine all releases that changed language processing and a justified selection of other releases. The report should describe that selection rather than call it statistically representative without support. If only two releases can be reconstructed, show the resulting coverage limit. The number of screenshots collected does not compensate for an incomplete population.

For human review, distinguish a completed queue status from evidence of substantive review. Examine whether the source was available, the reviewer had the required authority, the recorded rationale addressed the issue and any correction reached downstream records. A fast approval may be appropriate for a simple case; timing alone is not proof of poor oversight. Use the actual decision evidence.

For performance, establish the reference judgement independently of the system being audited. If the same supplier labels the test set and scores its own output, assess that dependency. Record disagreements between qualified reviewers and how they were resolved. A screening accuracy percentage is not meaningful without the task definition, denominator, class balance and treatment of ambiguous calls.

Test escalation, restrictions and exit evidence

A bank should know what happens when the service produces unreliable flags or becomes unavailable. The audit can witness a safe exercise showing how the affected function is restricted, who decides on manual review and how pending cases remain visible. The proposed test should preserve normal banking operations and use authorised fixtures; it does not require disrupting customer calls.

Examine the escalation path between supplier operations, the bank service owner, cybersecurity, compliance and the relevant risk function. A contractual notification address is only useful if messages reach a monitored route and contain enough information to act. Test the contact process within agreed rules, and distinguish a simulated exercise from evidence about an actual incident.

Exit review should examine whether the bank can obtain required records in a usable form, end access and account for retained supplier data. A contract clause stating that data can be exported does not demonstrate that the export includes review history or version information. A masked trial export can reveal missing fields before the organisation depends on them during termination or a dispute.

Make the audit conclusion useful to compliance and management

Organise findings by the question they answer: applicability, arrangement approval, control design, operating effectiveness, model behaviour or evidence limitation. Avoid one blended percentage that treats these categories as interchangeable. An unresolved materiality assessment and an observed access-control failure may both require action, but their evidential status and accountable owners differ.

An illustrative finding could state that a supplier release changed the tested screening behaviour, while no bank impact assessment was available for that release. It should cite the version evidence, approved rule, test cases and records requested. It should not state that SAMA has rejected the service, that every prior flag was wrong or that a regulatory breach is established unless the relevant evidence and authorised interpretation support those claims.

Agree remediation evidence before closing the finding. Correcting the screening configuration may need a repeat test, while improving the supplier notification route may need an exercised change process. A policy update alone does not demonstrate both. The retesting plan should identify each closure claim and preserve any remaining restriction on use.

Management reporting should identify the decision now required, such as restricting an unverified output, obtaining missing supplier evidence or resolving applicability. State who can accept a residual risk and which matters need specialist or regulatory engagement. Risk acceptance is not permission to disregard an applicable requirement, and an auditor's report is not regulatory approval.

Preserve entity boundaries when services are shared

A group-wide supplier contract can create useful common evidence, but it does not erase differences between participating entities. One bank unit may use the screening service only to select calls for review, while another imports the flags into a staff dashboard. Record the actual use and local approvals for each scoped entity. Do not extend a conclusion from the first arrangement to the second merely because both share the same supplier account.

Use a shared-control record to identify the evidence that genuinely applies across deployments, then attach local configuration and operating evidence. If the central control fails, identify the potentially affected services without assuming that every service experienced the same consequence. If the local control fails, avoid describing the entire supplier platform as defective. This distinction supports proportionate escalation and prevents unnecessary duplication of testing.

Agree who receives cross-entity findings and how restricted evidence is shared. A central summary can reference evidence held by the relevant owner without copying customer recordings into a group-wide folder. The reporting design should preserve confidentiality while allowing authorised reviewers to understand the basis and limits of the common conclusion.

Prepare a focused SAMA-related audit brief

Before commissioning the review, assemble the entity and activity description, current applicability register, supplier classification decision, contracts, recent changes and evidence-access conditions. Add a masked example of a complete screening-to-review record. Explain whether the intended audience is internal audit, management, a procurement committee or another authorised recipient; the report form and required evidence may differ.

Separate this engagement from adjacent work that needs its own scope, including PDPL processing review, specialist bank model validation and independence assessment. The goal is coordinated coverage with explicit responsibilities. An AI auditing engagement can bring the evidence together while preserving those distinct conclusions.

Request an audit scoping discussion for a SAMA-related AI service review with the service purpose, supplier boundaries and the decision the bank needs to make. A defensible engagement specifies what will be tested, against which applicable criteria, by whom and with what access. That gives management a practical basis for action without promising a universal compliance certificate.