AI Audit for Banking

An AI audit for a Saudi bank should follow the point at which an AI result becomes a banking record, operational action or decision input. The audit needs to distinguish a model suggestion from a verified fact, and a completed workflow status from an authorised decision. Those distinctions matter even when the AI performs a modest task such as checking whether an onboarding document pack appears complete.

This guide is for banking operations, compliance, model-risk, cybersecurity and internal audit teams. Its hypothetical example is an AI assistant that extracts document types and identifies missing items during customer onboarding. The assistant does not establish identity, approve a customer, determine beneficial ownership or make a credit decision. The review examines whether those boundaries hold in the complete operating workflow.

Choose the bank workflow, not the generic AI category

Begin with the specific process and the decision it supports. Onboarding completeness, transaction monitoring, customer communications and regulatory credit models need different criteria. An organisation-wide AI inventory can help prioritise work, but the fieldwork scope should identify the actual product, channel, users, records, versions and review period.

SAMA's bank-account rules on KYC and AML/CFT describe customer knowledge and risk assessment responsibilities. The AML Implementing Regulation includes risk-based due-diligence and identity-verification measures. Detecting a document in a file is not equivalent to completing those obligations.

Have the bank's responsible functions map current requirements and internal policies to the scoped process. SAMA-related AI audit planning addresses that applicability work in detail. This page concentrates on the operating evidence: what the assistant proposes, what the bank verifies, who decides and what the downstream systems actually record.

Define the states that must remain distinct

In the example, an uploaded file moves through several states: received, readable, classified, assessed for completeness, reviewed and accepted for the relevant next processing step. These are proposed operational distinctions to tailor to the bank's process. None should be silently renamed identity verified or customer approved unless the authorised process and evidence establish that different decision.

Write a state dictionary that the operations, technology and compliance teams can all understand. A label such as accepted may mean that the upload passed a format check, that an employee accepted the classification or that a separate control approved the evidence. If one field is used for all three meanings, the audit should investigate the resulting ambiguity before measuring performance.

Map who or what can change each state. A model can propose a document category; a deterministic check may reject an unreadable file; an authorised reviewer may resolve an exception. Identify the separate authority for any regulated decision. The audit should inspect actual service permissions and integration mappings rather than infer authority from the wording of a user-interface button.

Original state-and-evidence map for AI-assisted onboarding completeness

State or claim	Evidence needed	What it must not imply
File received	Upload identity, customer context and receipt record	The document is genuine or sufficient
Document classified	Model output, confidence handling and source reference	The extracted identity has been independently verified
Pack appears complete	Applicable checklist version and item-level evidence	All due-diligence or acceptance decisions are complete
Exception reviewed	Authorised reviewer, rationale and disposition	Every other customer control has passed
Next step authorised	The bank's relevant decision and required supporting controls	An AI recommendation alone supplied that authority

Trace one document pack across system boundaries

Follow a synthetic customer pack from intake through storage, extraction, classification, completeness assessment, review and the destination system. Record the identifiers used to maintain the customer relationship and the version of the requirements checklist. Include retries, replacements and manual uploads where they occur within scope. The most important error can arise in a mapping or state transition rather than in the model's text output.

Check that a replacement document is associated with the intended customer and supersedes the correct prior item. A new upload can be readable and accurately classified while linked to an outdated request or another case. Preserve the distinction between the document's own identifier, the onboarding case and the customer's record. Test authorised cases that exercise those relationships without using real identities unnecessarily.

Inspect what downstream systems receive. A completeness flag might be mapped into a generic approval field, or a low-confidence classification might become a definitive document type after export. The audit should compare source semantics with destination semantics. An interface response saying success proves only the operation it actually confirms, not every later business outcome.

Worked test: prevent completeness from becoming customer approval

Assume the bank's approved workflow requires human handling when a document category is uncertain. A hypothetical integration maps the assistant's output pack_complete=true to a destination status that normally means the onboarding stage is approved. The audit question is whether this mapping allows the AI completeness result to bypass the intended decision. The test uses a fictional customer and authorised synthetic documents.

1. Record the approved checklist, destination-state definition and authority needed for the next step.
2. Create a controlled pack containing all expected file types but one deliberately ambiguous, harmless document specimen.
3. Run the proposed configuration and capture item-level classifications, confidence treatment and completeness output.
4. Trace the destination state and any next-step action through authorised read-only evidence.
5. Confirm whether the exception reaches the responsible reviewer before the relevant action is permitted.
6. Repeat after a controlled correction to establish the intended permitted path as well as the blocked path.

A blocked user-interface action does not prove the backend transition is blocked. Conversely, a queue entry marked approved may be an internal label with no authority to proceed. Establish the meaning and actual effect before concluding that a bypass exists. The finding should identify the tested path and evidence, not infer a regulatory breach solely from an unfortunate field name.

If the test identifies a genuine mapping defect, separate it from model classification quality. Improving the model will not necessarily correct an integration that treats every output as approval. The remediation might require a distinct state field, enforced authorisation and a reconciled destination check. Retesting should show that uncertain cases remain in the intended review route and that valid cases can still proceed appropriately.

Evaluate completeness against the right policy version

A document pack can be complete under one set of requirements and incomplete under another. The audit should identify how the applicable checklist is selected, including relevant customer type, product and process conditions determined by the bank. Do not invent a universal document list in the audit. Test the organisation's approved rules and their authorised interpretation.

Record the rule version used at the time of the decision. A later checklist update should not silently change the explanation of an earlier result. If a case remains open during a policy change, examine the bank's approved treatment of that transition. The technology should support the decision rather than assume that the latest checklist always applies retrospectively.

Use controlled variants to test whether the right rule is selected. For example, two fictional cases can have different approved process categories but similar-looking documents. The audit examines category selection and evidence traceability, not the merits of the bank's customer acceptance policy. Unclear policy conditions should be referred to the responsible owner rather than resolved by an improvised model instruction.

Measure errors with denominators that match the decision

Define the unit before reporting accuracy: file, extracted field, document item, complete pack or customer case. High field-level extraction accuracy does not establish pack-level correctness. One missing mandatory item can matter more to the process than several correctly extracted optional fields. Report relevant error categories and consequences rather than blend everything into one percentage.

In an illustrative test of 100 synthetic packs, 70 are complete and 30 incomplete according to the approved reference. Suppose the assistant flags 27 of the incomplete packs and incorrectly flags five complete packs. The incomplete-pack detection rate is 27 divided by 30, or 90%; the flagged set contains 32 packs, of which 27 are genuinely incomplete. Its precision is about 84.4%. These figures answer different questions.

Three incomplete packs were missed, and five complete packs would receive unnecessary review in this test. Their operating consequences depend on the downstream controls and workload. This purposeful example is not a market benchmark, a regulatory threshold or a forecast of production performance. Any population estimate requires an appropriate selection design and treatment of dependence, missing outcomes and changing case mix.

Examine the reference labels. Qualified staff should apply the agreed checklist consistently and document ambiguity. If reference reviewers disagree, do not silently force a label to improve the score. Evaluation integrity and sampling design help determine what the evidence can support and how uncertainty should be reported.

Test Arabic, English and document variation explicitly

Define the language and format conditions relevant to the actual channel. These may include Arabic and English documents, mixed-language fields, scanned images, rotated pages, missing pages and low-quality uploads. The test should distinguish unreadable material from a confident but incorrect extraction. A usable fallback route is often more important than forcing an answer for every file.

Use synthetic names, dates and identifiers with expected interpretations. Check preservation of leading zeros and the association between values and their labels. Where date conventions matter, the application should use the bank's approved handling rather than silently infer a calendar or format. The audit should report the conditions tested and avoid extrapolating to unsupported document families.

Review the explanation shown to staff. A correct missing-item flag can still be hard to resolve if it points to the wrong page or gives an English-only explanation in an Arabic workflow. Test whether the reviewer can locate the source and understand the uncertainty. Language coverage should include the review process, not only the extraction model.

Examine access, evidence retention and supplier boundaries

Map access to uploaded documents, extracted fields, review notes and audit records. The assistant's service account may have a broader technical role than the employee using the interface. Test whether the authorised customer and case boundary is enforced at the relevant retrieval and destination points. Use synthetic cross-case fixtures within approved rules rather than searching for actual customer information.

Determine what content reaches suppliers, support systems and diagnostic logs. Compare the processing route with the bank's approved arrangements and retention decisions. A masking rule in the primary request path may not cover a support attachment or failed-job export. PDPL-focused review should address the relevant personal-data processing questions with the responsible specialists.

For supplier evidence, identify the service, period and customer responsibilities covered. A provider's general security report may not establish the bank's case-level permissions or the behaviour of a customised connector. SAMA applicability and supplier evidence should remain linked to these operating tests without being treated as a substitute for them.

Test resilience without losing the case's decision state

Examine what happens when classification is delayed, a service is unavailable or a request is retried. A pending case should not become approved merely because an exception handler chooses a convenient default. The audit should inspect the approved fallback and the actual destination state. A manual route needs clear ownership and records so staff can continue the process without losing the reason for review.

For retries, distinguish duplicate processing from duplicate business effects. Reclassifying the same synthetic file may be harmless, while creating two conflicting review tasks or replacing a newer document with an older result may not be. Test safe, authorised conditions and reconcile final records. A timeout alone does not establish whether the original operation completed.

Record how the bank returns from fallback to normal operation. Pending manual cases should remain visible and should not be silently reprocessed into a contradictory state. The audit can examine a controlled exercise and selected operating evidence, while keeping test-environment capability separate from demonstrated production effectiveness.

Examine review quality and exception ageing

A human queue can become the place where unresolved risk accumulates. Review the population of open, completed, reassigned and abandoned exceptions. Define the age calculation and explain paused cases. A dashboard that excludes cases waiting for supplier clarification can make the process appear faster without resolving the underlying exposure.

Check whether reviewers receive the correct source, rule version and reason for referral. An employee should not have to reconstruct the entire extraction process from an opaque confidence score. Examine selected decisions for rationale and authority, and trace corrections to the final record. Do not assume that every completed status represents a substantive review.

If the bank uses automated routing, assess whether unusual document types or language conditions repeatedly enter a poorly supported queue. That observation may require operational investigation, but it should not be labelled discriminatory without the relevant evidence and analysis. Preserve the difference between a routing pattern, its cause and its consequence.

Report findings in the language of banking decisions

A useful finding identifies the control objective, tested configuration, population or case, observed behaviour and effect on the decision. For the worked example, it might state that an uncertain synthetic document pack reached a destination state reserved by the bank's policy for an authorised review outcome. The report should identify whether an actual next-step permission was exercised or merely available.

Keep model risk, integration risk and governance interpretation distinguishable. A wrong document label, a state-mapping defect and an unclear policy condition can lead to similar operational symptoms but require different owners. Retesting should verify the specific corrected mechanism and relevant variations, with closure evidence linked to the original finding.

Do not imply that a completeness audit validates credit estimates or counter-fraud models. Bank model validation is a separate specialised objective. A banking audit programme can coordinate these reviews while preserving their different populations, methods and conclusions.

Reconcile multiple cases without merging their authority

A customer may have more than one open request or may move between a digital channel and a branch-assisted process. The audit should examine how the system distinguishes shared customer information from evidence attached to a particular case. Reusing a verified record may be appropriate under the bank's approved process, but copying a completeness status from one product request to another can create an unsupported conclusion.

Use two fictional cases with a common synthetic customer identifier and different approved requirements. Submit a replacement document to only one case and observe the authorised updates. Check whether the other case's status changes, whether the evidence link remains correct and whether staff can see why information was reused. The expected result should follow the bank's rule, not an assumption that all reuse is either allowed or prohibited.

Also examine the handover record when a case changes channel. A branch employee should be able to distinguish an AI-proposed item from a separately verified fact and identify any outstanding review condition. If the digital interface hides that distinction during transfer, the operational risk can increase even though the model output itself has not changed.

Report the specific association and authority boundaries tested. A successful single-case demonstration does not establish correct handling of concurrent requests, replaced documents or channel transitions. These variations should be selected because they affect the scoped decision, with the resulting coverage and exclusions made explicit.

Commission a review that connects evidence to action

Prepare the process map, state dictionary, approved rule versions, authority model, supplier boundaries and a masked example of the complete case history. Explain the intended report audience and the decision awaiting support. These inputs help the audit team target the transitions where an AI result could acquire more authority than intended.

For additional commercial context, Trusted's financial-services page explores banking and insurance AI applications. Use it to identify possible operating areas, while the bank's actual controls and authoritative requirements determine what this audit can conclude.

Request an audit scoping discussion for an AI-assisted banking workflow through OpenForAI's AI auditing service. The expected output is an evidence-based view of the scoped process, actionable findings and clear limits. It should help the bank decide what to correct, restrict, investigate or verify next without presenting a model output as a completed banking obligation.