Bank model validation
Model Validation for Banks
Challenge bank predictive models through as-of data, mature outcomes, calibration, implementation reconciliation and independent review.
Model validation for a bank is a structured challenge of whether a model is conceptually appropriate, correctly implemented and sufficiently supported by evidence for its intended use. Reproducing a developer's headline score is only one part of that work. The validator must examine the target definition, data available at the decision date, observed outcomes, uncertainty, limitations and the controls governing how people use the result.
This AI auditing guide concentrates on a hypothetical one-year probability-of-default model for an SME lending portfolio. It does not recommend a credit decision, capital treatment or provision. It distinguishes model validation from the broader operational review covered in AI auditing for banking. A commercial technical review also does not automatically constitute supervisory approval or a statutory assurance opinion.
Map the model's use before applying regulatory criteria
A model used for an internal early-warning queue is not automatically governed by the same detailed requirements as an internal ratings-based capital model. Record the legal entity, portfolio, model purpose, relevant approvals and actual downstream uses. A score initially built for monitoring may later enter underwriting or capital calculations; validation should not rely only on its original project label.
Within SAMA's credit-risk IRB requirements, Section 8 on validation of internal estimates addresses comparisons between realised defaults and estimated PDs, including at least annual updates to the specified analysis and documentation. That scoped requirement should not be presented as a universal annual audit rule for every AI application in a bank.
The bank's regulatory and model-risk owners should establish the applicable criteria and their current versions. The SAMA audit guide addresses that obligation mapping. This article provides an original validation method that must be adapted to the model's use, portfolio and evidence. A generic checklist cannot decide whether a particular model is approved for a regulated purpose.
Write the validation question precisely
For the illustrative SME model, specify whether the output estimates default over the next twelve months for an obligor, a facility or another defined unit. Define the default event and the observation start. State whether the probability aims to reflect current conditions or a longer-run calibration concept. Those choices determine which outcome comparison is meaningful and which apparent deviations need a different interpretation.
Describe the population to which the model is intended to apply. New borrowers, established customers and borrowers with thin financial histories may have different evidence. Exclusions should be operationally enforceable, not merely written in a development report. If staff can score an excluded segment through the production interface, the validator should examine whether the limitation actually constrains use.
Separate the raw model output from rating mappings, overrides and downstream policy rules. A sound probability estimate can be used inappropriately, and a correct policy rule can receive a damaged model output. The validation plan should identify which layers are tested and how the final decision process connects them. It should not silently attribute every outcome to the statistical model.
Reconstruct the data as it existed at the prediction date
For each feature, establish its source, transformation, availability date and permitted use. A financial statement with a period-end date before scoring may have arrived after the score was produced. Using the latest database value in a historical backtest can therefore introduce future information. The relevant question is when the bank could actually have known the value, not only the period the value describes.
Inspect revised records and late corrections. A current warehouse may replace an earlier turnover figure or borrower classification without preserving the prior value. The validator needs a credible reconstruction or an explicit limitation. Where exact reconstruction is impossible, sensitivity analysis can show the possible effect, but it cannot turn uncertain historical inputs into observed facts.
Trace missing-value treatment and categorical mappings into production. A development script may treat a missing field differently from an API or batch job. Check a controlled set of boundary cases, including unseen categories and exceptional values. Retain intermediate calculations sufficient to identify where a score diverges, rather than comparing only the final rounded grade.
The SAMA rating-system design requirements, within their IRB scope, address model documentation, data suitability and out-of-time and out-of-sample testing. The practical procedures here provide evidence for relevant questions; citing the rule is not evidence that a particular implementation meets it.
Construct an outcome cohort that matches the target
A one-year outcome test needs an appropriately observed horizon. Recently scored borrowers may not yet have twelve months of outcomes. Do not classify them as non-default simply because no default has appeared so far. Report immature observations separately or use a justified method appropriate to incomplete observation, with its assumptions visible.
Define treatment of exits, transfers, restructurings and repeated defaults under the approved target definition. A closed account may or may not provide a fully observed outcome for the intended borrower-level question. Resolve that issue with the model and data owners. Excluding difficult cases without explanation can alter both the default rate and the population the model appears to serve.
Multiple facilities for one obligor and repeated monthly scores are related observations. A validation that counts every row as independent can overstate its evidence. Choose an analysis unit that matches the target, preserve relevant grouping and use uncertainty methods consistent with that structure. The sampling and population guide helps define what the records actually represent.
Separate ranking power from probability calibration
A model may rank borrowers usefully while systematically understating their default probabilities. Conversely, a portfolio average can be well calibrated while the model poorly distinguishes risk within the portfolio. The Basel validation research discusses discrimination and calibration as different validation questions. It is methodological background, not a substitute for applicable Saudi requirements.
For ranking, examine an appropriate discrimination measure and its uncertainty, plus performance in material segments. Do not treat a single AUC or similar score as a complete validation result. For calibration, compare estimated probabilities with observed frequencies at relevant grades or bands and over suitable periods. Interpret the comparison in light of the model's stated calibration concept and economic conditions.
| Illustrative band | Obligors | Average predicted PD | Expected defaults | Observed defaults | Observed rate |
|---|---|---|---|---|---|
| Lower band | 200 | 1% | 2 | 3 | 1.5% |
| Middle band | 200 | 3% | 6 | 8 | 4% |
| Higher band | 100 | 12% | 12 | 19 | 19% |
| Total | 500 | 4% weighted average | 20 | 30 | 6% |
Expected defaults are the sum of the individual predicted probabilities; the band averages in this simplified example produce the same total. Thirty observed defaults versus twenty expected gives an observed-to-expected ratio of 1.5 and an overall rate difference of two percentage points. Those calculations identify a discrepancy to investigate. They do not alone establish a regulatory breach, a defective model or the appropriate capital response.
The three bands remain ordered by observed default rate, but that is not enough evidence to claim strong overall discrimination. Nor does one cohort establish long-run calibration. Review uncertainty, borrower dependence, outcome definitions, portfolio changes and economic conditions before deciding whether the discrepancy is material under the bank's approved criteria.
Challenge conceptual soundness rather than only performance
Ask why the selected features and modelling approach should estimate the intended risk. A relationship can be statistically strong because it reflects a data-processing artefact or a policy decision made after emerging distress. Examine plausible mechanisms, stability and reliance on variables whose meaning changes across segments. Document where the explanation is uncertain.
Test sensitivity to meaningful input changes and boundary conditions. An implausible score movement may reveal an implementation error, an unstable interaction or a limitation that requires constrained use. Sensitivity tests should preserve economically coherent combinations where the question requires them. Arbitrarily changing one field can create impossible borrowers and produce results that are difficult to interpret.
Compare with a defensible challenger or simpler baseline. The purpose is to understand what the added complexity contributes, not to assume that the simplest model is always preferable. Use comparable data, horizons and preprocessing. If a challenger uses information unavailable to the production model, explain that advantage rather than treating the comparison as a fair contest.
Keep development, validation and test decisions traceable
Record which data influenced feature selection, model fitting, tuning and final evaluation. Repeatedly selecting a model against the same holdout turns that holdout into development information. Preserve a genuinely separate assessment where feasible and disclose limitations where it is not. The evaluation integrity guide covers leakage and selective reporting in detail.
Out-of-time testing should represent a later period not used in the relevant development decisions. Out-of-sample testing addresses a different separation question; neither label alone proves there is no leakage. Related borrowers, overlapping observation windows or reused outcomes can cross an apparent split. Inspect the actual grouping and dates rather than relying on dataset names.
Maintain the lineage from data extract through transformations to results. A validator should be able to identify which model artefact produced a result and how exclusions changed the cohort. If a developer reruns the analysis after a challenge, preserve the earlier result and explain the change. Quietly replacing the evidence makes it difficult to distinguish a correction from selective reporting.
Test the implementation used by the bank
A validated development model and the deployed scoring service may differ. Reconcile feature values, transformation outputs, probabilities, grade mappings and relevant rounding. Use cases selected to expose missing values, limits and policy transitions, not only ordinary records. Establish the acceptable numerical tolerance for the implementation and explain its purpose.
Review batch and interactive scoring separately if they use different pipelines. Check the model version recorded with each score and the behaviour when a required input is unavailable. A fallback may be justified, but the user should not unknowingly receive a score from a different model or an outdated input set. The audit needs evidence of the actual fallback and its permitted use.
Inspect access and change controls around model artefacts, mappings and overrides. This does not replace a full security assessment, but it matters to the validity of the result being reviewed. A well-tested model cannot support a production conclusion if an uncontrolled mapping changes the score's meaning after validation.
Review overrides as decisions with their own evidence
Human overrides may incorporate relevant information outside the model, but their existence does not prove better outcomes. Record the original score, override reason, authorised decision-maker and resulting grade or action. Examine whether reasons are specific enough to review and whether the same rationale is applied consistently to comparable cases.
Where outcomes are mature, compare override groups cautiously. People select which cases to override, so a simple comparison with non-overridden borrowers may be confounded by their different circumstances. The validator should describe the selection mechanism and limits rather than claiming a causal benefit from an unadjusted default-rate comparison.
Check whether overrides are masking a model limitation that should be addressed directly. Frequent corrections in one segment may justify model redevelopment, clearer use restrictions or improved data. The decision belongs within the bank's governance process, supported by the validation evidence and the consequences of continued use.
Use a validation decision matrix
| Question | Evidence | Possible finding | Responsible next decision |
|---|---|---|---|
| Was information available as of scoring? | Source availability dates and historical feature reconstruction. | A later financial update entered the backtest. | Data owner reconstructs the cohort; validator reassesses affected results. |
| Are outcomes sufficiently observed? | Scoring dates, follow-up windows and default records. | Immature cases were counted as non-default. | Correct the outcome design before interpreting calibration. |
| Does the implementation match? | Independent score reconciliation across boundary cases. | Production missing-value handling differs from development. | Engineering corrects the pipeline and obtains focused retesting. |
| Is the use supported? | Approved purpose, segment evidence and actual downstream use. | A newly added borrower segment lacks validation evidence. | Model owner proposes restriction or further validation for approval. |
| Can the challenge be independent? | Roles, prior development involvement and review arrangements. | The reviewer would assess their own design decisions. | Appoint appropriate independent challenge or disclose the limitation. |
The matrix is an internal planning aid, not a mandated SAMA template. It deliberately separates a finding from the management decision it informs. A validation report can recommend restrictions or additional work, but it should not claim that the auditor independently authorises the bank's regulated model use.
Handle sparse defaults and external benchmarks carefully
A portfolio with few defaults may not support a precise validation result at every grade. Combining grades can increase the event count while concealing a local weakness; extending the observation window can introduce older underwriting standards or economic conditions. Describe those trade-offs explicitly. A larger dataset is not automatically more relevant to the current use.
External benchmark data can add context, but compare definitions, borrower mix, geography, observation horizon and reporting practices before interpreting a difference. A published default rate for a different population is not a substitute for the bank's own outcome evidence. If data is pooled, document the weighting and why the combined population informs the model under review.
Do not treat zero observed defaults as proof that a predicted probability should be zero. Equally, a small number of defaults concentrated in related borrowers can create an apparent signal that needs dependence-aware interpretation. The validator should explain what the evidence can distinguish, what remains uncertain and which use restrictions or further observations management should consider.
For the illustrative twenty-expected, thirty-observed cohort, a responsible next step is to reconcile every observed event and the relevant predictions, then examine whether the excess concentrates in a segment or period. Check the outcome definition and historical features before attributing the discrepancy to economic change. A convenient macroeconomic explanation should remain a hypothesis unless the evidence supports it.
Agree the bank's escalation criteria before selecting the most reassuring test. If several reasonable methods produce different interpretations, present that sensitivity and its assumptions. Choosing only the method that avoids escalation would undermine the purpose of independent challenge. The conclusion can be useful even when it is conditional: for example, identifying a specific evidence gap that must be resolved before extending the model to another segment.
Make independence and competence specific to the engagement
Identify who developed the model, who operates it, who validates it and who approves its use. Prior design or implementation involvement can create self-review concerns. Assess those concerns for the actual engagement and document safeguards, separate challenge or limitations. The independence guide explains why a general claim of independence is not enough.
Competence should match the questions: credit-risk interpretation, statistical methods, data engineering and production implementation may require different specialists. A team should not conceal a missing capability behind a large checklist. Where expertise or access is unavailable, narrow the conclusion or arrange additional review rather than implying full validation.
Report limitations, conditions of use and retest requirements
The final report should identify the model and implementation versions, intended uses, tested cohorts, outcome maturity, methods, material findings and unresolved evidence gaps. State whether the evidence supports the proposed use under specified conditions. Avoid a simple pass label that hides a material segment exclusion or an unresolved implementation difference.
Define what must happen before a finding can close. A corrected extract may require all affected metrics to be recalculated, while a deployment mismatch may need targeted reconciliation and broader impact review. Use retesting evidence to distinguish a completed remediation task from a verified correction. Record management's acceptance of remaining uncertainty separately.
Review triggers should reflect model changes, data-source changes, portfolio shifts, performance signals and the applicable governance schedule. A calendar review does not excuse ignoring a material change between reviews. Conversely, a small change should receive a reasoned impact assessment rather than an automatic claim that the entire model has become invalid.
For a scoping discussion, prepare the model inventory entry, use statement, development report, data dictionary, cohort definitions and one reproducible result with masked data. Discuss evidence and scope for bank model validation to establish what can be independently challenged and which decisions remain with the bank's accountable authorities.
FAQ
Your questions, answered.
Does SAMA require annual validation of every AI model in a bank?
Do not generalise a scoped rule to every AI application. The cited IRB internal-estimates provisions include an annual update requirement for specified comparison analysis and documentation. Applicability depends on the entity, activity, model use and relevant framework. The bank’s regulatory and model-risk owners should map those requirements. A customer-service assistant, an early-warning model and an approved capital model can raise different validation and governance questions.
What is the difference between PD calibration and discrimination?
Discrimination concerns how well the model separates relative risk, while calibration concerns whether predicted probabilities correspond appropriately to observed outcomes under the stated target and calibration concept. A model can rank borrowers usefully while understating default probabilities. A portfolio average can also look correct while important segments are poorly estimated. Validation should assess both, with suitable cohorts, uncertainty and economic interpretation rather than relying on one headline score.
How do validators detect future information in a credit-risk backtest?
Trace when each input became available to the bank, not just the period it describes. A financial statement or corrected record may have a historical date but arrive after the prediction. Reconstruct the as-of feature set, inspect transformations and separate development from evaluation information. Where the historical state cannot be recovered, disclose the limitation and use sensitivity analysis carefully; it cannot substitute for missing observations.
Can recent borrowers be treated as non-defaults in a one-year validation?
Not merely because no default has been observed yet. Their outcome horizon may be incomplete. Define a mature cohort or use a justified method for incomplete observation, with explicit assumptions and exclusions. Treatment of closed accounts, transfers and restructurings also needs to match the approved target. Counting immature cases as non-default can distort calibration and make the model appear more accurate than the evidence supports.
Does a high AUC mean a bank model is validated?
No. It addresses only part of predictive discrimination under a particular dataset and method. Validation also examines calibration, conceptual soundness, data integrity, implementation, permitted use and limitations. The score needs uncertainty and segment interpretation, and the evaluation must be free of material leakage or selective reporting. A model may have strong ranking performance but produce unsuitable probabilities or differ materially from the deployed scoring implementation.
How should a bank validate a proprietary vendor model?
Establish the intended use and the evidence needed for a meaningful challenge, including input definitions, limitations, reproducibility and implementation behaviour. Restricted intellectual-property access may require controlled review arrangements or alternative evidence, but it should not be converted into an unsupported conclusion. For IRB rating systems, the cited SAMA design provisions explicitly address vendor-model documentation responsibilities. Any remaining access limitation should be visible in the validation decision.
Can human overrides be assumed to improve credit model outcomes?
No. Record the original score, reason, authority and resulting change, then examine mature outcomes where appropriate. Overridden cases are selected rather than random, so a simple comparison with other borrowers may not identify the effect of the override. Frequent overrides may reveal a model limitation or data problem. Validation should examine that mechanism and governance, rather than treating the presence of human judgement as automatic evidence of improvement.
What evidence closes a bank model validation finding?
Closure depends on the finding. A data-timing defect may require cohort reconstruction and recalculation of affected results; an implementation mismatch may require independent score reconciliation and impact review. Preserve original evidence, identify the corrected version and test relevant boundary cases. Management may accept a remaining limitation, but that is separate from verified remediation. The report should state which uses and conclusions the new evidence supports.