AI Severity Rating Framework

An AI audit severity rating should explain why a finding matters for a real decision. It should not be an unexplained colour, an automatic product of two ordinal numbers or a negotiation over how uncomfortable the report looks. A defensible rating considers credible consequences, actual exposure, the effectiveness of existing controls and the uncertainty in the evidence. It also distinguishes the original condition from the exposure remaining after verified containment.

This guide proposes an internal severity framework for audit leads, risk owners and reviewers. It is an original method for discussion and adaptation, not a NIST scoring system, an ISO requirement or a Saudi regulatory scale. A hypothetical supplier payment-term extraction workflow illustrates how the same content error can have different consequences depending on what the surrounding process permits.

Rate the consequence of the finding, not the sophistication of the AI

A finding involving a complex model is not automatically more severe than one involving a simple rule. The question is what credible harm or control failure follows from the observed departure in its actual context. An incorrect term in a draft field may be corrected before use. The same field, automatically transferred into an operational payment schedule without review, can create a materially different exposure. Describe the pathway before selecting a label.

Separate observed effects from plausible effects. If the audit established an incorrect extracted term but did not inspect payment execution, it cannot claim an actual incorrect payment. It can explain the credible pathway and identify the evidence needed to determine whether it occurred. This distinction preserves urgency without turning a risk scenario into a fabricated incident.

Define severity as a property of an assessed finding within a scope. A system-wide risk rating, an incident classification and an audit finding rating may use different conventions. Do not import one scale silently into another. State which decision the rating informs: immediate containment, management escalation, remediation priority, deployment restriction or further investigation. A rating that changes no decision is mainly decoration.

NIST SP 800-30 Rev. 1 provides a general risk-assessment reference for information systems and organisations. The proposed approach below draws on the need to examine risk in context, while using its own definitions and decision rules. It does not claim that NIST prescribes these labels, deadlines or thresholds for AI audit findings.

Use five separate assessment questions

Original severity assessment dimensions

Dimension	Question	Evidence to seek
Consequence	What credible adverse outcome could follow, and for whom?	Workflow dependency, affected decision, reversibility and stakeholder impact.
Exposure	Which cases, users, versions or actions can encounter the condition?	Permissions, deployment scope, operational volumes and triggering conditions.
Likelihood basis	What evidence supports occurrence in the relevant context?	Observed cases, representative analysis, mechanism evidence and important uncertainty.
Existing controls	Which controls interrupt the pathway, and were they shown to operate?	Approval, detection, correction and escalation records, not policy statements alone.
Evidence confidence	How reliable and complete is the basis for the assessment?	Source authenticity, scope coverage, unresolved alternatives and missing records.

These dimensions should remain visible even if the organisation needs a final label. Two findings can share the same label for different reasons. One may involve frequent reversible errors; another may involve a rare but potentially serious boundary failure. The response and verification needs can differ substantially. A short rationale should explain the decisive factors rather than imply that the colour contains the entire assessment.

Keep consequence categories meaningful for the organisation. Financial, operational, privacy, safety, legal and customer effects cannot always be added into one numerical score. Where several consequences are credible, describe them and identify the one governing the decision. Do not invent a universal monetary threshold or claim a regulatory threshold without checking the applicable requirement and entity context.

Define labels through response expectations

The following labels are a proposed starting point. An organisation should calibrate them to its risk appetite, decision authority and applicable obligations. Avoid attaching universal remediation deadlines: a fixed number of days may be inappropriate when immediate containment is needed or when evidence is required to determine whether exposure exists. Separate the urgency of containment from the time needed for a durable correction.

Illustrative internal severity labels, not external requirements

Label	Decision meaning	Typical response logic
Critical	A credible severe consequence requires an immediate decision about exposure.	Escalate promptly; consider suspension or restriction under authorised incident and risk processes.
High	A material control weakness needs priority management action and explicit interim conditions.	Assign accountable leadership, contain where feasible and establish a verified correction plan.
Moderate	A meaningful departure requires planned correction, with exposure bounded by evidenced controls.	Agree ownership, milestones and proportionate verification; monitor assumptions about protection.
Low	A limited departure has a contained consequence under the evidenced context.	Correct proportionately and retain the rationale; low does not mean no issue.
Assessment pending	The evidence does not yet support a defensible severity judgement.	Identify missing evidence and interim precautions; do not treat the absence of a rating as approval.

“Assessment pending” is an evidence state, not a fifth severity level. It prevents a team from assigning a reassuring low label to a case it cannot understand. A potentially consequential but uncertain issue may still need precautionary containment while evidence is gathered. Record the basis for that precaution separately from any eventual finding rating.

Use defined escalation triggers for especially consequential conditions. The trigger should refer to a credible pathway, such as an unauthorised external action with serious consequences, rather than a fashionable technical term. A prompt-injection demonstration with no access to sensitive data or consequential tools differs from a demonstrated boundary failure in an authorised test of an operational action. The technical label alone does not determine severity.

Worked case: supplier payment-term extraction

In the hypothetical workflow, an assistant extracts payment terms from approved supplier documents into a draft record. A source states 60 days, while the extracted field states 30 days. These figures describe an illustrative content discrepancy, not actual contractual advice or a payment recommendation. The audit first confirms the source version, the submitted input and the output before assessing the consequence.

In configuration A, the field remains a draft and a qualified employee compares it with the source before any schedule is changed. The selected evidence shows that this error was corrected before downstream use. The assessor still has a content-control finding, but should not claim that this case caused an incorrect payment. A proposed moderate rating might be supportable if the departure is meaningful and the protective review is evidenced within the tested scope.

In configuration B, the field can be exported automatically into a payment-planning system, and the audit confirms that the relevant review control can be bypassed under the tested permissions. The same extraction error now has a different credible pathway. A proposed high rating may be justified by the operational exposure and control weakness, even if the audit has not established that an incorrect payment actually occurred. The rationale must state that distinction.

In configuration C, the owner asserts that review always occurs, but the relevant event records are unavailable. The assessor cannot simply inherit configuration A's protection or assume configuration B's bypass occurred. Record the evidence limitation, obtain alternative support if feasible and decide whether interim restriction is warranted. A missing log is not itself proof of every feared outcome, but it can prevent a reassuring conclusion about the control.

Illustrative rating memo for configuration B

Memo element	Recorded reasoning
Observed departure	The tested extraction changed 60 days to 30 days; the approved criterion requires faithful extraction or referral.
Credible consequence	An unreviewed field can influence payment planning; actual payment execution was outside the available evidence.
Exposure basis	The tested role and integration path permit transfer without the intended review event.
Existing protection	The asserted review step was not enforced on the demonstrated path.
Proposed severity	High under the illustrative internal scale, subject to the stated scope and evidence.
Immediate decision	The authorised owner should decide whether to restrict the transfer path pending correction and verification.
Uncertainty	Wider deployment exposure and actual downstream outcomes require further evidence.

Do not multiply ordinal labels into false precision

Some organisations assign numbers to likelihood and impact and multiply them. That can be a practical sorting convention if its limitations are explicit. It is not automatically a quantitative estimate of risk. A label coded as four is not necessarily twice as severe as a label coded as two. Multiplication can also make different combinations appear equivalent even when they require very different responses.

If a scoring matrix is required, publish the definitions, boundary rules and override process. Explain how uncertain inputs are handled and which consequences cannot be traded away through a lower likelihood score. Retain a narrative rationale beside the score. Do not present a calculated number with decimal places as more objective than the judgements used to create it.

Quantitative estimates require defensible quantities and units. Expected financial loss, for example, needs a supported probability and consequence model, with assumptions and uncertainty. A targeted audit test does not supply an operational probability merely because it produced several failures. Keep sampling interpretation separate from severity judgement and avoid deriving population frequency from deliberately difficult cases.

Assess controls by operation, not by their presence in a policy

A human review requirement reduces assessed exposure only to the extent supported by evidence about its actual operation. Examine who reviews, what they can see, whether they have time and authority to challenge the output and what happens when they disagree. A checkbox or an approval button does not by itself show that the source was examined. The relevant question is whether the control interrupts the identified pathway.

Detection also differs from prevention. A daily report that identifies erroneous fields after transfer may support recovery but not prevent the initial exposure. Assess the time between error, detection and correction, and whether the consequence can be reversed within that window. Do not give equal credit to controls that operate at different points merely because both are called “monitoring”.

Use human oversight evidence where protection depends on review. Record residual uncertainty where the audit observed only a small or selected set of interventions. A control can work in one case and fail in another. The severity rationale should state the tested basis for relying on it rather than claiming universal effectiveness.

Separate original severity, current exposure and urgency

The original severity records the condition assessed before corrective action. Current residual exposure reflects verified containment or correction. Urgency concerns how quickly a decision or action is needed. These can move differently. Restricting an integration may reduce immediate exposure while the underlying defect remains unresolved. A deadline for an upcoming release may increase urgency without changing the inherent consequence of the defect.

Keep all relevant assessments in the findings register with their dates and evidence. Do not overwrite the original rating merely to improve the dashboard. If containment is temporary, record its expiry or review trigger and the assumptions on which it depends. A manual workaround that is sustainable for a short pilot may be unreliable at a larger volume.

Link rating changes to the remediation plan. A lower current exposure can justify a different sequence of work, but it should not remove the need for a durable correction or authorised acceptance. Conversely, a correction that addresses the tested defect does not automatically lower unrelated risks. Explain precisely which pathway changed and what evidence supports the reassessment.

Calibrate reviewers without forcing agreement

Use a small set of anonymised or hypothetical cases to compare how reviewers apply the scale. Ask them to state consequence, exposure, controls and uncertainty before choosing a label. Differences often reveal ambiguous definitions or missing evidence rather than poor judgement. Discuss those reasons and revise the guidance where necessary. Calibration should improve consistency without suppressing legitimate disagreement.

For the payment-term example, ask reviewers what additional evidence would change their rating. Verified mandatory review might change exposure; proof of wider deployment might change reach; evidence of actual downstream effect might change the consequence assessment. This exercise makes the reasoning testable. A reviewer who cannot identify any evidence that would change a rating may be applying a fixed preference rather than assessing the case.

Maintain a decision log for difficult boundaries. Record the case characteristics, competing interpretations and final rationale without turning every past judgement into an immutable rule. New contexts may differ in important ways. A catalogue of examples is a calibration aid, not a substitute for examining the current workflow and evidence.

Test the rating against alternative explanations

Before finalising a rating, identify the assumptions that carry the conclusion. In the supplier example, the assessor assumes the approved source really governs the extracted field, the recorded output is the version used in the workflow, and the integration path operates as observed. A later amendment to the supplier document, an employee correction or a disabled production integration could change the interpretation. Investigate these possibilities where credible rather than treating every alternative as either irrelevant or automatically exculpatory.

A useful sensitivity note states what would change the assessment. If the transfer path is demonstrably disabled across the affected scope, immediate exposure may fall while the extraction defect remains. If the same path is active in additional deployments, the reach may be wider than first established. If review evidence is missing only for an interval, avoid extending that uncertainty to periods with adequate evidence. The note should connect each possible change to the specific dimension it affects.

Do not let an average outcome conceal concentrated consequences. A small subset of documents with unusual layouts may contain most of the observed extraction errors. The overall sample rate alone may therefore be a poor guide to exposure for that subset. Explain whether the subgroup was selected representatively or discovered through investigation, and avoid claiming a precise subgroup rate without a suitable design. The severity assessment can still identify the credible pathway and request focused evidence.

Apply the same consequence logic across Arabic and English workflows while respecting differences in source quality, document structure and actual use. A translation error is not inherently minor, and an Arabic-language finding should not receive a different label merely because reviewers find it harder to assess. Obtain competent language review, identify unresolved meaning and state the limitation. If the authoritative document or applicable interpretation is disputed, refer that question to the responsible qualified owner before building a confident severity argument on it.

Record benefits separately from the finding rating. A workflow may save considerable effort while retaining a consequential control weakness. Management can consider benefits when choosing a response, but the benefit does not erase the observed departure or turn an ineffective control into an effective one.

Handle disagreement and communicate what the rating requires

Management can challenge facts, applicability, consequence assumptions or the proposed response. Record the disputed element and supporting evidence. Correct factual errors promptly, but do not lower a rating solely because the finding is commercially inconvenient. Equally, do not retain a severe rating after reliable new evidence removes its basis. The register should show why the assessment changed or remained unchanged.

Communicate the label together with the decision it requires, interim conditions, owner and verification expectation. A high rating without an accountable next step leaves the organisation with a warning but no response. A low rating should still state the departure and proportionate action. The audit report should preserve the rationale and material uncertainty so readers do not interpret the colour more broadly than the evidence allows.

For a scoped AI audit, prepare the current severity definitions and a masked example where reviewers disagree. These materials help determine whether the issue is missing evidence, unclear criteria or inconsistent application. Discuss a defensible severity approach for your AI audit findings.