Bias and Fairness Testing

Bias and fairness testing asks whether an AI-supported process creates unjustified differences in errors, access or treatment, and whether the evidence is strong enough to explain those differences. It is not a search for one number that declares a system fair. The auditor must understand the task, the people affected, the meaning of the reference labels and the consequences of both correct and incorrect outputs.

This guide is for model owners, service managers and review teams evaluating systems that sort or prioritise people's submissions. Its hypothetical example is a scholarship-application completeness tool. The tool checks whether required documents appear present and routes uncertain submissions for assistance. It does not rank applicants, assess merit or decide scholarship awards. That distinction matters: fairness in document routing does not establish fairness in the eventual award process.

NIST SP 1270 treats AI bias as a wider sociotechnical issue, rather than only an intentional act or a model property. An audit therefore examines the surrounding process as well as outputs. The proposed method below supports a bounded AI audit, with criteria agreed for the actual use and affected population.

Define the harm before selecting a fairness metric

In the example, incorrectly marking a complete submission incomplete may delay review or burden an applicant with unnecessary resubmission. Incorrectly marking an incomplete submission complete may consume reviewer time or allow an unresolved documentation issue to persist. Neither error is the same as denying a scholarship, but each can affect access to the later process. Map those consequences before deciding which differences matter.

Identify who encounters the tool and who is missing from its records. Applicants who abandon an inaccessible upload process never appear in the model's scored population. A fairness assessment limited to successfully submitted files cannot establish equal access to the application process. It can still be useful, provided the report states that narrower boundary and identifies the missing population.

Agree the operational decision under review. Is the tool recommending a queue, automatically requesting another document or blocking progress? The same prediction has different consequences in these workflows. Record the possibility of correction, assistance and appeal. A system with review opportunities can still impose unequal burdens if one group repeatedly needs those opportunities to overcome preventable errors.

Challenge what the labels actually measure

A label called “complete” should correspond to a defined document requirement, not an earlier reviewer accepting the application for unrelated reasons. Examine the rule version, the evidence available at the time and the process used to establish the reference. If reviewers used later-uploaded documents when labelling earlier submissions, the benchmark may judge the model against information it could not have seen.

Test inter-reviewer disagreements on consequential cases. An unclear scanned certificate or an alternative document format may expose an ambiguous policy rather than an AI failure. The policy owner should resolve the rule or preserve the uncertainty. Forcing every disputed case into a binary label can make a precise-looking error rate conceal unresolved judgement.

Historical outcomes are not automatically neutral ground truth. A previous process may have accepted familiar document formats more readily, or applicants with more time may have supplied corrections more often. The auditor should examine how the labels were produced and whether they capture the intended construct. That review can reveal why a technically accurate imitation of past decisions is not the right objective.

Select groups with a justified purpose and lawful data handling

Choose groups because they relate to a plausible harm or an agreed assessment question. In the worked example, the comparison is between Arabic-first and English-first submission workflows, using the recorded interface route. These are operational cohorts, not proxies that automatically establish nationality, ethnicity or another personal characteristic. Do not infer sensitive attributes from names or language simply to fill an analysis table.

Where an assessment needs personal attributes, establish authority to process them, access restrictions and the minimum data required. The decision belongs with the appropriate privacy and legal owners. The PDPL audit guide addresses processing evidence and applicability; a fairness objective does not itself remove data-protection obligations. Missing attributes should be reported rather than silently guessed.

Review intersections where the task supports a meaningful question, such as language route combined with document format. Avoid subdividing until every group is too small to interpret. Keep group counts visible and predefine the main comparisons. Exploratory analysis can suggest a mechanism, but a pattern discovered after many comparisons needs cautious interpretation and preferably confirmation on fresh evidence.

Keep selection rates and error rates distinct

The Fairlearn documentation distinguishes selection-rate and error-rate criteria and explains that metric choice carries assumptions. The original example below shows why a single parity measure cannot settle the audit question. It uses an adjudicated, hypothetical test population with the same underlying completeness proportion in two workflow cohorts.

Hypothetical completeness-routing results; positive means ready for substantive human review

Measure	Arabic-first workflow	English-first workflow	Meaning
Total submissions	200	100	All cases in the illustrative comparison.
Actually complete	160	80	Reference completeness is 80% in each cohort.
Actually incomplete	40	20	Reference cases needing document correction.
Complete but marked incomplete	16	16	False negatives: 10% of 160 versus 20% of 80.
Incomplete but marked complete	8	2	False positives: 20% of 40 versus 10% of 20.
Marked ready for review	152 of 200, or 76%	66 of 100, or 66%	Correctly ready plus incorrectly ready; not award acceptance.

The equal count of sixteen false negatives hides a different burden: one tenth of complete Arabic-first submissions and one fifth of complete English-first submissions are incorrectly delayed. Meanwhile, the Arabic-first workflow admits a higher share of incomplete submissions. The ten-percentage-point difference in routing rates does not explain either mechanism on its own. This is an arithmetic illustration, not evidence about real Arabic or English users.

The positive label must remain explicit throughout the analysis. If a later chart treats “incomplete” as the positive class without changing the definitions, false-positive and false-negative interpretations can reverse. Save the confusion-matrix definitions with the results and ask a second reviewer to reconcile counts to the underlying cases. Simple bookkeeping errors can otherwise become consequential fairness claims.

Quantify uncertainty without overstating what the sample proves

The table describes the selected test population. To estimate performance in a larger applicant population, the audit needs a justified selection mechanism and uncertainty method. Cases may cluster by institution, document issuer or template. Repeated pages from the same submission are not independent applicants. State the unit being counted and preserve the connection between pages and submissions.

Small groups can produce unstable rates. A zero-error cell does not prove that the relevant group faces no risk, and a large percentage based on very few cases may not support a precise comparison. Report counts and uncertainty appropriate to the design, together with the practical consequence. Statistical uncertainty does not mean a serious individual error should be ignored; it limits claims about the wider rate.

Separate planned comparisons from exploratory findings. If many group and metric combinations are examined, chance patterns become more likely. A defensible report describes that search, avoids presenting the most dramatic result as if it were the sole preplanned test, and identifies what further evidence would confirm or weaken the proposed explanation. Do not import a legal numerical threshold from another jurisdiction as a universal fairness pass mark.

Investigate mechanisms with paired and stratified tests

A measured disparity is an observation, not a complete causal explanation. Examine whether the workflows differ in scan quality, document mix, upload route, missingness or reference-label quality. These factors can help explain an error pattern, but controlling for them mechanically may also remove part of the very burden the service should address. The chosen comparison must follow the question, not simply maximise apparent parity.

Use carefully constructed paired cases where the substantive document evidence remains equivalent while a relevant presentation feature changes. For example, qualified reviewers can prepare equivalent Arabic and English cover instructions around the same synthetic document bundle. Check that the pair truly preserves meaning and required evidence. A literal translation that changes a condition is not a controlled test of language handling.

Inspect the complete pipeline. An OCR component may omit a date, a parser may misclassify a document and the final model may correctly process the damaged input. Attribute the observed error only as far as evidence allows. The evaluation integrity guide helps separate input contamination and reference defects from genuine model differences.

Review the human response after the model's output. If a reviewer routinely corrects one type of mistake but another persists through the process, model-level parity may not reflect the final burden. Measure the time and effort required to resolve incorrect routing where the scope permits. Use human oversight auditing to examine whether reviewers can identify and reverse the errors they are expected to catch.

Compare interventions against both benefit and displaced harm

Changing a threshold can alter more than one error rate. Improving access for complete submissions may also route more incomplete submissions to reviewers. That may be an acceptable trade-off, but it needs an explicit decision and evidence about capacity. A fairness improvement should not be declared solely because one selected measure moved closer to equality.

Original intervention review worksheet for the hypothetical completeness tool

Candidate change	Expected mechanism	Evidence before approval	Residual question
Improve document extraction	Recover fields lost in a particular scan or script format.	Paired extraction review and downstream routing tests.	Does the correction generalise beyond the known template?
Route uncertainty to assistance	Reduce unsupported incomplete decisions.	Queue capacity, response time and reviewer correction evidence.	Does one cohort bear a longer assistance delay?
Clarify accepted alternatives	Resolve ambiguous reference rules.	Policy-owner decision and relabelled adjudicated cases.	Are older results comparable after the rule changes?
Change the decision threshold	Alter the balance between routing errors.	All relevant error rates on a held-out set, by cohort.	Which new burden requires management acceptance?
Improve upload instructions	Prevent avoidable submission problems before scoring.	User testing and evidence of completed submissions.	Are people who abandon the process now better represented?

This worksheet is a proposed internal decision aid, not a prescribed regulatory method. The accountable owner should document the chosen intervention, rejected alternatives, expected effects and monitoring conditions. If the intervention changes the task definition, establish a new baseline rather than comparing unlike results as though only model quality changed.

Protect the test from tuning and selective reporting

Preserve an evaluation set that the improvement team has not repeatedly used to optimise thresholds or rules. Retain the original test cases for regression checks, but add independently adjudicated cases to challenge whether the correction extends beyond them. Disclose when the supposedly final test has already influenced development decisions.

Report exclusions by cohort and reason. Missing documents, unreadable files, unresolved references and technical failures may be part of the service burden. Excluding them from a clean accuracy calculation can be appropriate for a narrow technical question, but the operational impact still needs separate reporting. A transparent denominator is more informative than a polished score with unexplained omissions.

Keep the audit conclusion separate from management's value judgement. Evidence may show that a change reduces one disparity and increases another. The responsible owner decides whether that trade-off is justified in the actual process, with specialist advice where needed. The auditor should document the evidence and unresolved questions rather than presenting a mathematical optimisation as a complete ethical or legal decision.

Report a bounded finding that supports action

A useful finding identifies the affected workflow, the error definition, the cohort counts, the reference basis and the consequences. It distinguishes a demonstrated disparity from an inferred cause. In the illustrative table, the finding could describe a higher false-incomplete rate in the English-first test cohort, while treating extraction quality as a hypothesis until traces and paired tests support it.

State what was not assessed: award selection, applicants who never completed submission, unrecorded demographic groups or later reviewer decisions may be outside scope. Those limitations do not erase the observed routing issue, but they prevent an unsupported claim that the entire scholarship process is fair or unfair. The coverage guide helps present these boundaries alongside the conclusion.

Assign corrective ownership and define a retest that checks both the targeted error and displaced harms. Reconcile the new results to the old definitions, or explain why comparability has changed. Retesting evidence should support closure of a specific finding; an improved aggregate score alone does not establish that the affected cohort's burden was corrected.

Follow the applicant's path after an incorrect result

A model error count does not describe the full burden of correction. Trace a small, authorised set of synthetic journeys from the initial completeness decision through the assistance process. Inspect the message sent, the document requested, the time allowed and the route for disputing an incorrect request. Determine whether the applicant can understand what is missing without being told to resubmit everything.

For equivalent test submissions, compare whether the Arabic and English interfaces explain the same correction. An accurate model classification can still lead to unequal treatment if one message identifies the missing page while the other gives only a generic rejection. Preserve the exact communication version and the task state. This is a workflow observation, and it should not be attributed automatically to the classification model.

Check how corrected applications re-enter the queue. A complete submission wrongly delayed by the tool might lose its original position or face a closing deadline before assistance responds. The audit should examine the documented policy and observed operation, without deciding scholarship entitlement. Where the consequence depends on another team, identify that dependency and the evidence needed to assess it.

Appeal records also have selection effects. People who challenge a decision may differ from those who accept it or leave the process. A low reversal rate among recorded appeals cannot establish that initial decisions were fair for everyone. Likewise, a high reversal rate may reflect good targeting of the appeal route rather than a reliable estimate of all errors. Explain the population to which each result applies.

Write the decision memo before choosing a preferred result

An original fairness decision memo can use five fields: the affected decision, the evidenced burden, the available interventions, the remaining uncertainty and the accountable decision-maker. In the illustrative example, the evidenced burden is incorrect completeness routing in the tested cohorts. It is not an established difference in scholarship awards. The memo should preserve that distinction even if senior stakeholders want a single overall fairness label.

Record why the selected intervention addresses the supported mechanism and how its effect will be checked. If management chooses assistance routing while extraction is investigated, specify who owns the queue and what would trigger another review. If the evidence does not support a cause, authorise an investigation rather than presenting a speculative fix as verified remediation. Retain dissenting technical observations and explain how unresolved disagreements affect the conclusion.

This memo makes the audit useful without pretending that mathematics resolves every policy choice. It also creates a stable record against which later changes can be assessed: what was known, what was assumed, which burden was accepted and which evidence management expected next.

Prepare the evidence for a fairness audit

Useful preparation includes the task definition, routing consequences, policy versions, label-production process, authorised group fields, cohort counts and known complaints or corrections. Provide masked examples of disputed cases and explain where data is missing. Do not create sensitive group labels retrospectively without an approved purpose and handling basis.

The first scoping decision should be which question the available evidence can answer responsibly. It may support a focused language-workflow comparison while leaving broader demographic questions unresolved. Discuss bias and fairness testing for your actual decision process to define a review that connects metrics to people, evidence and accountable corrective decisions.