AI Performance and Evaluation Integrity

AI performance and evaluation integrity concerns whether a reported result means what its reader thinks it means. A score may be calculated correctly yet rest on familiar test examples, contaminated reference data, selective exclusions or an evaluator that rewards fluent mistakes. An audit examines the chain from the performance claim to the population, test design, judgements and calculation supporting it.

This guide is for technical buyers, model owners and reviewers who need to challenge evidence before relying on it. The hypothetical example is an address-standardisation service that converts supplied address text into defined fields. It does not verify legal residence, certify an official address or authorise a delivery. Its evaluation must establish preservation of the supplied address meaning, including unit numbers, rather than merely producing a tidy-looking string.

The objective within an AI audit is a reproducible, bounded interpretation. A lower but credible result is more useful than a higher figure whose denominator or reference basis cannot be explained. The method below is an original audit approach; its illustrative numbers are not benchmarks for Saudi address systems or any provider.

Write the performance claim as a testable sentence

“The model is 95% accurate” omits the task, population, period and definition of correctness. A useful claim identifies which address inputs were tested, which fields were required, how ambiguous cases were handled and whether the result applies to completed outputs or all attempted cases. It should also name the configuration and whether the system could use external reference information.

Distinguish syntactic validity from semantic correctness. A record can meet the required JSON schema while changing apartment 57 to apartment 75. Exact text matching can also mark a harmless formatting variation wrong. The evaluation needs a rule that separates equivalent representations from material changes, approved by people who understand the operational use.

Define the decision the score supports. A result adequate for suggesting a standardised address to a human may not support unattended updates to a customer database. The audit should identify this use boundary rather than letting a benchmark result travel into a more consequential application without further evidence.

Inspect the origin and relationship of test cases

Inventory the sources used for training, tuning, examples in prompts, reference lookup, test construction and final adjudication. Record who could access each set and when. A test case can influence the system without appearing in model training: it may be copied into instructions, used to tune a retrieval index or repeatedly reviewed during prompt development.

In the address example, different spellings of the same premise may appear as separate rows. A random row split can put one version in development and another in the test set. The model may then appear to generalise to new addresses while mostly recognising familiar underlying premises. Define the grouping unit according to the claimed generalisation, not according to the easiest file format.

Exact duplicate removal is insufficient where near-duplicates share the same underlying answer. Review entity identifiers, normalised representations and meaningful variations. Be careful not to merge distinct units within one building as though they were the same address. The grouping method itself needs validation and a record of ambiguous matches.

Audit the split before auditing the score

The scikit-learn guidance on data leakage explains that transformations learned from data must not learn from the test set. This principle extends to auditing the actual preparation process: inspect which records influenced normalisation, feature selection, dictionaries and other learned decisions, rather than relying on a file named “holdout”.

Original evaluation-integrity register for the address-standardisation example

Integrity question	Evidence to inspect	Possible misleading result	Corrective test
Are underlying premises separated?	Entity grouping and cross-split overlap review.	Familiar address variants inflate apparent generalisation.	Evaluate independently held-out premises.
Was preparation fitted only on permitted data?	Transformation code and fitting records.	Test information influences normalisation rules.	Rebuild preparation within the valid split.
Did test cases influence prompt changes?	Experiment history and access records.	A repeatedly consulted holdout becomes development data.	Use a fresh, separately governed assessment set.
Are difficult cases retained?	Case inventory and exclusion reasons.	Abstentions and failures disappear from the headline rate.	Reconcile all intended cases and report outcomes separately.
Does the judge detect material errors?	Blind human adjudication and disagreement analysis.	A neat output receives credit despite a changed unit number.	Validate the judge against consequential controlled examples.

The register identifies questions, not automatic findings. An overlap may be acceptable for a claim about recurring known addresses, but incompatible with a claim about unseen premises. The auditor should make that distinction explicit. Correcting the split changes the question answered; it should not be presented as a minor formatting adjustment to the same evidence.

Preserve separate development and assessment decisions

Repeated experimentation can erode a holdout even without deliberate misconduct. If a team sees the results, adjusts the system and tests again, it learns about the set. Preserve the sequence of experiments and explain which data influenced the final choice. A separate final assessment can provide stronger evidence, but its access and use must also be governed.

Keep known failing cases for regression tests. They are valuable precisely because the team knows them. Do not describe success on that familiar set as fresh independent evidence of general performance. Combine regression checks with newly selected, independently adjudicated cases, and report the purposes separately.

Where a vendor model's training data is unknown, do not claim the absence of contamination merely because the local team did not train it. Document the access limitation, use task-specific fresh material where appropriate and narrow the conclusion. A provider's broad statement about training practices may be relevant, but it needs to match the model and the particular claim under review.

Make reference answers auditable

Define correctness at the field and case level. The address service may need to preserve building, street, district and unit information while normalising presentation. If the input lacks a field, the expected behaviour may be to leave it unknown rather than invent it. The reference should identify what is known, what can be normalised and what requires clarification.

Have qualified reviewers adjudicate ambiguous cases without first treating the model's answer as the reference. Keep disagreements and their resolution. A reference may need more than one acceptable rendering, but the alternatives should preserve the same address meaning. Do not allow a broad similarity rule to excuse a different building or unit.

Check whether reference information was available to the system under the stated test conditions. An evaluator with access to a later corrected master record may judge the model against knowledge it was not permitted to use. That can be appropriate for one question, but the report must distinguish it from accuracy based solely on the supplied input.

Validate automated judges rather than treating them as ground truth

An automated judge can increase review capacity, but it is another measurement component. Record its model or rules, instructions, inputs and output interpretation. Test it against independently adjudicated examples that include material errors, harmless formatting differences, incomplete inputs and appropriate abstentions. High agreement on easy cases is not enough.

Inspect disagreement patterns. A judge may reward a longer explanation, ignore a digit transposition or prefer an output that resembles its own wording. Compare judgements without revealing which system produced each candidate where practical. If answer order can influence a comparison, vary or control that order and record the design.

For the address example, include a controlled pair where one output preserves unit 57 and another changes it to 75 while keeping every other field plausible. The test asks whether the evaluator notices the consequential difference. It does not estimate how often such an error occurs in normal traffic. Repeat with equivalent Arabic and English field presentations to test the measurement process across languages.

The NIST ARIA evaluation planning manual presents an approach combining model, adversarial and user testing. Its broad lesson for planning is that different tests answer different questions. The manual is not a binding audit standard, and invoking it does not validate an automated judge.

Reconcile denominators before comparing percentages

Consider an original hypothetical evaluation of 200 intended cases. The service returns 170 completed outputs, abstains on 20 and encounters technical failure on 10. Qualified adjudication finds 161 of the completed outputs correct and nine incorrect. A report can describe conditional correctness among completed outputs, but it must not silently present that as correctness across all attempted cases.

Worked denominator reconciliation for the hypothetical 200-case evaluation

Outcome or measure	Count or calculation	Interpretation
Correct completed output	161 cases	Meets the defined reference criteria.
Incorrect completed output	9 cases	Returned an output with a material error.
Abstention	20 cases	Requires separate review of whether abstention was appropriate.
Technical failure	10 cases	No completed result under the defined test conditions.
Conditional correctness	161 divided by 170, approximately 94.7%	Correctness among completed outputs only.
Correct completion across intended cases	161 divided by 200, or 80.5%	Share of all intended cases producing a correct completed output.

Neither figure alone determines suitability. Appropriate abstention can protect users from invented details, while excessive abstention can make the service unusable. Technical failure may reflect the test environment or the deployed system. Preserve those categories and investigate causes. The audit conclusion should connect the measures to the intended use rather than selecting the most flattering number.

Keep comparisons fair across systems and releases

Use the same task definition, reference criteria and permitted information when comparing candidates. If one system can query an authoritative address source and another cannot, the result measures two different service configurations. That comparison may be commercially relevant, but it should not be presented as a pure model comparison.

Record inference settings, tool availability, retry policy and resource constraints where they affect the result. A candidate allowed several attempts may produce a better final answer at a different cost or latency. Report the complete procedure and the outcome of failed attempts. Do not compare one system's best-of-many response with another's first attempt without disclosure.

Where outputs vary between runs, predefine the repeat design. Repeating only failed cases until they pass changes the evidence. Keep the original outcomes and distinguish repeated-run variability from performance across different cases. A deterministic-looking interface can still depend on changing provider infrastructure or retrieval material.

Interpret uncertainty at the right unit

Two hundred rows may represent fewer independent premises, customers or input templates. State which unit supports the generalisation. If related variants are deliberately included to test robustness, retain their grouping when reporting uncertainty. Treating each variant as an independent new address can make an interval appear more precise than the design supports.

Targeted challenge sets answer whether a failure can occur under selected conditions. They cannot estimate how often it occurs in the wider service population. Conversely, a representative sample may contain too few rare but consequential cases to test a particular boundary. The sampling guide explains how both forms of evidence can be used without confusing their meanings.

State which comparisons were planned and which were explored after seeing results. Multiple candidate systems, metrics and subgroups create opportunities to select a favourable pattern. Preserve the analysis history and report material adverse results. A performance claim should remain defensible when the reader sees the full decision process.

Review Arabic evaluation as a measurement problem

Arabic address inputs may include different digit forms, abbreviations, mixed scripts and transliterated place names. The test reference needs explicit rules for acceptable representation without asserting an unsupported official addressing standard. Include competent reviewers who can distinguish a harmless language variation from a change in location meaning.

Do not assume an English-oriented normalisation or similarity metric treats Arabic fairly. A preprocessing rule can remove a meaningful distinction or make different inputs appear identical. Review the transformation before interpreting the score. If the evaluation normalises both candidate and reference text, inspect whether that process hides the very error the application must avoid.

Keep language-specific counts and material errors visible. If Arabic inputs contain a different mix of incomplete addresses, a pooled score may primarily reflect case composition. A paired subset can test equivalent meanings across languages, while a service-representative sample addresses actual use. Explain the purpose and limitation of each.

Turn an integrity problem into an actionable finding

A finding should identify the unsupported claim, the evidence defect and the affected results. If cross-split premise overlap is found, quantify the overlap under the stated matching method and preserve disputed matches. Do not assume every overlapped case was memorised or that every performance result is invalid. Explain which conclusion is no longer supported and what a corrected evaluation must establish.

If the automated judge misses unit changes, the immediate action may be to suspend that judge for material field acceptance and use qualified review while it is corrected. This is an illustrative response, not a universal prescription. Reassess results that depended on the judge and identify the period and configurations affected.

Use retesting evidence to verify the corrected evaluation process. A new benchmark file is not enough if the same contamination path remains. Test the split, references, judge and calculation as relevant to the finding, then rerun the affected assessment with a documented procedure.

Check whether the tested conditions resemble the proposed service

A quality score from an offline run may not describe the service under operational constraints. Record whether the evaluation used cached responses, live reference lookups, background retries or manual corrections before scoring. These can be legitimate parts of a service, but they must be included in the claim. A buyer should not interpret a reviewed, corrected output as the first unaided response.

If latency or cost is reported, define the start and end of measurement and the units counted. Does elapsed time include queueing, retrieval, retries and required human review? Does cost include failed attempts? Compare like conditions and show relevant variation instead of relying only on an average. A fast result may be incomplete, while a correct result may arrive too late for the proposed workflow.

Test the effect of operational limits on the evaluation. A timeout setting may turn a difficult case into a technical failure; a shorter context limit may omit part of an address description; a changed reference source may alter available evidence. Preserve those settings with the result. If the production configuration differs, identify the gap and decide whether a focused operational test is needed before extending the performance claim.

Make the final evidence packet reproducible. It should connect the case inventory, permitted inputs, configuration, raw outputs, reference decisions, exclusions and calculation version. Record the date and identity of the person approving the interpretation. Another competent reviewer should be able to recover the reported counts and understand disputed cases without relying on a verbal explanation from the original experimenter.

Protect that packet appropriately. Address data can identify people or premises, so the audit should use authorised, minimised records and controlled access. Synthetic cases are useful for mechanism tests, while representative operational evaluation may need separately governed real data. State which was used. Realism and privacy are design questions to resolve together, not reasons to conceal the provenance of the benchmark.

Connect credible evaluation to a bounded audit conclusion

Evaluation integrity does not by itself establish security, fairness, legal compliance or operational suitability. It makes the performance evidence more trustworthy for its stated purpose. The fairness testing guide examines group burdens, while audit coverage records the wider questions that were and were not tested.

For systems evaluated repeatedly in service, connect result versions to the continuous audit programme. A trend is only interpretable when the task, population and measurement process remain comparable or their changes are explained. Preserve enough evidence for a reviewer to reconstruct the result at the time it informed a decision.

Prepare the performance claim, case inventory, split logic, references, evaluator configuration, experiment history and complete outcome counts. Discuss an audit of your AI evaluation evidence to identify whether the result supports the use being proposed and what further testing would resolve the material uncertainties.