Supplier evidence quality
Supplier Evidence for AI Systems
Assess authenticity, relevance, reliability and sufficiency before relying on an AI supplier dossier.
Supplier evidence is useful when it supports a specific claim about the AI service you actually use. A large dossier can still leave the decisive question unanswered. The report may cover another product version, a different customer configuration, an old operating period or a metric unrelated to your decision. Audit work should establish authenticity, relevance, reliability and sufficiency before relying on the material.
This guide is for assurance, technology and supplier-management teams evaluating the quality of evidence already requested or received. It focuses on what the evidence establishes. Contractual acceptance rights belong in AI audit for procurement teams, while auditing vendor-controlled systems addresses strategies when access is restricted. Keeping these tasks distinct helps a buyer avoid treating a well-written contract or an impressive document bundle as proof of system behaviour.
Start with the claim, then identify the evidence object
Ask the supplier to state the claim in a form that can be examined. “Enterprise-grade AI” is not an evidence request. “The deployed service prevents one customer from retrieving another customer's records” identifies a boundary. “The predictor was evaluated on previously unseen room-use periods using the same operating threshold” identifies a performance claim. Each needs different evidence.
An evidence object might be a test result, configuration export, operating record, evaluation dataset description, signed report or incident record. Record its origin, version, date, scope, producer and relationship to the claim. A presentation summarising a test is different from the test's underlying records. Both may be useful, but they should not be assigned the same evidential weight without examining what was omitted.
NIST's 2026 supplier due-diligence guide treats provenance and supply-chain context as relevant to ICT supplier assessment. It is a cybersecurity resource, not proof of AI predictive quality. The practical approach below extends the evidence question to the particular AI claims the buyer needs to examine.
Four tests for evidence quality
Authenticity asks whether the material is what it purports to be and came from the stated source. Relevance asks whether it applies to this service, configuration, period and claim. Reliability asks whether the method and records are trustworthy enough for the intended use. Sufficiency asks whether the available evidence, taken together, supports the conclusion despite uncertainty and gaps.
These dimensions should not be averaged into a single score that conceals a fatal weakness. An authentic report on the wrong product is irrelevant. A relevant spreadsheet with unexplained formulas may be unreliable. Reliable results from a narrow sample may be insufficient for a broad claim. The auditor should identify which dimension is unresolved and what action could address it.
Evidence can be accepted for one purpose and unsuitable for another. A supplier architecture diagram may help define scope without proving that the depicted controls operate. A demonstration may confirm that a feature exists without establishing period-wide effectiveness. Make the intended use explicit in the evidence register so a planning document is not later cited as conclusive test evidence.
Worked example: meeting-room utilisation predictions
Consider a hypothetical facilities supplier whose model predicts whether a meeting room will be occupied during defined time windows. A customer uses the predictions to plan room availability and service schedules. The audit concerns the evidence supporting those predictions and the configured workflow. It does not assess building safety, certify occupancy limits or authorise emergency decisions.
The supplier provides a model card, a benchmark summary, a spreadsheet of evaluation outcomes, a platform security report, a customer configuration export and a change log. Management initially treats the bundle as proof that the deployed service is accurate and controlled. The auditor instead maps each item to a claim and checks whether the same model, threshold, input definitions and operating period appear across the records.
| Evidence item | Potential use | Question before reliance |
|---|---|---|
| Model card | Understand intended use and known limitations | Does it identify the deployed model and the customer's room-use context? |
| Benchmark headline | Locate claimed performance and evaluation method | What population, threshold and exclusions produced the number? |
| Outcome spreadsheet | Recalculate selected metrics and inspect exceptions | Are rows complete, labels credible and formulas traceable? |
| Platform security report | Examine covered platform controls | Does scope include this service, period and relevant customer responsibilities? |
| Configuration export | Identify operating threshold and enabled features | Is the export from the deployed customer environment and time? |
| Change log | Understand material changes after testing | Does it connect changes to affected claims and updated evidence? |
The outcome of this review is not simply “documents received”. Each item gets a disposition: usable for a named purpose, usable with a limitation, clarification required, superseded or not relevant. The register also names the reviewer and unresolved question. This makes it possible to explain why one missing annex matters more than several additional marketing documents.
Verify origin without mistaking integrity for truth
Confirm the issuer and delivery channel through an appropriate authorised route. For a third-party report, check the report's identity, scope and permitted use, and verify material uncertainty with the issuer where access and confidentiality arrangements allow. A logo, scanned signature or familiar filename is not enough. Avoid assuming that a document is false merely because it is incomplete; establish the facts first.
A cryptographic hash can help show whether a file changed after collection. It does not prove that the original contents were true or that the stated issuer produced them. A digital signature can support origin and integrity when its verification and trust conditions are understood, but it still does not establish that the audit method or conclusion is appropriate for your claim.
Record transformations such as redaction, extraction and conversion. A redacted report may remove information necessary to understand exceptions or scope. An exported spreadsheet may omit hidden sheets, formulas or filter settings. Preserve enough context to assess the evidence, while following authorised handling and minimisation arrangements. Evidence handling should govern access and retention rather than collecting sensitive supplier material indiscriminately.
Match the evidence to the deployed service
Check product name, service identifier, model version, configuration, region where relevant, customer-specific settings and the review period. A supplier may operate several services under one brand. A test of its general model does not necessarily cover the integration you use. A platform report may exclude customer-managed permissions or data preparation, leaving those controls with your organisation.
Time relevance needs more than a recent issue date. A newly published report can describe an old period. A model can change after the evaluated window while the document remains current in the supplier portal. Ask what changed between the tested baseline and the deployed configuration, and whether the change affects the claim. A bridge explanation can be useful, but it is not automatically equivalent to new operating evidence.
For the room predictor, a different occupancy threshold changes false-positive and false-negative outcomes even when the model is unchanged. A report at one threshold cannot be used without explanation to support another. Similarly, a model evaluated on booking records may not establish actual room use if bookings and observed attendance differ. The evidence needs to match the target being predicted, not just the name of the metric.
Reconstruct the denominator before trusting a headline
Suppose the hypothetical supplier's evaluation register contains 2,400 room-time windows. Of these, 240 lack a usable reference label and are excluded from the reported calculation. The remaining 2,160 contain 600 true occupied predictions, 120 false occupied predictions, 180 missed occupied windows and 1,260 correct empty predictions. These are invented figures for explaining evidence review, not a supplier performance claim.
On the labelled windows, correct classifications total 1,860, giving approximately 86.1% accuracy. The missed-occupied proportion among actually occupied labelled windows is 180 divided by 780, approximately 23.1%. The excluded 240 windows are 10% of the full register. Each calculation answers a different question. A single accuracy headline conceals both the consequence of missing occupancy and the missing-label population.
| Measure | Calculation | Interpretation limit |
|---|---|---|
| Labelled-window accuracy | 1,860 correct divided by 2,160 labelled windows: about 86.1% | Does not describe the 240 unlabelled windows |
| Missed occupied windows | 180 misses divided by 780 actually occupied labelled windows: about 23.1% | Depends on the reference labels and operating threshold |
| Missing reference labels | 240 divided by 2,400 registered windows: 10% | Missingness may be concentrated in difficult periods |
The auditor should ask why labels are missing, whether windows are duplicated and whether neighbouring observations are correlated. If missing labels cluster during peak use, the available subset may not represent the intended operation. The arithmetic can be correct while the inference is weak. Evaluation integrity addresses deeper questions about leakage, reference quality and the validity of performance claims.
Look for contradictions rather than collecting only support
Compare the model card with the configuration export and change log. A card may state that the system abstains when inputs are incomplete, while the customer configuration permits a default prediction. A benchmark may use one definition of occupancy while a dashboard uses another. These inconsistencies can be more informative than another certificate because they reveal where the claim and the deployed system diverge.
Do not jump from inconsistency to misconduct. The cause may be a stale document, an export error, a different threshold or a genuine control weakness. Record the contradiction and request evidence capable of distinguishing these explanations. The finding should state what is established and what remains a hypothesis. A supplier's explanation can resolve a factual mismatch, but should not replace corroboration when the issue is material.
Track superseded evidence. If the supplier replaces a report, retain the relationship between the versions and note why the old item is no longer relied upon. Otherwise a later reviewer may unknowingly cite an obsolete result. The evidence register should make the current reliance decision visible without erasing the history of clarification.
Assess independence at the level of the claim
A supplier-authored evaluation can contain valuable information, but it is not independent merely because it has a formal layout. An external report may have independence advantages while covering a different subject. Ask who selected the cases, ran the tests, set the criteria and reviewed the result. Also ask whether the reviewer designed the relevant control or benefits from a favourable outcome.
Independence and technical quality are separate. A poorly designed external test does not become persuasive through its author's distance from development. A rigorous internal test does not automatically satisfy a requirement for an independent conclusion. The audit should explain the contribution and limitation of each source rather than treating “third party” as a universal quality mark.
When is the evidence sufficient?
Sufficiency depends on the claim and consequences, not a fixed file count. For a narrow configuration question, a verified export and corroborating controlled test may be enough. For a claim about operation across a period, a design document and single demonstration are unlikely to answer the question. The reviewer should explain what coverage is required and why the available evidence does or does not meet it.
Use a claim-level disposition: supported within stated limits, contradicted by observed evidence, or not established because of a named gap. Avoid marking missing evidence as a pass. Equally, absence of a document does not prove a control failed unless the document itself was a required output. The report must distinguish a proven exception from a limitation on the conclusion.
If a gap remains material, identify the next evidence request and the decision it could change. For the room predictor, that may be reference-label records for excluded periods or results at the deployed threshold. A request for “all AI documentation” is less likely to resolve the question and can create unnecessary handling burden. The request should be narrow enough that the supplier knows what a satisfactory response would contain.
Trace a headline through its transformations
A useful evidence review follows a result from the original event population through exclusions, labels, calculations and presentation. Ask which transformations occurred and who controlled them. A spreadsheet may contain correct formulas while drawing from a filtered export that omitted difficult periods. A dashboard may round or aggregate a result so heavily that a material subgroup disappears. The audit should locate the transformation that changes the meaning, rather than treating every inconsistency as a model defect.
For the room predictor, compare the registered windows with the labelled subset and the reported table. Check whether the same window appears more than once after an export join, whether cancelled bookings are treated consistently and whether the reference describes actual occupancy or merely a reservation. These are proposed evidence checks for the example; their relevance depends on the service's real data definitions. A result cannot be interpreted responsibly until those definitions are understood.
Use a small reconciliation record that names the source, transformation, resulting population and unresolved difference. If the supplier corrects a join error, retain the old and new counts and the reason for the change. If it explains that two reports use different populations, record both purposes. The objective is not to force every number to match; it is to understand why legitimate differences exist and identify those that remain unexplained.
Make the reliance decision reviewable by another person
For each important evidence item, write a short rationale that goes beyond “reviewed”. State the claim supported, checks performed, material limitations and any corroboration. A second reviewer should be able to understand why the item was accepted for that purpose. If reliance depends on a supplier statement that could not be verified, say so explicitly rather than allowing the statement to become an established fact in later summaries.
Consider two apparently contradictory documents: a model card permits a particular input range, while an evaluation report excludes part of that range. The correct response is not automatically to reject both. Determine whether the card describes intended capability and the report describes a narrower tested population. If so, the evidence may support a limited claim while leaving the remainder unestablished. Procurement and management can then decide whether that remainder matters to their use.
Set a refresh trigger for evidence whose relevance can expire. A model replacement, changed threshold, revised data source or material incident may require a new assessment of reliance. The trigger should name the affected claim and owner, not simply require an annual document upload. A fresh file that repeats an old evaluation is not necessarily fresher evidence. This discipline keeps the dossier connected to the service rather than to a calendar reminder.
Before accepting the dossier, ask a reviewer who did not prepare it to follow one important claim from headline to source and explain every exclusion.
A usable evidence request and review outcome
An original request for this example could ask: “Provide the evaluation population definition, exclusion reasons and outcome counts for the deployed model and threshold, including the relationship between booking data and the reference occupancy labels. Identify changes after the evaluation and any customer configuration excluded from the results.” This is precise without demanding unrelated proprietary information.
The review outcome should record what was supplied, which checks were performed, what contradictions were resolved and what remains open. If the supplier cannot provide a requested item, document the reason and the effect on reliance. Possible next steps include alternative evidence, controlled observation, a narrower conclusion or a procurement decision about the remaining uncertainty. The audit should not invent an equivalent source just to close the request.
For an audit scoping discussion about supplier evidence, bring a masked claim list and document inventory rather than an uncontrolled archive. Open for AI's AI auditing services can assess the evidence needed for the decision and identify where supplier participation is essential. A useful dossier makes the path from claim to conclusion traceable; its value is not measured by the number of files it contains.
FAQ
Your questions, answered.
What supplier documents are needed for an AI audit?
Begin with the claims to examine rather than a universal document list. Relevant items may include evaluation methods and results, configuration records, change history, operating evidence and reports covering specific controls. Each should identify its scope, version, period and producer. A model card or certificate alone rarely answers every question about the deployed customer workflow. Ask how each item relates to the customer’s actual configuration, not merely the supplier’s generic product.
Does a supplier security report prove AI accuracy?
No. It may support the controls actually covered by its scope and period, subject to its conditions and exceptions. Predictive or generative performance needs relevant evaluation evidence. Check customer responsibilities and excluded components as well. A useful evidence map links each report to the specific claim it supports instead of treating one third-party document as comprehensive AI assurance. Read exceptions before deciding how far the report can be reused.
How can an auditor check that supplier evidence is authentic?
Confirm origin through an appropriate authorised channel, inspect issuer and report identity, and resolve material doubts with the issuer where permitted. Record how the material was received and transformed. Hashes can show that a collected file has not changed, but do not prove the original statement is true. Authenticity must be considered alongside relevance and methodological reliability. Resolve material inconsistencies between the delivered file and the issuer’s confirmed version.
Can an old evaluation support a newly deployed AI version?
Only to the extent that its relevance can be justified. Identify changes to the model, inputs, threshold, configuration and workflow since evaluation, then assess which claims those changes affect. A recent document issue date does not necessarily mean recent testing. Material changes may require additional evidence or a narrower conclusion rather than a simple statement that the old report remains current. Separate the document issue date from the actual evaluation and operating periods.
What should happen when a supplier excludes failed or unlabelled cases?
Ask for the population definition, exclusion reasons and counts, and examine whether exclusions are concentrated in difficult conditions. Recalculate metrics using the stated denominator and keep missing outcomes visible. Exclusion may be justified, but it limits what the result describes. A correct calculation on available cases does not automatically represent the full intended operating population. Do not treat an unknown outcome as a correct prediction simply to complete the table.
Is a supplier model card enough to establish audit readiness?
It can explain intended use, evaluation context and limitations, but it is not automatically evidence of the deployed configuration or operating effectiveness. Compare it with current configuration, change records and relevant tests. If the card is generic or outdated, record that limitation and request the evidence needed for the specific claim rather than treating the card as a pass. Useful readiness evidence also identifies who can answer questions about the underlying evaluation.
Should missing supplier evidence be classified as a failed control?
Not automatically. Missing evidence can prevent a conclusion without proving that the underlying control failed. If retaining or providing that evidence is itself an agreed requirement, the absence may support a separate finding. Keep the evidence limitation, any proven breach and the remaining uncertainty distinct so management can choose the appropriate next action. If evidence retention is the failed requirement, state that requirement and its scope explicitly.
How much supplier evidence is enough?
Enough means sufficient relevant and reliable evidence for the particular claim and intended conclusion, not a fixed file count. A narrow configuration question may need a small corroborated set; a period-wide operating claim may need more extensive records and selection. The reviewer should explain remaining gaps, their effect on reliance and what additional evidence could change the decision. Record why the selected evidence is sufficient, including any corroboration and remaining conditions.