Comparable evidence for a selection decision
AI Vendor Evaluation Scorecard
Compare AI suppliers with mandatory gates, anchored ratings, transparent weights and sensitivity analysis, using a fully worked fictional scorecard.
An AI vendor scorecard makes competing proposals comparable when the buyer has already defined the problem. It should expose the trade-offs that matter: task quality, operating fit, controls, total cost, support and exit. A spreadsheet becomes misleading when unverified claims receive precise scores or a low price compensates for a requirement that cannot be compromised.
This guide is for procurement teams, evaluation panels and business sponsors choosing among AI providers. It includes an original scoring model and a worked comparison. The weights and ratings are illustrative, not market benchmarks or recommendations about real suppliers.
Set the evaluation boundary before issuing the request
Describe the intended users, tasks, languages, information, integrations and permitted actions. State what is excluded and what must be delivered at acceptance. Ask suppliers to distinguish available functionality, configuration work, custom development and future plans. Otherwise apparently similar proposals may describe different products or amounts of work.
The UK AI procurement guidance covers planning, evaluation and ongoing management. Use that lifecycle perspective without treating UK procurement guidance as Saudi law. A scorecard supports one selection decision; it does not replace contract management or specialist review.
Confirm the option itself is credible through the feasibility study. If the required information or action authority is unavailable, ranking vendors more carefully will not solve the underlying constraint.
Apply mandatory gates before weighted scoring
List the conditions that every acceptable proposal must satisfy. Examples may include an essential information-handling arrangement, a required interface, demonstrable access separation or a non-negotiable delivery responsibility. Define the exact evidence and responsible reviewer for each gate.
Mark a gate as satisfied, failed or unresolved. Do not use “unknown” as a favourable assumption. A failed essential condition makes the proposal unsuitable for the defined scope. An unresolved condition may justify further evidence or a bounded evaluation, but should not disappear into a weighted average.
Use vendor due diligence to verify the supporting claims. Where relevant, determine NCA applicability before specifying requirements. Do not ask every supplier to assert identical blanket compliance without understanding the buyer's actual scope.
Choose weights from the decision, not the demonstrations
Assign weights before viewing final evaluation results. Explain why each criterion matters to the service. A high-consequence document task may place more emphasis on verified quality and control evidence; another use may place greater emphasis on integration and operating support. The total should be clear and the criteria should not double-count the same advantage.
Use a common rating scale with criterion-specific anchors. A general starting point is 0 for no demonstrated capability, 1 for a major gap, 2 for partial fit, 3 for meeting the defined requirement, 4 for demonstrably stronger fit and 5 for exceptional verified fit against a stated anchor. Missing evidence should remain visibly unresolved rather than being casually labelled poor performance.
For cost, define rating bands from comparable total cost of ownership assumptions. Include required implementation, licensing or consumption, support, evaluation, maintenance and exit work where relevant. A low initial fee can coexist with a high cost to operate. The consulting pricing guide helps identify the commercial components to clarify.
Use a practical scorecard
| Criterion | Weight | Evidence basis | Vendor A rating | Vendor B rating |
|---|---|---|---|---|
| Task quality | 30% | Buyer-controlled Arabic and English retrieval evaluation | 4 | 3 |
| Operating and integration fit | 20% | Agreed workflow and interface demonstration | 3 | 4 |
| Controls and information handling | 20% | Reviewed evidence for the intended configuration | 4 | 4 |
| Total cost of ownership | 15% | Comparable scope, demand and support assumptions | 2 | 4 |
| Support and service responsibility | 10% | Contracted coverage and incident exercise | 3 | 4 |
| Exit and portability | 5% | Export inspection and handover terms | 2 | 4 |
Both fictional vendors have satisfied the mandatory gates for this example. Calculate each contribution as weight × rating ÷ 5. Vendor A totals 24 + 12 + 16 + 6 + 6 + 2 = 66 out of 100. Vendor B totals 18 + 16 + 16 + 12 + 8 + 4 = 74. Vendor A performs better on the weighted task-quality criterion, while Vendor B's broader fit produces the higher total.
The numbers do not make the decision by themselves. Review why the ratings differ, which differences matter in practice and whether the evidence is strong enough. Record the evaluator's explanation next to every material score so a later reviewer can reconstruct the reasoning.
Run comparable demonstrations and evaluation cases
Give suppliers the same task definition, permitted preparation, information boundary and output requirements. Include ordinary cases and relevant difficult cases. For the contract-search example, the task is to locate and reference relevant text, not provide unqualified legal advice or change contract records.
Keep an appropriate set of buyer-controlled cases separate from supplier tuning material. Use the PoC design guide to define the evaluation. Record the product version, configuration, assistance provided during the demonstration and failures as well as successful examples.
Ask evaluators to score independently before moderation where practical. During moderation, discuss evidence and criterion definitions rather than bargaining toward a preferred supplier. Preserve material disagreements and explain the final rating. A consensus number without reasons can hide uncertainty.
Keep evidence confidence beside the score
Add an evidence-status column to your working file: verified in the agreed test, supported by reviewed documentation, dependent on an unresolved condition, or not evidenced. This distinguishes strong performance evidence from a persuasive proposal. Do not multiply scores by arbitrary confidence factors unless the method has been explicitly designed and agreed.
Where evidence is missing, ask whether a limited additional test could change the selection. Avoid demanding an expensive demonstration for a criterion whose result cannot affect the decision. Conversely, do not rush past an uncertain requirement merely because one vendor leads on easier criteria.
If the supplier offers a phased engagement, specify what will be learned and when commitment increases. The engagement-model guide helps align commercial stages with evidence needs.
Test the sensitivity of the recommendation
In the worked example, move ten percentage points from total cost to task quality solely as a sensitivity scenario. The new weights are 40% for quality and 5% for cost, with other weights unchanged. Vendor A becomes 70 and Vendor B 72. The ranking remains the same, but the gap narrows from eight points to two.
This tells the panel that the recommendation depends partly on how the business values quality relative to cost. It does not authorise changing the official weights after the results to obtain a preferred outcome. If business priorities genuinely change, document the reason, agree the revised method and consider whether suppliers need an equal opportunity to respond.
Also test uncertain costs and operating assumptions. If a proposal's advantage disappears under a plausible support requirement, investigate that requirement before signing. A scorecard should reveal a fragile decision rather than disguise it with decimal places.
Record the selection and its conditions
The decision record should include the requirement, mandatory gates, agreed weights, evidence, scores, sensitivity, conflicts, recommendation and any conditions. Identify the authorised selection owner. Evaluators advise within their remit; they should not silently acquire commercial approval authority.
Disclose delivery or commercial interests among advisers and panel members. An interested party can provide useful technical input, but the buyer should control the method and decision. If the highest-scoring proposal is not chosen, document the legitimate reason and the authority for that decision.
Carry promised capabilities and unresolved conditions into the contract and acceptance plan. Otherwise the scorecard may evaluate a proposition that the delivery agreement never requires. Retain the evidence for future renewal and performance discussions.
Open for AI's agentic AI consulting can help define use boundaries, requirements and an evaluation plan. Bring a redacted requirement and comparable proposals to discuss a buyer-controlled AI vendor evaluation. Agree any advisory or delivery conflicts before assigning evaluation responsibilities.
FAQ
Your questions, answered.
How is an AI vendor scorecard different from due diligence?
Due diligence verifies claims and identifies conditions affecting whether a supplier is acceptable for the intended use. A scorecard compares acceptable proposals against agreed criteria. Connect them, but keep their decisions distinct: a strong weighted score should not conceal an unresolved mandatory requirement or unsupported supplier statement.
Should security and compliance be scored or treated as gates?
Use mandatory gates for conditions the buyer cannot compromise. Additional verified capabilities may be weighted where they create a legitimate trade-off. Define applicability and evidence first. Avoid a design in which a cheap price or attractive feature can compensate for failure of an essential information or authority requirement.
How should the panel score a capability with missing evidence?
Keep the evidence status unresolved and request what is needed to evaluate it. Do not confuse absence of evidence with demonstrated failure or assume the capability exists. Decide whether a bounded additional test could affect selection. If the missing evidence concerns a mandatory gate, resolve that gate before treating the proposal as fully acceptable.
Can we change the weights after supplier demonstrations?
Avoid changing official weights merely because the results are inconvenient. Use sensitivity analysis to understand the decision. If business requirements genuinely change, record the rationale, approve the revised method and consider the effect on fairness and supplier responses. Preserve the original evaluation so the change remains transparent.
What should we do when vendors have similar total scores?
Examine the material differences and evidence confidence rather than adding arbitrary decimal precision. Test whether unresolved assumptions could change the ranking. Consider the buyer's actual operating constraints and the cost of obtaining more evidence. Document the final trade-off and authority instead of treating a near tie as an automatic price contest.
How do we preserve buyer control if an adviser also sells implementation?
Agree the evaluation role, disclose relevant interests and keep requirements, scoring rules and selection authority with the buyer. Use separate challenge or recusal where appropriate. The adviser may contribute technical evidence, but should not shape criteria covertly to favour its own delivery offer or approve its own selection.