AgentOps Scorecard Design

An AgentOps scorecard should help an owner decide whether to continue, investigate, restrict or change a service. It connects outcomes, operating controls, human effort and cost to a defined population of work. A page full of green metrics is not useful if the numbers describe different tasks or hide an important weak cohort.

This guide is for operations leaders designing a scorecard for production agents. It uses a bilingual procurement-catalogue enquiry example to show how an aggregate result can improve while both language cohorts worsen. Within AgentOps, the scorecard should make that contradiction visible before it shapes a release or investment decision.

Start with the decision and the unit of work

Define one task in business terms. For the illustrative catalogue agent, a task is one eligible enquiry asking for an item description or catalogue reference within the approved scope. The agent does not place orders, approve suppliers or commit expenditure.

Define an accepted outcome before calculating the score: the answer addresses the enquiry, uses the permitted current catalogue evidence, preserves relevant uncertainty and reaches the required review or delivery state. A held enquiry can have a valid operational disposition without being counted as an accepted answer.

Count unique tasks, not every model call or retry. Record the cohort entry rule, observation window and exclusions. The outcome evaluation framework provides the acceptance definition; the scorecard explains how that definition becomes a consistent, decision-relevant metric.

Worked example: a better aggregate can conceal worse cohorts

Assume two illustrative reporting periods, each with 100 eligible enquiries and the same acceptance rule at the agreed observation point. The language mix changes between periods. The numbers are a teaching example, not observed OpenForAI performance or evidence about any language model.

Illustrative acceptance by language cohort

Period	English enquiries	Arabic enquiries	All eligible enquiries
A.	36 accepted out of 40: 90%.	36 accepted out of 60: 60%.	72 accepted out of 100: 72%.
B.	68 accepted out of 80: 85%.	10 accepted out of 20: 50%.	78 accepted out of 100: 78%.

The overall result rises from 72% to 78%, even though the English cohort falls from 90% to 85% and the Arabic cohort falls from 60% to 50%. More of period B's work belongs to the higher-performing English cohort. The aggregate calculation is arithmetically correct, but an unqualified claim that quality improved would be misleading.

Show the counts and the mix beside the percentages. Investigate task complexity, source coverage and operating changes before attributing the difference to language alone. The example demonstrates an aggregation problem; it does not establish statistical significance or a causal explanation for any real service.

Design a metric contract for every important number

A metric contract should state its purpose, eligible population, numerator, denominator, timing, evidence source, owner and decision trigger. Record how repeated attempts, cancelled work, unresolved tasks and missing evidence are treated. Keep the definition versioned when the service changes.

For accepted-answer rate, the denominator might be all eligible enquiries entering a defined cohort, with status assessed at a declared cut-off. If some tasks remain pending, show that count and the age of the pending work. Do not remove them silently to make completion look stronger.

A quality sample has a different denominator from the full task population. Report how it was selected and which outcomes were independently reviewed. A score based only on completed tasks can miss the hardest cases that never reached completion.

Use several decision views without collapsing them into one score

Proposed scorecard views and their decisions

View	Example measure	Decision it supports
Useful outcomes.	Accepted answers with eligible counts and cohort breakdown.	Continue, investigate quality or narrow scope.
Control integrity.	Observed unauthorised actions, blocked attempts and evidence coverage.	Contain or repair a control boundary.
Service delivery.	Task age, completion within the agreed window and unresolved work.	Adjust capacity or response.
Human work.	Review effort, rework and exception backlog.	Improve evidence or resource the intended review.
Resource use.	Cost by accepted task and relevant workload cohort.	Investigate waste or revise the operating design.

A critical action-control failure should not be averaged away by low cost and fast response. Keep blocking conditions visible beside performance measures. Similarly, zero observed incidents does not prove a control worked if the relevant activity was not observable.

The SLO guide defines service commitments and eligible populations. The scorecard can show their status, but it should not quietly redefine an SLO by changing dashboard filters or excluding difficult tasks after the fact.

Show uncertainty and evidence coverage

Distinguish an outcome known to be unacceptable from one whose final state cannot be established. Report missing confirmation, missing review evidence and unclassified cases separately. A completeness metric helps the reader judge how much confidence to place in the performance result.

For small cohorts, counts may be more informative than a large percentage movement. A change from one to two failures can look dramatic without enough evidence to support a broad conclusion. Use a suitable statistical method when making statistical claims; do not attach significance language to a dashboard comparison by habit.

Where a model judge contributes to a metric, disclose its role and the human calibration or review arrangement. Do not present an unvalidated judge score as independent proof of business acceptance. Keep scorecard interpretation linked to the actual evidence method.

Segment only where the split changes a decision

Language is one useful dimension for the catalogue example. Others may include item family, source completeness, enquiry complexity, deployment version or review route. Choose segments that expose a meaningful operating difference and can be acted on by an owner.

Avoid hundreds of tiny slices that generate apparent patterns without enough evidence. Start with the material cohorts, inspect their counts and investigate a specific hypothesis. The Arabic and English operations guide addresses language-sensitive evaluation; this scorecard keeps the reporting populations and decisions explicit.

When cohort definitions change, preserve the old view or explain the break in comparability. A new definition can improve measurement, but it should not be presented as an improvement in agent performance merely because the headline number changes.

Connect cost to the same outcome population

Use costs from the same scope and period as the outcomes being compared, with a clear allocation rule for shared resources. The FinOps Foundation's unit-economics guidance distinguishes resource measures from business-oriented units. The practical implication is to make the cost denominator meaningful to the service decision.

The cost per accepted task guide provides a detailed method. In the scorecard, show whether a lower unit cost came from less waste, more accepted outcomes, reduced review effort or a changed cohort mix. A saving achieved by skipping required review is a control issue, not an efficiency success.

Give the review meeting a set of possible actions

For each material movement, ask what changed, which cohort is affected, how strong the evidence is and who owns the next step. The period A/B example should trigger investigation of both language cohorts, even though the aggregate appears better.

Define actions before the meeting: continue observing, inspect a sample, repair a source, change capacity, restrict an action or pause a release. Tie the response to consequence and evidence rather than setting arbitrary universal thresholds. Record the decision and the next observation that will confirm or challenge it.

Use the post-mortem process when a metric movement reflects a material incident or recurring control failure. Routine scorecard review and incident learning should inform each other without turning every small fluctuation into an emergency.

Check for incentives that distort the work

A target can change behaviour around the agent. If teams are rewarded only for fast completion, they may move difficult enquiries to an excluded queue or accept a weak answer rather than preserve uncertainty. If cost is the sole target, required review may be treated as avoidable overhead. Inspect how the metric affects the process it is intended to improve.

Use balancing evidence. Faster answers should be read alongside accepted outcomes, rework and unresolved tasks. Lower review effort should be checked against the quality of decisions and the suitability of the cases entering review. This does not require a large new metric for every possible incentive; it requires an owner to ask what behaviour would improve the number without improving the service.

For the catalogue agent, inspect whether complex enquiries are being reclassified as out of scope after entry. Some exclusions may be legitimate, but the rule and decision need to be visible. Preserve the original intake cohort and show how cases were disposed of. Otherwise the scorecard can reward moving work out of sight rather than resolving it. Review a small evidence sample whenever a surprising improvement cannot be explained by a known operational change.

Keep the scorecard reproducible

Retain the metric definitions, query or calculation version, source references and relevant cut-off. Verify that sampled underlying tasks reconcile with the displayed counts. If the data pipeline drops a category, the scorecard should show a coverage problem rather than silently improving the result.

Discuss an AgentOps scorecard with the outcome definition, one reporting cohort and the decisions the scorecard must support. The objective is a small set of interpretable measures that reveal operating choices, including the uncomfortable cases a single aggregate can conceal.