LLM-as-Judge Evaluation for Agents

LLM-as-judge evaluation uses a language model to assess a defined property of an agent's work. It is useful for semantic questions that are difficult to express as exact rules, such as whether an explanation preserves a qualification. It becomes unreliable when a broad quality score is treated as proof that the agent acted correctly.

An AgentOps team should evaluate the judge before using its decisions to evaluate the agent. This guide sets out a calibration protocol, a concrete disagreement analysis and the operating controls needed to keep a model judge useful as part of AgentOps.

Give the judge a narrow decision

Start with the property being assessed. “Is this a good agent?” combines factual support, completeness, style, authority and business success into one vague question. Instead, ask whether a particular statement is supported by the supplied source, whether a required uncertainty was preserved or whether a response satisfies an explicitly anchored communication rubric.

Use deterministic checks for exact identifiers, required fields and calculations. Use authoritative application evidence for whether an action occurred. The judge should not infer permission from the politeness of an answer or completion from the agent's claim. The broader outcome-evaluation contract determines which evidence is needed beyond language assessment.

Specify the judge's inputs and allowed outputs. Include the task, relevant evidence, candidate result and criterion. Define pass, fail and insufficient evidence where appropriate. A forced binary decision can turn missing information into confident error. A numeric score needs anchors that explain what each level means.

Worked example: bilingual volunteer-role descriptions

The illustrative agent prepares Arabic and English volunteer-role descriptions from an approved role catalogue. The evaluation question is whether a description preserves the distinction between a preferred qualification and a required condition. The judge does not decide an applicant's eligibility, assign volunteers or approve publication. A volunteer coordinator retains those decisions.

For role VR-27, the source says previous event experience is preferred, and occasional weekend availability may be requested. A fluent description that makes experience mandatory and every weekend compulsory should fail the relevant criteria. Another description may preserve those conditions but omit an important accessibility contact. These are separate findings, not interchangeable reasons to give the same unexplained score.

A focused rubric for a volunteer-role description judge

Criterion	Passing evidence	Failure or uncertainty
Qualification fidelity	The description preserves the distinction between preferred and required experience.	It imposes an unsupported eligibility condition or the source is unavailable.
Commitment fidelity	The stated availability agrees with the supplied role record.	Occasional availability becomes an unconditional weekly commitment.
Role identity	A deterministic check confirms VR-27.	A different role identifier blocks the case before judging.
Language clarity	An anchored style rubric is satisfied.	Meaning is confusing despite factually correct statements.

Give the judge only the evidence needed for those criteria. A long catalogue dump can add irrelevant text and make it harder to determine which record is authoritative. Include source revision and the expected description language; keep the publication decision outside the model's output schema.

Build a human-adjudicated calibration set

Have suitable reviewers label representative cases independently before resolving disagreements. Preserve their rationales and the source spans supporting the agreed labels. If experts cannot consistently apply the rubric, refining the rubric may be more valuable than trying a stronger model.

Include clear passes, clear failures and borderline cases. Deliberately test a description that is elegant but unsupported, a correct short answer, a verbose answer with one material error, and Arabic wording that changes the strength of a qualification. Keep calibration examples separate from the final held-out test.

The original MT-Bench LLM-as-judge research identifies position, verbosity and self-enhancement biases, as well as reasoning limitations. Its benchmark results are not a reliability guarantee for a Saudi enterprise workflow or for this description task. Local calibration must match the actual criterion and language.

Read disagreement by consequence, not only agreement rate

Suppose a hypothetical held-out set has 80 adjudicated cases: 40 acceptable descriptions and 40 unacceptable descriptions. The judge correctly accepts 35 acceptable descriptions and correctly rejects 37 unacceptable ones. It incorrectly rejects 5 acceptable descriptions and incorrectly accepts 3 unacceptable descriptions.

Illustrative judge calibration results

Human reference	Judge accepts	Judge rejects
Acceptable: 40	35	5
Unacceptable: 40	3	37

Overall agreement is 72 divided by 80, or 90%. Yet the false-acceptance rate among unacceptable descriptions is 3 divided by 40, or 7.5%, while the false-rejection rate among acceptable descriptions is 5 divided by 40, or 12.5%. Publication risk and reviewer workload depend on different errors. An impressive aggregate can conceal an unacceptable false-acceptance pattern.

These assumed results describe one small, balanced test set. They do not estimate production prevalence or establish a universal threshold. Inspect the actual disagreements, their language, source quality and consequence. Report unresolved human labels separately rather than forcing them into the reference class that flatters the judge.

Test order, length and language effects

For pairwise comparisons, swap candidate order and compare the decisions. Randomising order helps expose an ordering effect but does not prove it has disappeared. If the same pair receives different winners after a swap, preserve that instability and route the case according to the agreed policy.

Keep substantive content constant while testing unnecessary verbosity. A longer description should not win merely because it sounds authoritative. Also test equivalent Arabic and English cases, mixed-script identifiers and negation. Do not use English agreement as evidence of Arabic calibration.

A 2025 primary study of judge position bias examines how ordering effects vary across judges, candidates and tasks. The operational lesson is to test the actual configuration rather than declare one model family universally unbiased. Record judge version, instructions and comparison format in each evaluation release.

Protect the evaluator from the content it assesses

Candidate text can contain instructions aimed at the judge. Treat it as untrusted material and clearly separate it from the scoring contract. Include adversarial cases that ask for a favourable grade, imitate a system message or claim an exemption from the rubric. A delimiter is helpful structure, not an independent security boundary.

Restrict evaluator tools and data access. A description judge normally needs no publishing credential or unrestricted repository search. Use agent guardrails to bound the evaluation service as deliberately as the production agent. A compromised judge should not gain authority over the workflow it is inspecting.

Require concise evidence references and criterion-level findings. Do not demand hidden chain-of-thought as an audit artefact. A source span and a specific altered condition are easier to verify than a lengthy persuasive explanation of why the judge feels confident.

Use abstention and human review intentionally

Define what happens when evidence is missing, the language is unsupported or the criterion cannot be applied. An abstention is not automatically a failed agent task; it means the evaluator cannot establish the answer. Keep evaluator failure separate from agent failure in reporting.

Route disagreements and consequential cases to reviewers with the right authority. Monitor queue size and ageing, because a judge that sends almost everything to people may add little value. The human-oversight operating model should define who resolves the issue and how a corrected reference label returns to the evaluation set.

Use risk-based sampling for apparently passing cases as well. Reviewing only failures reveals false rejections but can leave false acceptance invisible. Document sampling probabilities or strata when estimating production error so an intentionally difficult review sample is not mistaken for the whole population.

Version and monitor the judge as a production dependency

A judge change can move the quality score even when the agent does not change. Preserve the judge model, instructions, rubric, examples, schema and reference-set version. Run a bridge evaluation on the same cases before replacing a judge in a trend report.

Separate agent-release comparisons from evaluator-release comparisons. If both change together, the cause of a score movement becomes ambiguous. Release control should identify which component is changing, the evidence supporting it and the owner accepting the remaining uncertainty.

Include judge cost and latency in the operating design. Evaluation need not run on every task at the same depth. Use exact checks broadly, calibrated semantic evaluation where useful and specialist review where consequence demands it. The runtime budgeting guide helps bound repeated grading calls and retries.

Connect each judgement to a permitted use

A judge calibrated for helping reviewers prioritise cases is not automatically suitable for unattended publication. State the decision its output can influence and the consequence of a wrong pass. A requirement that all critical criteria pass should not be replaced by an average score that lets elegant language compensate for an invented eligibility condition.

Keep model-reported confidence separate from measured calibration. A confident explanation is not evidence that the confidence number predicts correctness. If confidence is used for routing, evaluate that routing rule on adjudicated cases and inspect the errors it sends away from human review.

In the volunteer-role example, a coordinator might use findings to focus review on experience and availability wording while retaining publication authority. Expanding that use requires a new evidence decision. Document the permitted use beside the rubric so a later dashboard consumer does not mistake a narrow semantic assessment for approval of the complete workflow.

Make the purchase decision about trustworthy evidence

Ask a supplier to show the calibration cases, disagreement matrix, language-specific results, abstention behaviour and change history. A dashboard containing judge scores is not evidence that the scoring method is fit for your task. Require access to enough records to reproduce material decisions within the agreed data controls.

Discuss calibrated agent evaluation with an anonymised example, the source that defines correctness, a draft rubric and the mistakes reviewers consider most consequential. The practical objective is a judge whose limits are known and whose output improves an accountable operating decision.