Source-linked AI summary
trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
Hadi Mohammadi
TL;DR
Outcome-only judges can miss process faults in tool-using agents when correct final answers conceal incorrect trajectories. This paper evaluates that blind spot with constructed ground truth and finds that outcome survival sharply changes recall, while step views improve silent-fault detection at higher cost. The study releases an offline-reproducible evaluation artifact, but its results are limited to one support-desk domain and local models.
Problem
Outcome-only evaluation lacks evidence about whether agents reached correct answers through valid processes, especially when customer-visible outcomes survive.
Method
The paper uses a deterministic support-desk environment, a tested oracle, single-fault replay mutations, and five judges evaluated on stratified trajectories.
Results
0.840 loud-fault recall falls to 0.451 for silent faults for the outcome-only judge, while the step judge reaches 0.766 silent recall with zero false alarms.
Takeaways & Limitations
Judge evaluations should report recall stratified by outcome survival because pooled recall hides process faults that outcome-only views cannot expose.
Takeaways & Limitations
The study uses one support-desk environment and local qwen2.5:14b and llama3.1:8b models, and does not measure frontier-judge performance.
Abstract
from arXiv · showhide
Outcome-only evaluation is the production default for LLM agents: show a judge the request and the final reply and ask whether it was handled well. The metric is structurally blind to an agent that reaches the right answer the wrong way. We measure that blind spot where ground truth is known by construction: a deterministic tool-using support-desk environment, a scripted oracle policy that always solves it, and a fault injector that breaks exactly one thing at a known step, stratifying faults by whether the customer-visible outcome survived (silent) or not (loud). Five judges (programmatic rules, outcome-only, step-rubric at two model sizes, and a self-consistency ensemble) are scored on detection, step localisation, fault typing, calibration, and cost over 400 trajectories. The outcome-only judge catches 84% of loud faults but 45% of silent ones while flagging 33% of correct trajectories; a step-rubric judge reaches 77% silent recall with zero false alarms at 3x the cost. No judge reads the final reply: an invented promise appended to an otherwise perfect trajectory evades the rules entirely and the step judge 82% of the time, and self-consistency triples cost while improving nothing. We argue that judge evaluations must stratify recall by outcome survival, and release the environment, the injector, all raw verdicts, and an analysis pipeline that rebuilds every number offline.
1 Introduction
Outcome-only judging can miss process faults whenever the final answer survives, so this paper constructs exact ground truth and evaluates judges by outcome survival. It compares five judge designs across detection, localisation, typing, calibration, and cost, with an offline-reproducible artifact.
- Outcome-only evaluation is structurally blind to skipped checks, tool misuse, or unsupported promises when the final answer is nevertheless correct.
- A deterministic oracle and single-fault injector assign exact fault, step, type, and outcome-survival labels without human annotation.
- 0.840 recall on loud faults falls to 0.451 on silent faults, showing why pooled recall obscures the key measurement.
- The study compares five judge designs on detection, localisation, typing, calibration, and cost using stratified evaluation.
2 Related work
Prior work studies judge reliability, agent success, failure attribution, process supervision, calibration, and mutation testing. This paper differs by testing evaluators of multi-step tool trajectories against labels known by construction and stratifying detection by outcome survival.
- Existing judge research documents position, verbosity, self-preference, objective, and difficulty-related pathologies against human labels.
- Agent benchmarks increasingly score trajectories or subgoals, whereas this paper fixes the agent as an oracle and evaluates the evaluator.
- Failure-attribution work and fault injection provide labelled traces, but this paper adds a five-way instrument comparison separating detection from attribution and outcome-survival recall.
- Process-supervision research supports step-level feedback, while this paper asks whether an off-the-shelf judge can detect process faults when the final answer is right.
- Mutation testing motivates the validity distinction that injected mutants need not represent real-fault prevalence while still measuring detector capability.
3 A testbed with ground truth by construction
The testbed combines a permissive support-desk environment, a provably correct oracle, and deterministic single-fault mutations whose replayed observations remain internally consistent. Exact trajectory labels enable rule-based and learned judges to be compared against known failures.
- The support desk encodes seven tools and procedures for identity, order lookup, policy, eligibility, refunds, escalation, and evidence-grounded replies.
- A permissive environment allows an unchecked refund, leaving the strict checker—not the world—to identify the process violation and expose silent failures.
- Each trajectory is labelled by fault status, failure step, one of six fault types, and whether its customer-visible outcome matched expectations.
- The six-step oracle is tested across all instances to verify every process rule and expected outcome rather than assuming correctness.
- Each mutation edits one call-list step, replays against a fresh environment, regenerates observations, and is constructed to break exactly one rule.
- The rule engine covers four of six fault types at 100% and two at 0%, because wrong-tool intent and reply prose are outside its rule representation.
- The judges are compared under controlled views, with outcome-only and step-based renderings differing in whether trajectory steps are included.
4 Five judges
The judge comparison controls model, decoding, procedure, taxonomy, confidence instructions, and context handling so that the principal experimental difference is the trajectory view available to each judge.
- Both LLM judges receive identical procedures and taxonomies, use grammar-constrained JSON, and differ principally in whether their rendering includes trajectory steps.
- Confidence is clamped to [0.5, 1.0], unparseable responses count as clean at confidence 0.5, and judges cannot access ground-truth labels.
- The evaluation fixes an 8192-token context and uses explicit de-biasing so truncation or an underspecified prompt does not become a hidden comparison factor.
5 Metrics and uncertainty
The evaluation separates detection, localisation, typing, calibration, and cost, while stratified bootstrap intervals preserve the fixed design structure and distinguish conditional localisation from trajectory-level detection.
- Detection uses precision, recall, and F1 over all 400 trajectories, plus silent recall, loud recall, and false-alarm rate on clean trajectories.
- Localisation is exact-match failure-step accuracy, scored only when both the label and verdict contain a step and the trajectory was flagged.
- Table 3 covers 400 trajectories—100 clean, 175 silent, and 125 loud—and defines silent recall as the outcome-surviving fault column.
- Typing is six-class macro-F1 on faulty trajectories, while false alarms affect precision rather than being charged again to typing.
- Calibration uses ten-bin population-weighted expected calibration error against binary verdicts, with Brier scores additionally reported in Appendix E.
- 95% stratified paired percentile-bootstrap intervals use 10,000 replicates within eight design cells, while exact Clopper–Pearson intervals replace them for proportions at 0 or 1.
6 Results
The results show that outcome-only recall substantially overstates detection of faults whose customer-visible outcome survives, whereas step-based judging improves silent-fault detection without false alarms.
- 0.840 loud recall versus 0.451 silent recall exposes the outcome-only judge’s outcome-survival blind spot.
- 0.330 of clean trajectories are falsely flagged by the outcome-only judge.
- 0.766 silent recall is achieved by the same-model step judge with zero false alarms across 100 clean trajectories.Of 257 flagged trajectories, all 257 were genuinely faulty.
- 0.973 step-exact localisation and 0.033 ECE make the step judge nearly exact and nearly calibrated among its scored cases.The outcome judge’s ECE is 0.253.
- 30.2 versus 10.4 seconds per trajectory shows that the self-consistency ensemble triples cost without improving quality beyond noise.
- Per-type recall must be compared against each judge’s false-alarm anchor row, because recall at that baseline is not evidence of detection.The 8B column is uniformly 1.00, including its anchor row.
7 Analysis
Changing only the judge’s view from outcome-only to step-level substantially improves detection of silent faults and removes false alarms, while exposing calibration and final-reply blind spots. Self-consistency adds cost without meaningful quality gains, and a high-recall baseline can be almost entirely indiscriminate.
- The blind spot is structural, not a weak model: 0.766 silent recall versus 0.451 results solely from changing the outcome judge’s view to step-level, while false alarms fall from 0.330 to zero.The paired difference is +0.314 with a 95% CI of [+0.240, +0.389].
- The blind spot is structural, not a weak model: 276 of 400 outcome-only verdicts use confidence 0.92, while ECE is 0.253 versus 0.033 for the step judge.The outcome-only judge is most certain where it is least accurate; the rule engine’s confidence uses two hand-set constants.
- Case study: nobody reads the final answer: 0.18 unsupported-claim recall for the step judge and 0.16 for the ensemble show that invented promises can evade trajectory judging even when the reply is visible.The rule engine catches this fault type 0% of the time because no rule reads the reply.
- The capability floor: 1.00 recall with a 1.00 false-alarm rate makes the always-say-faulty 8B baseline a capability floor rather than a useful judge.Its F1 is 0.852, reflecting the fault base rate, while its rationales average 57 completion tokens.
- Self-consistency: three times the cost, nothing back: 3× cost produces no meaningful quality improvement for the self-consistency ensemble, while calibration worsens by ECE +0.051.The ensemble costs 30.2 versus 10.4 s/traj, and systematic unsupported-claim misses persist across samples.
- The blind spot is structural, not a weak model: 0.923 detection F1 coexists with 0.606 type macro-F1, so the step judge detects nearly everything but often assigns the wrong fault type.It finds all 50 hallucinated_argument cases but labels 35 as wrong_tool.
8 What faults occur organically
The organically observed fault distribution differs sharply from the benchmark’s uniform injected faults. In 60 model-driven episodes, most observed rule-visible failures were premature stops concentrated in escalation cases, and silent faults were uncommon.
- Organic fault distribution: 13 of 60 organic episodes were flagged or reached a wrong outcome, including 10 loud and 3 silent failures.These rates were reported separately from the controlled comparison and are lower bounds because the rule engine is blind to two fault types.
- Organic fault distribution: 10 of 10 already_refunded episodes and 3 of 10 wrong_customer episodes failed through premature_stop.The model also made 91 escalate calls across 60 episodes and checked eligibility in only 30 episodes.
- Organic fault distribution: Uniform six-type injection measures judge capability rather than deployment frequency, because the organic fault mix is concentrated rather than uniform.The paper keeps the organic distribution separate from the controlled comparison.
9 Limitations
The study’s conclusions are bounded by a single support-desk domain, synthetic single-fault mutations, local models, and several measurement assumptions. The authors also identify taxonomy ambiguity and an unbalanced judged set as scope constraints.
- Scope and measurement boundaries: One support-desk environment limits generalization to open-ended coding or browsing agents without comparable rule engines.The paper measures only this domain and does not claim results for other settings.
- Scope and measurement boundaries: Single-fault mutations are cleaner than real failures, which can occur in cascades and have a different organic distribution.The injected benchmark breaks exactly one thing at one step, whereas the organic mix is concentrated rather than uniform.
- Scope and measurement boundaries: Results use qwen2.5:14b and llama3.1:8b local models, so frontier-judge performance is not measured.The authors state that a frontier judge might close part of the unsupported_claim gap but do not claim or evaluate that outcome.
- Scope and measurement boundaries: Confidence is self-reported for single-pass judges, while two fault types have a blurry attribution boundary.A policy fetched for an invented SKU can count as both an ungrounded argument and a wrong tool; detection is unaffected.
10 Reproducibility
The released artifact makes the study reproducible offline by pairing deterministic generation and mutation with raw verdicts, fixed-seed analysis, and tests that rebuild every reported result. Its environment, checker, and judge prompts also encode explicit boundaries around grounding, prose, and unverifiable procedure steps.
- Artifact and determinism: Every number regenerates offline from the released environment, oracle, injector, judges, 400 labelled trajectories, 2000 raw verdicts, and fixed-seed analysis pipeline.Continuous integration rebuilds tables and figures from committed raw verdicts and fails on drift.
- Artifact and determinism: Deterministic tests pin seeded instance generation, string-seeded mutation, byte-identical report rebuilds, and stratified dataset prefixes.Mutation replay also regenerates observations and checks that exactly one rule is broken.
- Operational boundaries: The environment excludes reply prose from world state, while the rule checker excludes prose from grounding and therefore cannot detect unsupported claims.Grounded arguments are traced to the goal or successful observations; escalate.reason and reply.text are deliberately treated as prose.
- Fault construction: The injector deterministically applies one call-list edit at a known step, replays it against a fresh environment, and recomputes outcome survival from the resulting trajectory.The six mutation types include wrong tools, hallucinated arguments, skipped preconditions, ignored observations, premature stops, and unsupported claims.
- Judge interfaces: The outcome judge sees only the goal and final answer, whereas the step judge uses a schema containing a failure step and failure type.The prompt explicitly says that unverifiable procedure steps are not evidence of failure, and its false-alarm rate is reported as 0.330.
E Full results
The full-results materials define the evaluation units, confidence and cost reporting, and per-type localisation analyses, while also separating organic agent episodes from the controlled comparison. They additionally document a major unsupported-claim detection gap and the ensemble’s lack of consistent quality gains.
- Evaluation design: Per-type detection recall is reported with 95% intervals, while localisation is measured within one step of the label and calibration uses Brier scores.Exact binomial intervals are used for observed proportions of 0 or 1, and bootstrap intervals otherwise.
- Cost: Cost reporting includes single-attempt wall-clock latency and serving-layer token counts, with the full run taking approximately 5.05 hours of judge time.The cost table includes transport and model loading in latency.
- Evaluation design: The judged set is resampled by trajectory across eight design cells, preserving clean, silent-fault, and loud-fault counts in every replicate.The study uses 10,000 bootstrap replicates with percentile intervals, including silent and loud margins of 175 and 125 trajectories.
- Ensembling: Ensemble quality deltas against one greedy pass straddle zero on every axis except calibration, where the ensemble is worse.This comparison is separate from the cost increase documented for judge runs.
- Organic episodes: Organic agent episodes are reported separately because their labels are lower bounds: the checker misses wrong_tool and unsupported_claim, and all flagged episodes were premature_stop.The 60 episodes had no failed tool calls, and three ended with empty final answers.