PaperlayerView exact revision on arXiv

Source-linked AI summary

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

Umesh Bodhwani, Thanh Tran, Kai Wei

arXiv:2609.12191v1cs.CLcs.LG

TL;DR

Task-oriented LLM agents are often selected using a low-cost simulator-plus-judge gate, but its ranking against verifiable reward and its relationship to task success need assessment. GAUGE audits the gate across benchmarks and finds a satisfaction–success gap alongside reduced resolution among near-equal agents.

  • Problem

    Teams lack a clean held-out test set for live user experience, while controlled human studies per configuration are prohibitively slow and agent configurations change often.

    Source: §1 · p. 1

  • Method

    GAUGE is a reusable offline protocol that separates ranking validity from construct validity and evaluates simulator-plus-judge rankings against verifiable reward.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    57.5% of conversations rated satisfied by the blind panel failed the task, and satisfaction was decorrelated from success across five rater populations and two substrates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A calibrate-then-trust cadence uses a judge-free completion bit as a zero-cost tripwire for truncation regressions, while out-of-sample recalibration does not transfer.

    Source: §1 · p. 2

    Generalization to live customer interactions remains untested, and the τ²-bench user instructions are synthetic.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Comparing and selecting task-oriented LLM agents increasingly relies on a low-cost offline evaluation gate: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. We introduce GAUGE, a reusable offline protocol that measures whether this gate's ranking matches a grounded verifiable reward across 25 agents from six providers on the $\tau^2$-bench and SimulatorArena benchmarks, separating two kinds of evaluation validity that release practices conflate: ranking validity and construct validity. First, a satisfaction-success gap: satisfaction carries essentially no information about task success, as conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer's task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated. Second, while the gate's ranking is robust across the broad capability span, it loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from $<$1% on wide-reward pairs to 31% on close pairs. The gate is thus human-validated yet mis-anchored. As a remedy, we propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions.

1 Introduction

GAUGE audits whether the low-cost simulator-plus-judge gate ranks agents like a verifiable reward and whether satisfaction reflects task success. It finds that broad ranking validity coexists with a satisfaction–success gap and unreliable decisions among near-equal agents.

GAUGE protocol

  • GAUGE separates ranking validity from construct validity in a reusable, fully offline audit of the simulator-plus-judge gate.The protocol is applied to 25 agents, six providers, four judges, two substrates, and approximately 3,700 transcripts.Source: §1 · pp. 1–2

Ranking validity

  • The gate promotes the lower-reward agent on 31% of close pairs, versus <1% of pairs with widely separated rewards, despite strong aggregate ranking validity.The introduction reports an aggregate ranking correlation of ρ=0.94.Source: §1 · p. 2

Calibrate-then-trust

  • The audit is intended as periodic calibration: teams learn a trusted operating region with a verifiable reward, then use the cheaper gate within it.The paper applies the approach where an offline verifiable reward is obtainable, including the τ²-bench oracle and SimulatorArena correctness grades.Source: §1 · p. 2

Satisfaction–success gap

  • 57.5% of satisfied conversations failed the task, and satisfaction carried essentially no information about success across five rater populations and two substrates.The paper reports a satisfaction–success correlation of ρ=−0.147.Source: §1 · p. 2

Calibrate-then-trust

  • A judge-free completion bit is a zero-cost tripwire for truncation regressions, but out-of-sample recalibration does not transfer.On broken-versus-working cases, the completion bit scores ρ=0.87 versus 0.80 for the full gate.Source: §1 · p. 2

Guidance

  • Subjective approval is decorrelated from task success across all dimensions, while outcome-grounded judging succeeds where transcript-only satisfaction fails, indicating evaluator reliability depends on evidence access, not model capability.The gate is valid only within a bounded operating region, requiring re-auditing on configuration change rather than on a fixed schedule.Source: §1 · p. 2

2 Related Work

Prior work studies user simulators, agent benchmarks, LLM-judge calibration, and measurement validity. GAUGE distinguishes itself by testing simulator-plus-judge ranking against a non-LLM verifiable reward and measuring the satisfaction–success gap.

User-simulator reliability

Agent and customer-agent benchmarks

  • Agent benchmarks include persona-simulator pass rates and verifiable substrates; GAUGE tests whether those pass rates rank agents like grounded reward and surfaces the satisfied-but-failed inversion.The passage cites ECom-Bench, τ-bench, and τ²-bench.Source: §2 · p. 2

LLM-as-a-judge calibration to humans

Measurement validity and proxy gaming

3 Method

GAUGE audits whether an offline simulator-plus-judge gate ranks agents like a non-LLM verifiable reward, separately measuring ranking validity and the satisfaction–success construct gap. It scores agents at the release-decision level, aggregating across personas and domains.

Problem formulation

  • GAUGE compares agent-level gate scores with verifiable reward and measures the failure rate among transcripts a signal rated satisfied.The protocol aggregates over persona strata and domains because per-cell scores are too noisy for release decisions.Source: §3 · p. 3

Substrates and ground truth

  • The audit runs agents against persona simulators on τ 2-bench and SimulatorArena, using non-LLM oracle or human-graded correctness as ground truth.τ 2-bench covers retail and airline; SimulatorArena provides a replication substrate in math tutoring.Source: §3 · pp. 3–4

Agents

  • The cross-provider ladder evaluates 25 model-temperature configurations from 14 models across six providers on 114 retail and 50 airline tasks.A separate controlled-degradation set uses 12 Sonnet-4.5 configurations across six strata and two domains to provide known-quality tiers.Source: §3 · p. 4

Judges and humans

  • Four judges apply a policy-aware rubric to full transcripts, while a process-blind shopper proxy and a blind human panel provide separate satisfaction ratings.The panel rates a stratified 150-transcript sample; the four judges include Opus-4.8 and GPT-5.5 as primary and out-of-family judges.Source: §3 · p. 4

Metrics

  • GAUGE reports agent-level Spearman correlation, broken-versus-working AUC, power, out-of-sample recalibration, and panel reliability.Correlation uncertainty uses model-clustered bootstrap confidence intervals.Source: §3 · p. 4

4 Results

Across the benchmarks, satisfaction ratings do not track task success, while the gate’s strong broad ranking loses resolution on near-equal agents. Judge robustness, persona variation, and a zero-cost completion signal further define where the audit and gate can be trusted.

4.1 Satisfaction does not imply task success

  • The policy-aware gate improves on the base failure rate, whereas satisfaction matches it; on the controlled degradation, reward falls from 0.66 to 0.05 while satisfaction stays nearly flat.The policy-aware gate is only partly sensitive to degradation (0.54 →0.21).Source: Table 2 · p. 5, §4.1 · p. 5, Figure 3 · p. 5
  • 57.5% of conversations rated satisfied by the human panel failed the task, and satisfaction’s success correlation was non-positive (ρ= −0.147).Satisfied-but-failed rates were concordant across five rater populations (47.6–59.5%) and appeared across retail, airline, and math tutoring.Source: §4.1 · p. 5

4.2 Ranking is robust across providers

  • 31% of near-equal pairs are mis-ranked by the Opus-4.8 gate, versus <1% of wide-reward pairs, despite ρ=0.94 ranking validity across 25 agents.Near-equal pairs differ in reward by less than 0.1; the gate cleanly separates broken from working agents on the controlled-degradation set (AUC=1.00).Source: §4.2 · p. 6

4.3 Judge robustness and self-preference

  • The four judges recover verifiable-reward ordering (ρ=0.84–0.94), but Opus-4.8 inflates Claude agents over GPT-5.5 by +0.75/7.The ordering remains stable across judges, while the self-preference effect leaves that ranking intact.Source: §4.3 · p. 6

4.4 Personas stress-test agents

  • The six-persona population lowers mean reward by 42% (0.47 →0.27) versus the default baseline, exposing failures masked by a single cooperative user.Across personas, satisfaction varied without tracking success, while whole-set reward ranking remained ρ=0.83.Source: §4.4 · pp. 6–7

4.5 A free signal catches severe regressions

  • The zero-cost completion bit reaches ρ=0.87 on controlled degradation but only ρ=0.51 on the six-provider grid, where just 3.5% of failures truncate.Of 1,485 grid failures, 96.5% terminate normally; the paid judge retains signal for semantic failures (ρ=0.37, AUC 0.72).Source: §4.5 · p. 7, Table 3 · p. 7

4.6 Negative results and additional analyses

  • Out-of-sample persona reweighting and judge averaging do not reliably improve the gate, whereas calibrated abstention halves error on pairs it ranks.Reweighting falls from ρ=0.80 in-sample to ρ=0.68 leave-one-stratum-out; abstention reduces error to 14.8% on ranked pairs.Source: §4.6 · p. 7

5 Discussion and Guidance

A gate can rank agents well yet mis-anchor evaluation to human satisfaction rather than task success. The discussion recommends certifying both experience and task completion, with verifiable checks alongside satisfaction signals.

Discussion

  • Satisfaction and task completion are complementary axes, so certifying human satisfaction alone does not guarantee ranking validity against success.The proposed cadence uses a cheap signal on each configuration change and triggers a verifiable audit for judge or simulator changes, candidate-pool shifts, or near-equal candidates.Source: §5 · p. 7

Guidelines

  • Never gate on satisfaction alone, especially during optimization; pair it with a verifiable check and aggregate scores to the variant rather than a variant-persona cell.For same-provider judge-candidate pairs, calibrate thresholds against an out-of-family judge because +0.75/7 self-inflation can flip an accept/reject bar.Source: §5 · p. 7

Limitations

The human panel was small, and the controlled-degradation set covers budget-style truncation failures rather than every failure mode. The study also uses synthetic τ 2-bench instructions, leaving generalization to live customer interactions untested.

Panel size

  • Three annotators rated 150 transcripts; a larger panel would tighten per-dimension confidence intervals.The headline result remained within the 56.4–60.5% band when any one annotator was removed.Source: Limitations · pp. 7–8

Degradation coverage

  • Budget-style perturbations detect truncated agents rather than every failure mode; the capability ladder adds naturalistic-failure coverage.Source: Limitations · p. 8

Synthetic substrates

  • τ 2-bench user instructions are synthetic, and generalization to live customer interactions remains untested.Absolute pass rates are not leaderboard-comparable because the harness uses Bedrock with a tolerant output parser.Source: Limitations · p. 8

Ethics Statement

The human study used de-identified transcripts from synthetic tasks and was IRB-exempt; its annotators were consenting adults compensated above the local minimum wage.

Study materials

  • The study rated de-identified synthetic-task transcripts with no personal data and was IRB-exempt.Source: Ethics Statement · p. 8

Annotators

  • Three consenting adult annotators were representative lay raters and independent of the project team.Source: Ethics Statement · p. 8

Compensation and blinding

  • Annotators were compensated above the local minimum wage and rated blind to AI scores, variant, and outcome.Source: Ethics Statement · p. 8

A Satisfied-but-Failed Rates by Rater Population

Satisfied ratings frequently co-occurred with task failure across rater populations and subjective dimensions. Controlled degradations expose the mismatch, while gate scores generally rank success but do not provide a reliable calibrated probability.

A Satisfied-but-Failed Rates by Rater Population

  • Every rater population exceeded the lenient 20% false-accept ceiling; τ 2 satisfaction signals sat at the base failure rate.The human panel and four LLM scorings on τ 2 rejected the ceiling at p < 10^-6, and SimulatorArena human ratings did so at p=0.013.Source: A Satisfied-but-Failed Rates by Rater… · p. 11, Table 4 · p. 11

B The Construct Gap Is Not Specific to Satisfaction

  • All five subjective dimensions were weakly associated with success (|ρ| ≤0.17), and 56–64% of highly rated conversations failed.The dimensions were highly collinear (pairwise Spearman 0.61–0.97), suggesting they measure a similar favorable-interaction signal.Source: B The Construct Gap Is Not Specific to… · p. 11, Table 5 · p. 11

C The Controlled-Degradation Set

  • The 12 flag-only configurations held the prompt and tool set fixed while varying inference-time flags, isolating behavior changes from provider or prompt-style confounds.The set was designed as a positive control with known good, medium, and degraded configurations.Source: C The Controlled-Degradation Set · pp. 11–12
  • Degraded configurations averaged 0.05 verifiable reward versus 0.65 for good/medium configurations, while subjective signals still rated degraded agents mid-scale.D7 had reward 0.00 and the highest degraded-configuration human satisfaction, 4.64, after a step cap cut off the task midstream.Source: C The Controlled-Degradation Set · p. 12, Table 6 · p. 12

C.1 Gate-score calibration

  • Both gate-score series rose overall with success (ρ=0.86 controlled set; ρ=0.96 six-provider grid), but the controlled set's mid-range dip was not significant.The bins were small and intervals overlapped; the authors therefore treat the gate as a ranking instrument rather than a calibrated probability.Source: §C.1 · p. 12

D Second-Provider Simulator Replication

The construct gap persisted after changing the user-simulator provider, and the full agent ranking remained similar. Absolute satisfied-but-failed rates differed because the simulators produced different trajectories and rewards.

Construct-gap replication

  • Under both simulators, satisfied-but-failed rates exceeded 20%: GPT-5.4 yielded 66.7% for the gate and 69.8% for the proxy; Sonnet-4.5 yielded 43.6% and 52.9%.All rates had p < 10^-7; the authors interpret persistence as a qualitative invariant, not a rate comparison.Source: D Second-Provider Simulator Replication · pp. 12–13

Gate-score calibration

  • Gate-score calibration was near-monotonic on the six-provider grid (ρ=0.96); the controlled set's mid-range dip was not significant.The figure reports empirical success by gate score with Wilson 95% confidence intervals.Source: Figure 5 · p. 13

Construct-gap replication

Ranking replication

  • The full 25-agent ordering was preserved under GPT-5.4 simulation (ρ=0.93), close to ρ=0.94 under Sonnet-4.5, while top-11 agreement was weaker (ρ=0.51).The rerun used 114 tasks per agent and 2,850 transcripts with agent, tasks, rubrics, judge, and oracle held fixed.Source: D Second-Provider Simulator Replication · p. 13

E Per-Agent Data Behind Figure 4

Across agents, satisfaction surplus varies with capability and can rank agents differently from verifiable task success. All four judges nevertheless recover the broad 25-agent reward ordering.

E Per-Agent Data Behind Figure 4

  • Surplus declines with reward across 25 agents (Spearman ρ= −0.51, p = 8.5e−3, n=25), from +0.20 to +0.40 for Mistral and Qwen to near zero or negative for frontier agents.Meta’s surplus is mixed (−0.06 to +0.20).Source: E Per-Agent Data Behind Figure 4 · p. 13
  • Both frontier judges place OpenAI agents in their top three on satisfaction, whereas verifiable reward ranks GPT-5.4 configurations 7–9 and places Anthropic’s Opus-4.6 first on ground-truth task success.Source: E Per-Agent Data Behind Figure 4 · p. 13

F Per-Judge Ranking Validity Across Providers and Domains

  • All four judges recover the full 25-agent reward ordering, with Spearman ρ values from 0.84 to 0.94.The primary Opus-4.8 and GPT-5.5 judges have ρ=0.94 and ρ=0.84, respectively; Sonnet-4.5 and GPT-5.4 each have ρ=0.92 and ρ=0.94.Source: F Per-Judge Ranking Validity Across… · p. 13

F.1 The near-equal regime persists under frontier judges

The gate tracks reward across a broad capability range but has limited resolution for near-equal agents. The section also examines judge preference and stability, alternative signals, domain effects, and persona variation.

F.1 The near-equal regime persists under frontier judges

  • 31% of near-equal pairs are misranked by the Opus-4.8 gate, versus 0.9% of wide-reward pairs; the near-equal estimate has a 95% CI of [11.6, 50.0].The rate remains 32.0% after dropping the least-independent same-base-model temperature pairs, and is not sensitive to the exact reward-gap threshold.Source: §F.1 · pp. 14–15, Table 9 · p. 15
  • All four judges recover the full 25-agent ladder (ρ=0.84–0.94), but none recovers the near-equal top-11 ordering (ρ=0.25–0.51, all n.s.).This pattern holds for frontier judges whose verifiable pass rates match the strongest agents; the reported analysis attributes the limit to agent near-equality bounded by oracle resolution, not judge capability.Source: §F.1 · p. 14, Figure 7 · p. 15

F.2 Same-family self-preference

  • The Opus-4.8 judge rates Claude agents more favorably than GPT-5.5 does, with a larger difference for Claude than non-Claude agents (diff-in-diff +0.75/7).Ranking agreement remains ρ=0.92, so the reported self-preference affects absolute scores while ranking validity holds.Source: §F.2 · pp. 15–16, Figure 8 · p. 16

F.3 Judge stability

  • Re-scoring 48 transcripts yields judge self-reliability ICC=0.87 and variant-level rank stability ρ=0.92; five GPT-judge runs agree on agent ranking at mean ρ=0.74.Per-run validity against verifiable reward is noisier (ρ=0.24–0.59).Source: §F.3 · p. 16

G Cheap Signals for the Near-Equal Regime

  • Across 21 candidate signals, none beats the best single judge on near-equal pairs; averaging four judges leaves disagreement at 31.0% (27/87).Calibrated abstention reduces error to 14.8% on the 31–49% of near-equal pairs it ranks, buying selectivity rather than resolution.Source: G Cheap Signals for the Near-Equal… · p. 16

H Why Airline Ranking Is Weaker: Gate-Score Range Restriction

  • Airline gate scores span a narrower range than retail scores (2.03 vs 4.56 for Sonnet; 1.19 vs 3.56 for GPT-5.4), while reward spread is nearly identical (range ratio 1.17×).Both judges show the compression, which the passage identifies as the reason airline ranking validity is lower.Source: H Why Airline Ranking Is Weaker:… · p. 16, Figure 9 · p. 17

I Persona Strata

  • Six hand-authored persona overlays vary behavioral style while holding task facts fixed; two defined strata, S6 and S7, were not exercised in the reported runs.Personas cut mean verifiable reward 42% (0.47 →0.27) while agent ordering holds (ρ=0.83), and satisfaction varies sharply by persona without tracking reward.Source: I Persona Strata · pp. 16, 18, Table 10 · p. 17, Figure 10 · p. 17

J Dataset Statistics

The appendices describe the evaluation datasets, oracle patch, rating prompts, and illustrative transcripts. They report a parsing change that does not alter scoring logic and examples where favorable experience ratings coexist with task failure or an incorrect answer.

J Dataset Statistics

  • The evaluation includes 25 scored agent configurations, with conversations averaging 31 messages, 8.6 tool calls, and 7.6 user turns.The configurations come from 28 model-temperature cells, with three excluded for infrastructure reasons.Source: Table 11 · p. 18, J Dataset Statistics · p. 18

K Oracle Patch and Validation

  • The tolerant parser recovers wrapped JSON verdicts without rewriting parsed verdicts or adding pass conditions; a 16-case test checks valid, wrapped, and unparseable inputs.The τ²-bench reward multiplies task reward components; the patch changed the retail outcome in 113 of 2,537 applicable simulations (4.5%), while the airline oracle is fully deterministic.Source: K Oracle Patch and Validation · pp. 18–19

L Prompts and Rubrics

  • The release gate scores policy adherence, task resolution, efficiency, and professionalism, while proxy satisfaction and blind human ratings assess user-visible experience rather than task mechanics.The proxy sees only spoken turns; human annotators are blind to reward, variant, and model identity.Source: L Prompts and Rubrics · p. 19

M Representative Examples

  • In examples, τ² retail and airline conversations received satisfaction 7/7 despite reward 0.0, while a tutoring answer received a human rating of 10/10 despite being verifiably incorrect.In the retail example, a pending order was never modified; in the tutoring example, the shown work implies 330 passes, not the stated 165.Source: M Representative Examples · pp. 19–20
Loading 2609.12191v1…