PaperlayerView exact revision on arXiv

Source-linked AI summary

Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation

Darragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan

arXiv:2608.13564v1cs.AI

TL;DR

Agent evaluation often lacks affordable environment rewards, while existing language-model judges use hand-written rubrics or fine-tuned weights. RubricForge evolves rubric text from labeled trajectories; it reduces false passes and ranks graded outcomes more faithfully, without significantly improving binary agreement.

  • Problem

    When environment rewards are costly or unavailable, it remains an open question whether agent-judging rubrics should be hand-written or induced from ground-truth-labeled trajectories.

    Source: §I · p. 1

  • Method

    RubricForge reflectively evolves rubric text against ground-truth-labeled trajectories, then freezes it for held-out judging with one model call and no environment access.

    Source: §I · p. 1

  • Results

    0.173→0.115 false-pass rate on τ-BENCH; RubricForge also ranks graded WEBSHOP outcomes more faithfully (Spearman 0.410 vs. 0.370).

    Source: §I · p. 2

  • Takeaways & Limitations

    For reward-free evaluation, false-pass behavior is more decision-relevant than aggregate agreement.

    Source: §I · p. 2

    Binary agreement did not significantly improve over a generic G-EVAL judge (McNemar p = 0.248), and absolute-score calibration marginally favored the generic judge (|err| difference −0.048, p = 2 × 10−4).

    Source: §I · p. 2

Abstract

from arXiv · show

Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time. Such a judge is a reward-free proxy whose value depends on whether it can be trusted, yet existing judges either hand-write the scoring rubric, as in G-Eval, or fine-tune the judge's weights, and both tend to credit fluent but unsuccessful trajectories as successes. We instead induce the text of an agent-judging rubric from a small set of ground-truth-labeled trajectories, grounding it in true outcomes. We present RubricForge, which evolves a judge rubric by reflective evolution against labeled trajectories to maximize agreement with the environment reward, freezes it, and applies it to held-out trajectories in one model call with no environment access. The optimized artifact is human-readable text, so every verdict is attributable to named criteria. Using one frozen 7B model as both agent and judge, on tau-bench (173 labeled trajectories drawn from 220 rollouts) and WebShop (160), the principal gain is faithfulness rather than raw agreement. The edge over a generic G-Eval judge is not statistically significant (McNemar p = 0.248), and absolute-score calibration marginally favors the generic judge (|err| difference -0.048, p = 2x10^-4). Yet RubricForge over-credits failed trajectories roughly half as often (0.115 vs. 0.173 false-pass rate on tau-bench, with three over-credit catches and zero reversals) and ranks graded WebShop outcomes more faithfully (Spearman 0.410 vs. 0.370). For a reward-free evaluator the false-pass rate, not aggregate agreement, is the deployment-relevant quantity, since a false pass ships a broken agent whereas a false fail merely costs a retry.

I. INTRODUCTION

RubricForge induces an auditable judge rubric from labeled trajectories while keeping the judge frozen, then applies it to held-out trajectories without environment access. Its clearest reported benefit is fewer false passes, while aggregate agreement does not significantly improve.

I. INTRODUCTION

  • Environment rewards provide ground truth but may be unavailable or unsafe to query during development or deployment, motivating LLM judges as reward-free proxies [8], [9], [10].The paper contrasts rubric induction with hand-authored G-EVAL criteria [9] and fine-tuning judge weights [13], [14], [15].Source: §I · p. 1
  • False passes are more dangerous than false fails: they can certify a broken agent and distort reported success rates or policy selection, whereas false fails cost retries or discarded samples.The paper argues that deployment evaluation should therefore emphasize the asymmetric false-pass risk rather than aggregate agreement.Source: §I · p. 1
  • RubricForge evolves rubric text against ground-truth-labeled trajectories with a frozen judge, then scores held-out, by-task-disjoint trajectories in one call without environment access.The resulting human-readable criteria make verdicts attributable and inspectable; validation agreement rises from 0.654 to 0.769 on τ-BENCH and 0.667 to 0.750 on WEBSHOP.Source: §I · pp. 1–2
  • 0.115 vs. 0.173 false-pass rate on τ-BENCH shows RUBRICFORGE over-credits failed trajectories about half as often as the generic judge.The three-trajectory margin consisted of over-credit catches in RUBRICFORGE’s favor, with zero reversals.Source: Figure 1 · p. 2, §I · p. 2
  • RubricForge grounds prompt optimization in labeled outcomes to resist drift from true reward and packages false-pass, ranking-versus-calibration, attribution, and stratification probes into a reusable protocol.The paper connects false-pass risk to proxy-objective misalignment and Goodhart’s law.Source: §I · p. 2
  • McNemar p = 0.248: RUBRICFORGE’s binary-agreement edge over generic G-EVAL is directional but not statistically significant.On graded WEBSHOP outcomes, Spearman is 0.410 vs. 0.370, while absolute calibration marginally favors the generic judge (|err| difference −0.048, p = 2 × 10−4).Source: §I · p. 2, Figure 1 · p. 2

II. RELATED WORK · A. LLM-as-a-Judge and Automatic Evaluation

Prior work establishes LLM judging as a scalable evaluation approach while documenting biases that can produce false passes. RubricForge differs by evolving rubric text against ground-truth labels while keeping the judge frozen.

II. RELATED WORK

  • The related work spans LLM judging, reward modeling and learned evaluators, agent benchmarks, prompt evolution, and proxy misalignment and Goodhart’s law.Its stated differentiator is evolving rubric text rather than hand-writing it or fine-tuning judge weights.Source: §II · p. 3

A. LLM-as-a-Judge and Automatic Evaluation

B. Reward Modeling, RLHF, and Learned Evaluators · C. Agent Benchmarks and Evaluation · D. Prompt Optimization and Evolution

These sections position RUBRICFORGE as a trajectory-level, reward-free evaluator grounded in executable benchmark labels. It adapts reflective prompt evolution to induce an auditable rubric rather than introducing a new search algorithm.

B. Reward Modeling, RLHF, and Learned Evaluators

  • Unlike preference-trained response reward models used by optimizers, RUBRICFORGE scores whole trajectories against programmatic outcomes as a frozen, human-auditable, reward-free instrument.Related approaches include human-comparison reward models for RLHF [26], [27], RLAIF [28], [29], DPO [30], self- and meta-rewarding [31], [32], self-feedback, [34], and RewardBench.Source: B. Reward Modeling, RLHF, and Learned… · p. 3

C. Agent Benchmarks and Evaluation

  • τ-bench uses strict binary database-state matching [5], whereas WEBSHOP grades product match to instruction attributes and price [6].These executable outcomes provide the ground-truth labels used to induce RUBRICFORGE’s judge rubric.Source: C. Agent Benchmarks and Evaluation · p. 3
  • Broader executable-reward suites include AgentBench [36], WebArena [37], GAIA [38], and SWE-bench [7]; RUBRICFORGE predicts their oracle from trajectories when rerunning it is costly or unsafe.The trajectories can come from varied agent policies,, [39]–[45] and retrieval-augmented agents [46]–[49], widening the space of fluent but unfaithful traces a reward-free judge must screen.Source: C. Agent Benchmarks and Evaluation · p. 3

D. Prompt Optimization and Evolution

  • Prompt-search precedents include APE [50], OPRO, PromptBreeder [52], and DSPy [53],; RUBRICFORGE specifically builds on GEPA’s reflective prompt mutation [55].Its mutation target is the judge rubric, its rollout is a judging pass, and its fitness is agreement with ground-truth labels; it reuses the optimizer unchanged.Source: D. Prompt Optimization and Evolution · pp. 3–4
  • RUBRICFORGE grounds reflection in external outcome labels because self-grading can fail or hurt, unlike self-improvement methods that rely on model confidence [56]–.The passage distinguishes its use of external ground-truth labels from unaided self-correction.Source: D. Prompt Optimization and Evolution · p. 4
  • Rubric induction is test-time adaptation that spends extra compute before deployment to fit a rubric rather than update model weights [60]–[67].The passage relates this to test-time scaling and classical meta-learning and adaptation from limited supervision.Source: D. Prompt Optimization and Evolution · p. 4

E. Proxy/Metric Misalignment and Goodhart’s Law · III. PROBLEM FORMULATION

A reward-free judge is a proxy that can drift toward rewarding fluent form over actual outcomes, while RubricForge grounds rubric induction in labeled trajectories. The formulation freezes the judge and treats false passes as the primary faithfulness concern because they can silently certify a broken agent.

E. Proxy/Metric Misalignment and Goodhart’s Law

  • Goodhart’s law describes how an optimized proxy can drift from its true objective, including reward hacking that scores highly while violating intent.The paper operationalizes this drift for judges as over-crediting failed trajectories, measured by false-pass rate.Source: E. Proxy/Metric Misalignment and… · p. 4
  • Grounding rubric induction in per-trajectory ground-truth labels is presented as the contrast to an ungrounded judge that rewards fluency over outcomes.The passage analogizes this grounding to ToolTree’s per-instance checkable feedback and few-shot preference alignment, and contrasts it with ungrounded prompt-space optimization.Source: E. Proxy/Metric Misalignment and… · p. 4

III. PROBLEM FORMULATION

  • A reward-free evaluator maps visible trajectory text to predictions without querying the environment oracle at scoring time.The oracle supplies binary or graded reward after an episode, but scoring-time access is prohibited because it can be expensive, slow, or destructive.Source: §III · p. 4
  • The judge makes one deterministic call that returns a binary pass verdict, a graded score, and a one-sentence rationale.Its input includes only interaction-visible actions, results, replies, counters, and the final answer, not oracle reward or gold world-state.Source: §III · p. 4
  • Rubric induction maximizes agreement on labeled training trajectories, with selection on a held-out validation pool to avoid overfitting.The frozen judge backbone remains unchanged; optimization moves only the rubric text, which is then applied once per held-out trajectory.Source: §III · pp. 4–5
  • False-pass rate is the primary faithfulness criterion because a false pass can certify a broken agent, while a false fail costs only a retry.Raw agreement, κ, AUC, ranking, and absolute calibration are complementary diagnostics.Source: §III · p. 5

IV. METHOD: RUBRICFORGE · A. Stage 1: Rubric Induction

RubricForge induces a judging rubric by reflective evolution against labeled trajectories, freezes the best validation-agreement rubric, and applies it to held-out trajectories as a one-call reward-free metric.

IV. METHOD: RUBRICFORGE

  • RubricForge has three stages: reflective rubric induction on labeled trajectories, freezing the selected rubric, and one-call reward-free judging of held-out trajectories.The deployed judge has no environment access.Source: §IV · p. 5, A. Stage 1: Rubric Induction · p. 5

A. Stage 1: Rubric Induction

  • RubricForge reuses GEPA’s reflective prompt evolution unchanged, evolving only the rubric string ρ; a candidate “rollout” is a judging pass by the frozen backbone.The judging pass scores each training trajectory with the candidate rubric.Source: A. Stage 1: Rubric Induction · p. 5
  • Each trajectory’s fitness is binary agreement with its gold label, making agreement the optimizer’s objective.This is the per-trajectory fitness used during induction.Source: A. Stage 1: Rubric Induction · p. 5
  • After minibatch scoring, reflection uses mis-scored training trajectories and their true labels to revise the rubric toward observable symptoms of success rather than reward mentions or memorized labels.The reflecting model sees labels only for the training cases being revised; the deployed judge never sees a label.Source: A. Stage 1: Rubric Induction · p. 5
  • The search starts from the same competent generic G-Eval-style rubric used by the baseline, so accepted rubrics must improve on a real judge rather than a weak starting point.Candidates are pooled and evaluated by validation agreement.Source: A. Stage 1: Rubric Induction · p. 5
  • The retained rubric is the pooled candidate with the highest validation agreement.The selected rubric is frozen before reward-free application to held-out trajectories.Source: A. Stage 1: Rubric Induction · p. 5

B. Stage 2: Freeze

Stage 2 freezes the selected rubric verbatim, making it a fixed, reproducible instrument for judging trajectories. In these runs, the search improved over the seed on both benchmarks.

B. Stage 2: Freeze

  • RubricForge freezes the selected rubric verbatim, or records the generic default if the search never beats its seed.The search improved over the seed on both benchmarks in these runs.Source: B. Stage 2: Freeze · p. 5
  • Freezing makes the metric reproducible: the same trajectory always receives the same verdict.Source: B. Stage 2: Freeze · p. 5
  • The search improved over the seed on both benchmarks, accepting evolved rubrics into pools of 6 on τ-bench and 4 on WEBSHOP.Source: B. Stage 2: Freeze · p. 5

C. Stage 3: Reward-Free Application · V. EXPERIMENTAL SETUP · VI. RESULTS

RubricForge applies a frozen, text-based rubric to held-out trajectories without environment access, using one deterministic judge call. Results show a nonsignificant binary-agreement edge over G-EVAL but fewer false passes, while induction improves validation agreement and yields named, trajectory-grounded criteria.

C. Stage 3: Reward-Free Application

  • At test time, the frozen rubric and held-out trajectory are supplied in one deterministic call, with no environment access and a strict JSON verdict contract.The rubric remains separate from output formatting, and network or parse failures conservatively default to failure; named criteria can attribute verdicts to specific evidence.Source: C. Stage 3: Reward-Free Application · p. 5

V. EXPERIMENTAL SETUP

  • The evaluation used 173 τ-BENCH trajectories from 220 labeled rollouts and 160 WEBSHOP trajectories, including 116 graded scores.Trajectories came from ReAct and RSEA policies; τ-BENCH used a binary database-state-hash reward, while WEBSHOP used graded attribute-match rewards.Source: §V · pp. 5–6
  • A single frozen Qwen2.5-7B served as agent and judge at temperature 0, with trajectories split by task to prevent test-task leakage.The τ-BENCH split was 85 train, 26 validation, and 62 test trajectories; oracle rewards and gold world-state were withheld from the judge-readable trajectory text.Source: §V · p. 6

VI. RESULTS

  • Induction raised validation agreement from 0.654 to 0.769 on τ-BENCH and from 0.667 to 0.750 on WEBSHOP, yielding five named criteria.The τ-BENCH criteria cover authentication, order verification, confirmed request fulfillment, error-free tool use, and clear user communication, each grounded in observable trajectory evidence.Source: §VI · p. 7
  • RUBRICFORGE had 0.774 accuracy and +0.092 Cohen κ versus G-EVAL’s 0.726 and +0.026, but their McNemar comparison was nonsignificant (p = 0.248).The results characterize the binary-agreement edge as directional, not a significant win.Source: §VI · p. 7, §V · p. 7
  • Majority achieved 0.839 accuracy by predicting failure for every trajectory, but its κ and F1 were both 0.Its McNemar comparison with RUBRICFORGE was nonsignificant (p = 0.289).Source: §VI · p. 7
  • FewShot scored 0.613 accuracy and κ = −0.019, while the Heuristic collapsed to near-zero accuracy; both were worse than RUBRICFORGE.FewShot’s four examples increased its predicted-positive rate to 0.32 and biased it toward over-passing fluent traces; its McNemar p-value was 0.009.Source: §VI · p. 7
  • RUBRICFORGE reduced false passes from 9 to 6 while matching G-EVAL’s 2 true positives and 8 false fails on 62 τ-BENCH test verdicts.Its predicted-positive rate was 0.129 versus G-EVAL’s 0.177; FewShot had 17 false passes and a predicted-positive rate of 0.323.Source: §VI · p. 7

VII. ANALYSIS: FAITHFULNESS AND OVER-CREDITING · A. The Over-Crediting Probe

RubricForge’s advantage over G-Eval is concentrated in false passes: it over-credits fewer failed τ-BENCH trajectories, while the WEBSHOP gap is narrower. On τ-BENCH, the judges’ three disagreements are all G-EVAL false passes corrected by RubricForge, with no reversals.

VII. ANALYSIS: FAITHFULNESS AND OVER-CREDITING

  • The section probes why binary agreement may not distinguish RubricForge from a generic judge by examining false-pass rates, criterion attribution, ranking versus calibration, and stratification.The named stratifications are difficulty, fluency, and length.Source: §VII · p. 7

A. The Over-Crediting Probe

  • On τ-BENCH, RubricForge’s false-pass rate is 0.115 (6 of 52 failures) versus G-EVAL’s 0.173 (9 of 52).On WEBSHOP, the rates are 0.468 (22 of 47 failures) versus 0.489 (23 of 47), respectively.Source: A. The Over-Crediting Probe · p. 7
  • On τ-BENCH, all three judge disagreements are G-EVAL false passes that RubricForge correctly fails, with zero reversals.These three catches account for the entire margin between the judges in Table III; RubricForge is never worse than G-EVAL on an individual item.Source: A. The Over-Crediting Probe · p. 7
  • On WEBSHOP, RubricForge catches 2 over-credits and reverses 1, alongside false-pass rates of 0.468 versus 0.489 for G-EVAL.The gap is narrower than on τ-BENCH but in the same direction.Source: A. The Over-Crediting Probe · p. 7
  • In the τ-BENCH error decomposition, false-fail counts are identical at 8 each, while false passes fall from 9 for G-EVAL to 6 for RubricForge.The comparison stacks each judge’s 62 test verdicts by error direction.Source: A. The Over-Crediting Probe · p. 7
  • The three caught τ-BENCH cases move from the false-pass cell to the true-fail cell, leaving the false-fail cell untouched.The confusion-grid comparison shows the only changing cell is the dangerous over-credit cell, and it changes in the safe direction.Source: A. The Over-Crediting Probe · p. 7

B. Case Study: The Three Caught Trajectories · C. Per-Criterion Leave-One-Out Ablation

On τ-BENCH, RUBRICFORGE catches all three G-EVAL over-credits without reversals, and the leave-one-out ablation identifies Request Fulfillment as the clearest faithfulness driver. The named criteria explain the caught failures and make the rubric’s judgments interpretable.

B. Case Study: The Three Caught Trajectories

  • All three τ-BENCH judge disagreements were G-EVAL false passes that RUBRICFORGE caught, with zero reversals.The caught cases move from the false-positive cell to true negatives; both judges make the same eight false fails.Source: Figure 3 · p. 8, Figure 4 · p. 8, Table V · p. 9
  • The three caught airline failures trigger Request Fulfillment, Order Verification, or Successful Authentication criteria.Examples include an uncommitted downgrade, an unverified certificate, and a failed booking reported as successful.Source: B. Case Study: The Three Caught… · p. 8, Table V · p. 9
  • RUBRICFORGE identifies which named criterion a trajectory violates, rather than merely disagreeing with G-EVAL.In the cited cases, G-EVAL rewards confident final messages while RUBRICFORGE checks trajectory evidence against rubric criteria.Source: B. Case Study: The Three Caught… · p. 8

C. Per-Criterion Leave-One-Out Ablation

D. Ranking versus Calibration · E. Stratification: Where the Advantage Lives

On graded WebShop, RubricForge ranks outcomes more faithfully, while G-Eval has lower absolute error. Its false-pass advantage on τ-BENCH concentrates in the harder airline domain and fluent RSEA traces, not the longest trajectories.

D. Ranking versus Calibration

  • Spearman 0.410 (Kendall 0.347) for RubricForge versus 0.370 (Kendall 0.314) for G-Eval on graded WebShop, favoring RubricForge in ranking.Figure 6 reports n = 56; ranking measures whether higher judge scores track higher true rewards.Source: Figure 6 · p. 9, D. Ranking versus Calibration · p. 9
  • Mean |ŝ − r⋆| is 0.288 for G-Eval versus 0.336 for RubricForge; paired-bootstrap Δ = −0.048 favors G-Eval (p = 2×10−4).RubricForge’s decisive {0.2, 1.0}-style scores sharpen ranking but inflate error against smooth graded rewards.Source: Figure 6 · p. 9, D. Ranking versus Calibration · p. 9

E. Stratification: Where the Advantage Lives

  • +0.19 is the τ-BENCH false-pass gap in airline (G-Eval 0.438 vs. RubricForge 0.250), rising to +0.25 for fluent self-evolving airline|RSEA traces (0.375 vs. 0.125).The advantage concentrates in the harder airline domain and fluent RSEA policy, whose confident narration fools a generic judge.Source: Table VI · p. 9, E. Stratification: Where the Advantage… · p. 9
  • 0.00 is the false-pass gap in the longest-trajectory tertile; the gap is largest in shorter, surface-fluent traces.The paper attributes the faithfulness edge to surface fluency × task difficulty rather than trajectory length.Source: Table VI · p. 9, E. Stratification: Where the Advantage… · p. 9

D. Ranking versus Calibration

  • Ranking and absolute calibration measure different qualities: RubricForge suits agent comparison, while G-Eval is marginally better for absolute score regression.The paper recommends judging a reward-free evaluator used to compare or select agents by ranking, where RubricForge wins.Source: D. Ranking versus Calibration · p. 9

E. Stratification: Where the Advantage Lives

VIII. DISCUSSION · IX. CONCLUSION

RUBRICFORGE’s main advantage is faithfulness rather than aggregate agreement: it over-credits failed trajectories less often and ranks graded outcomes better, despite a calibration tradeoff. The discussion frames false-pass rate as the deployment-relevant metric and recommends a broader protocol for evaluating reward-free judges.

VIII. DISCUSSION

  • False-pass rate is the deployment-relevant metric because a false pass can ship a broken agent, whereas a false fail mainly costs a retry.The discussion argues aggregate agreement is the wrong headline for a reward-free evaluator, even though RUBRICFORGE does not significantly win that comparison.Source: §VIII · p. 10
  • Grounding rubric induction in labeled outcomes pressures the judge to distinguish fluent-looking failures using observable evidence, resisting drift toward fluency.The discussion presents this as the grounded counterpart to an ungrounded prompt-space meta-objective collapsing across users.Source: §VIII · p. 10, §IX · p. 10
  • The proposed reward-free evaluation protocol reports an oracle-reward ceiling, false-pass rate, ranking versus calibration, per-criterion attribution, and fluency/difficulty stratification.The authors recommend it for LLM judges of agent trajectories independently of RUBRICFORGE.Source: §VIII · p. 10, §IX · p. 10
  • Induction helps most on hard tasks with fluent, confident agents and least on easy retail, where over-crediting is already rare.The 7B judge is near its binary-agreement ceiling; τ-BENCH’s ∼16% positive class widens F1 and κ variance, and the WEBSHOP calibration loss is real. Renderer faithfulness is a load-bearing assumption that deployments must re-audit for their trajectory format.Source: §VIII · p. 10

IX. CONCLUSION

  • RUBRICFORGE evolves rubric text against ground-truth-labeled trajectories, freezes it, and applies it as a reward-free metric in one model call without environment access.The human-readable artifact attributes verdicts to named criteria.Source: §IX · p. 10
  • On τ-BENCH, RUBRICFORGE’s false-pass rate was 0.115 vs. 0.173, and its WEBSHOP Spearman was 0.410 vs. 0.370.The binary-agreement edge was not significant (McNemar p = 0.248), absolute graded-score calibration was marginally worse, and the τ-BENCH false-pass comparison had three catches and zero reversals.Source: §IX · p. 10
  • Evidence is limited to one frozen Qwen2.5-7B judge and two benchmarks, τ-BENCH and WEBSHOP; transfer and broader validation remain open.The method needs a modest pool of labeled trajectories; without labels it reduces to its generic seed. RUBRICFORGE ranks graded outcomes better but is marginally worse calibrated, so the authors recommend it for comparison rather than absolute-score regression.Source: §IX · p. 10

APPENDIX A INDUCED RUBRICS, SWEEP, AND HYPERPARAMETERS

RubricForge evolves rubrics against labeled trajectories, then freezes them for judging; in both domains, the selected rubric improves validation agreement over the generic G-Eval seed. The appendix specifies the induced rubrics and deterministic evaluation setup.

Induction hyperparameters

  • Induction used GEPA reflective evolution with minibatch 4 and seed 0, budgeting 200 τ-BENCH and 150 WEBSHOP rollout calls.The runs used 208 and 156 judging calls, respectively, and accepted pools of 6 and 4 rubrics; judging was deterministic at temperature 0 with a 220-token maximum.Source: Appendix A · p. 10

Induction sweep

  • 0.769 vs. 0.654 on τ-BENCH and 0.750 vs. 0.667 on WEBSHOP: the selected rubric raises validation agreement over the generic G-Eval seed.The selected rubric improves validation agreement by 11.5 and 8.3 points, respectively; accepted children were generally at or above the seed.Source: Figure 7 · p. 11, Appendix A · p. 11

Graded-test summary

  • WEBSHOP graded-test results favor RubricForge for ranking, while G-Eval wins absolute calibration (paired-bootstrap Δ|err| = −0.048, 95% CI [−0.083, −0.018], p = 2×10−4).The summary covers n = 56.Source: Table VIII · p. 11

Frozen rubrics

  • The frozen τ-BENCH rubric evaluates authentication, order verification, request fulfillment, tool usage, and user communication.It requires successful authentication and order retrieval, policy-compliant fulfillment with explicit confirmation, appropriate tool calls, and clear, relevant communication.Source: Appendix A · p. 11
  • The frozen WEBSHOP rubric checks query alignment, product-attribute match, price-limit compliance, and completion of Buy Now with final options matching selections.Failure is indicated if any criterion is unmet.Source: Appendix A · p. 11

Reproducibility

  • Verdicts are deterministic given the frozen backbone, rubric, and fixed by-task split (seed 0); analyses use persisted held-out per-item predictions.The persisted predictions support agreement, McNemar tests, bootstrap confidence intervals, over-crediting, leave-one-out ablation, calibration, and stratification analyses.Source: Appendix A · p. 11

Related papers

Loading 2608.13564v1…