PaperlayerView exact revision on arXiv

Source-linked AI summary

Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation

Darragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan

arXiv:2608.13564v1cs.AI

TL;DR

Reward-free language-model judges can mistake fluent failures for successful agent trajectories. RubricForge induces a human-readable judging rubric from ground-truth-labeled trajectories, then freezes and applies it without environment access. It roughly halves false passes while improving graded-outcome ranking, despite no significant binary-agreement gain.

  • Problem

    Reward-free agent judges must replace expensive or unavailable environment rewards, but existing hand-written or weight-tuned judges can credit fluent failures as successes.

  • Method

    RubricForge reflectively evolves a human-readable rubric against ground-truth-labeled trajectories with a frozen judge, then applies the frozen rubric to held-out trajectories without environment access.

  • Results

    RubricForge roughly halves tau-bench false-pass rates and ranks graded WebShop outcomes more faithfully, while not significantly improving binary agreement over G-Eval.

  • Takeaways & Limitations

    For reward-free evaluation, false-pass rate is more deployment-relevant than aggregate agreement because false passes can ship broken agents.

  • Takeaways & Limitations

    RubricForge does not significantly improve binary agreement and is marginally worse calibrated on absolute graded scores than a generic judge.

Abstract

from arXiv · show

Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time. Such a judge is a reward-free proxy whose value depends on whether it can be trusted, yet existing judges either hand-write the scoring rubric, as in G-Eval, or fine-tune the judge's weights, and both tend to credit fluent but unsuccessful trajectories as successes. We instead induce the text of an agent-judging rubric from a small set of ground-truth-labeled trajectories, grounding it in true outcomes. We present RubricForge, which evolves a judge rubric by reflective evolution against labeled trajectories to maximize agreement with the environment reward, freezes it, and applies it to held-out trajectories in one model call with no environment access. The optimized artifact is human-readable text, so every verdict is attributable to named criteria. Using one frozen 7B model as both agent and judge, on tau-bench (173 labeled trajectories drawn from 220 rollouts) and WebShop (160), the principal gain is faithfulness rather than raw agreement. The edge over a generic G-Eval judge is not statistically significant (McNemar p = 0.248), and absolute-score calibration marginally favors the generic judge (|err| difference -0.048, p = 2x10^-4). Yet RubricForge over-credits failed trajectories roughly half as often (0.115 vs. 0.173 false-pass rate on tau-bench, with three over-credit catches and zero reversals) and ranks graded WebShop outcomes more faithfully (Spearman 0.410 vs. 0.370). For a reward-free evaluator the false-pass rate, not aggregate agreement, is the deployment-relevant quantity, since a false pass ships a broken agent whereas a false fail merely costs a retry.

I. INTRODUCTION

RubricForge induces a human-readable judging rubric from ground-truth-labeled trajectories while keeping the judge frozen and using no environment access at evaluation time. Its main benefit is faithfulness: it reduces false passes and improves graded-outcome ranking, despite no significant binary-agreement advantage and slightly worse absolute calibration.

  • Method: RubricForge evolves a judge rubric against labeled trajectories with a frozen backbone, then scores held-out trajectories in one model call without environment access.The resulting criteria are concrete, human-readable, and grounded in trajectories.
  • Results: McNemar p = 0.248: RubricForge’s binary-agreement edge over generic G-EVAL is directional but not statistically significant.The paper distinguishes aggregate agreement from failures in the dangerous over-crediting direction.
  • Method: 0.654 → 0.769 on τ-BENCH and 0.667 → 0.750 on WEBSHOP: induction improves validation agreement.These gains are reported for the induced rubric relative to its pre-induction validation performance.
  • Results: 0.115 vs. 0.173 false-pass rate on τ-BENCH: RubricForge over-credits failed trajectories about half as often as a generic judge.It catches three fluent failures with zero reversals and never scores worse than G-EVAL on any item.
  • Results: Spearman 0.410 vs. 0.370 on graded WEBSHOP outcomes, while |err| difference −0.048, p = 2 × 10−4 favors generic-judge absolute calibration.The results separate ranking faithfulness from absolute-score calibration.

II. RELATED WORK · A. LLM-as-a-Judge and Automatic Evaluation

LLM-as-a-judge evaluation is a scalable alternative to human grading, but biased judges can over-credit fluent failures as successes. RubricForge addresses this false-pass risk by inducing an interpretable rubric from ground-truth labels while keeping the judge model frozen.

  • II. RELATED WORK: RUBRICFORGE differs from prior judges by inducing the rubric rather than hand-writing it or fine-tuning the judge’s weights.The literature is organized around LLM-as-a-judge evaluation, reward modeling and learned evaluators, agent benchmarks, prompt evolution, and proxy or metric misalignment.
  • A. LLM-as-a-Judge and Automatic Evaluation: LLM-as-a-judge evaluation grades one model’s outputs with another language model as a scalable alternative to human evaluation.G-EVAL uses a hand-written rubric and chain-of-thought form-filling, while MT-Bench and Chatbot Arena established pairwise judging as a benchmark substrate.
  • A. LLM-as-a-Judge and Automatic Evaluation: G-EVAL prompts a capable model with a hand-written rubric and chain-of-thought form-filling protocol to score natural-language generation.
  • A. LLM-as-a-Judge and Automatic Evaluation: MT-Bench and Chatbot Arena established LLM-and-human pairwise judging as a benchmark substrate for chat models.
  • A. LLM-as-a-Judge and Automatic Evaluation: Dedicated open evaluators such as Prometheus and Prometheus 2 represent a complementary line that fine-tunes models to apply evaluation criteria.
  • A. LLM-as-a-Judge and Automatic Evaluation: LLM evaluators are biased estimators swayed by answer position, verbosity, superficial fluency, and preference for their own generations.These biases can score a fluent-but-failed trajectory as a success, producing the false-pass error isolated by the over-crediting probe.
  • A. LLM-as-a-Judge and Automatic Evaluation: RUBRICFORGE leaves the judge model frozen and optimizes the rubric string against ground-truth labels.The resulting text artifact is cheaper than fine-tuning, more faithful than handwriting, and interpretable by construction.

B. Reward Modeling, RLHF, and Learned Evaluators · C. Agent Benchmarks and Evaluation · D. Prompt Optimization and Evolution

The paper situates learned evaluators within preference-based alignment, evaluates agents on interactive benchmarks with executable rewards, and frames RubricForge as reflective prompt evolution without gradient-based weight updates. Its approach builds on prompt-optimization and self-improvement methods while grounding reflection in external ground-truth labels.

  • B. Reward Modeling, RLHF, and Learned Evaluators: Reward models trained on human comparisons drive RLHF, while RLAIF, constitutional self-critique, and direct preference optimization provide related learned-evaluator paradigms.Self-rewarding and meta-rewarding schemes additionally let models generate training rewards or judge their own judgments.
  • B. Reward Modeling, RLHF, and Learned Evaluators: Learned evaluators therefore have a broader lineage spanning human-feedback rewards, AI feedback, constitutional critique, implicit reward modeling, and self-evaluation.These approaches connect reward-based alignment to increasingly automated sources of evaluation.
  • C. Agent Benchmarks and Evaluation: τ-bench evaluates tool-agent-user dialogues against hashed gold database states for a strict binary reward, whereas WEBSHOP measures graded product matching to instructions’ attributes and price.Broader benchmark suites include AgentBench across eight environments and WebArena for realistic web tasks.
  • D. Prompt Optimization and Evolution: RUBRICFORGE induces its judging rubric through reflective prompt evolution rather than gradient descent.The resulting optimization targets the rubric text instead of updating model weights.
  • D. Prompt Optimization and Evolution: Prompt-optimization precedents include APE, OPRO, PromptBreeder, DSPy, and GEPA, which uses natural-language reflection over execution traces to mutate prompts.GEPA has been shown to rival reinforcement learning at far lower sample cost.
  • D. Prompt Optimization and Evolution: Reflective prompt evolution belongs to a wider family of self-improvement methods involving bootstrapping, sampled reasoning paths, and revision of prior attempts.Unaided self-correction can fail or hurt when models grade themselves, motivating RubricForge’s grounding in external ground-truth labels.

E. Proxy/Metric Misalignment and Goodhart’s Law · III. PROBLEM FORMULATION

The paper frames reward-free judging as a proxy vulnerable to Goodhart-style drift and formalizes rubric induction against oracle-labeled trajectories. It distinguishes raw agreement from deployment-relevant faithfulness, especially false passes that certify broken agents.

  • E. Proxy/Metric Misalignment and Goodhart’s Law: Goodhart’s law predicts that optimized reward proxies can drift from the true objective, enabling policies to score highly while violating intended behavior.The paper connects this drift to reward over-optimization and reward hacking in machine learning.
  • III. PROBLEM FORMULATION: The formal setup represents each agent episode as a trajectory τ of actions and observations ending in a committed answer, with oracle reward r⋆(τ) binary or graded.The notation is summarized in Table I.
  • III. PROBLEM FORMULATION: The judge receives only the observable rendering x(τ), excluding oracle rewards and gold world-state, and returns a deterministic pass verdict, graded score, and one-sentence rationale.This prevents label leakage into the judge’s input.
  • III. PROBLEM FORMULATION: Rubric induction maximizes agreement on labeled training trajectories while selecting on held-out validation data to avoid overfitting.The candidate space consists of rubric strings reachable by the evolution operator.
  • III. PROBLEM FORMULATION: The judge backbone θ remains frozen while optimization changes only rubric text ρ, which is then applied once to each held-out trajectory.This makes the optimized artifact a text rubric rather than altered judge weights.
  • III. PROBLEM FORMULATION: A false pass on a truly failed trajectory silently certifies a broken agent, making false-pass rate the primary faithfulness criterion rather than raw agreement.Raw agreement, κ, AUC, Spearman ranking, and absolute calibration remain complementary diagnostics.
  • III. PROBLEM FORMULATION: The central empirical question is whether grounding the rubric in labels reduces false passes and makes the proxy harder to fool, even without improving raw agreement.The formulation explicitly separates the two error directions and treats false-pass reduction as the deployment-relevant test.

IV. METHOD: RUBRICFORGE

RubricForge induces a judging rubric through reflective evolution on labeled trajectories, freezes it, and applies it to held-out trajectories as a one-call reward-free metric.

  • Pipeline: RubricForge has three stages: induce a rubric by reflective evolution against labeled trajectories, freeze it, and apply it to held-out trajectories.The held-out evaluation uses one model call and no environment reward access.

A. Stage 1: Rubric Induction

RubricForge evolves a single text rubric for a frozen judge by scoring labeled trajectories and optimizing binary agreement with gold outcomes. Reflection focuses on mis-scored cases, while validation selects the best candidate for freezing and reward-free application.

  • Rubric Induction: RubricForge evolves only the rubric string ρ, the judge’s system prompt, while reusing GEPA’s reflective prompt evolution unchanged.Its novelty lies in the evolved component, judging-pass rollout, and fitness objective.
  • Rubric Induction: Binary agreement with the gold label is the per-trajectory fitness and exact optimization objective.The frozen backbone scores each training trajectory using the candidate rubric.
  • Reflective Evolution: Reflection presents the current rubric, mis-scored trajectories, and their true labels to produce a revised rubric that corrects failures and generalizes.Mis-scored cases are those where the judge’s verdict disagrees with the true label.
  • Selection and Freezing: Candidates are pooled, and the rubric maximizing validation agreement is retained for freezing.The procedure evaluates the seed rubric, accepts revised candidates when validation agreement improves, and selects the pool maximum.
  • Selection and Freezing: The frozen rubric is applied as a reward-free judge without environment access.Stage 3 maps each trajectory to the frozen judge’s verdict using ρ⋆.

B. Stage 2: Freeze · C. Stage 3: Reward-Free Application

RubricForge freezes the selected rubric into a fixed, reproducible judging instrument, then applies it to held-out trajectories through one deterministic, reward-free model call. The application separates evolved criteria from output formatting and conservatively assigns failure when network or parsing errors occur.

  • B. Stage 2: Freeze: The selected rubric ρ⋆ is frozen verbatim after search.If search does not beat its seed, the generic default is recorded as the effective rubric.
  • B. Stage 2: Freeze: In both benchmarks, search improved over the seed and accepted evolved rubrics into pools of 6 for τ-bench and 4 for WEBSHOP.
  • B. Stage 2: Freeze: Freezing makes the metric fixed and reproducible, so the same trajectory always receives the same verdict.
  • C. Stage 3: Reward-Free Application: At test time, frozen ρ⋆ is the judge’s system prompt and the rendered held-out trajectory x(τ) is the user message.
  • C. Stage 3: Reward-Free Application: A single deterministic call returns each verdict under a fixed JSON contract containing pass, score, and reason.The contract is kept separate from the evolved rubric so optimization shapes criteria rather than formatting.
  • C. Stage 3: Reward-Free Application: The system extracts and decodes the first brace-delimited JSON object from the model output.
  • C. Stage 3: Reward-Free Application: Any network or parse failure conservatively falls back to ˆp = 0.

V. EXPERIMENTAL SETUP · VI. RESULTS

The study evaluates reward-free judging on τ-BENCH and WEBSHOP using task-stratified labeled trajectories and compares RubricForge with generic, example-augmented, heuristic, and majority baselines. Reflective induction improves validation agreement, while test decomposition shows fewer false passes without additional false fails.

  • V. EXPERIMENTAL SETUP: τ-BENCH covers retail and airline customer service with a strict binary database-state-hash reward.
  • V. EXPERIMENTAL SETUP: The experiment uses a class-stratified 85/26/62 train/validation/test split with 173 cells and holds out 47 surplus cells.All cells from each task remain in one split, preventing task leakage.
  • V. EXPERIMENTAL SETUP: WEBSHOP contributes 160 trajectories, including 116 with graded attribute-match scores.
  • V. EXPERIMENTAL SETUP: The comparison includes G-EVAL, FewShot with k = 4 labeled examples, a surface-feature Heuristic, and a train-majority Majority baseline.G-EVAL uses the generic rubric that seeds induction, isolating the value of grounding.
  • V. EXPERIMENTAL SETUP: Judge comparisons use paired significance tests: McNemar’s exact test for binary agreement and paired bootstrap absolute error for graded scores.Directional but nonsignificant gaps are labeled ns, with confidence intervals from 104-resample nonparametric bootstrap.
  • VI. RESULTS: Reflective evolution raises validation agreement from 0.654 → 0.769 on τ-BENCH and from 0.667 → 0.750 on WEBSHOP.The τ-BENCH pool contains 6 rubrics and 208 judging calls; the WEBSHOP pool contains 4 rubrics and 156 calls.
  • VI. RESULTS: RubricForge and G-EVAL each obtain 2 true positives and 8 false fails, so induction adds no extra retries.
  • VI. RESULTS: False passes decrease from 9 for G-EVAL to 6 for RubricForge, producing the entire difference between the judges.The confusion-count decomposition shows the near-tie is asymmetric.

VII. ANALYSIS: FAITHFULNESS AND OVER-CREDITING · A. The Over-Crediting Probe · B. Case Study: The Three Caught Trajectories

RubricForge’s main advantage is faithfulness: it reduces false passes on τ-BENCH without increasing false fails, catching three cases that G-EVAL over-credits. In those cases, its named criteria identify the specific trajectory evidence that justifies failure.

  • VII. ANALYSIS: FAITHFULNESS AND OVER-CREDITING: The faithfulness probe focuses on false-pass rates, case-level attribution, ranking versus calibration, and stratification by difficulty, fluency, and length.These analyses ask what distinguishes RubricForge when binary agreement does not.
  • A. The Over-Crediting Probe: 0.115 versus 0.173: RubricForge over-credits 6 of 52 τ-BENCH failures, compared with 9 for G-EVAL.This is roughly half as often on truly failed trajectories.
  • A. The Over-Crediting Probe: Three τ-BENCH disagreements are all G-EVAL false passes that RubricForge correctly fails, with zero reversals.No item has RubricForge over-crediting while G-EVAL does not.
  • A. The Over-Crediting Probe: 8 false fails: both judges make the same number, while RubricForge cuts false passes from 9 to 6.The three caught cases move from the over-credit cell to the true-fail cell, leaving false fails unchanged.
  • B. Case Study: The Three Caught Trajectories: Each of the three caught τ-BENCH trajectories is fluent, prompting exhaustive case inspection.The case study examines why G-EVAL passes them while RubricForge fails them.
  • B. Case Study: The Three Caught Trajectories: 0.85–0.90: G-EVAL passes one airline trajectory with high confidence, whereas RubricForge fails it using a named criterion.In airline|RSEA|2, Request Fulfillment identifies that the agent narrated savings from downgrades but never committed or confirmed the update.
  • B. Case Study: The Three Caught Trajectories: RubricForge reads trajectory evidence rather than rewarding a confident final message, and reports which named criterion the trajectory violated.This is the interpretability benefit of inducing the rubric text.

C. Per-Criterion Leave-One-Out Ablation · D. Ranking versus Calibration

The ablation identifies Request Fulfillment as the criterion chiefly responsible for resisting over-crediting, while other criteria are redundant or leave metrics unchanged. On graded WebShop, RubricForge ranks outcomes better than G-Eval, but G-Eval is marginally better calibrated in absolute error, so score quality depends on use case.

  • C. Per-Criterion Leave-One-Out Ablation: Request Fulfillment is the only ablation that clearly degrades faithfulness, reducing accuracy 0.774 →0.758 and κ +0.092 →+0.068.It demands explicit user confirmation that the requested action was completed.
  • C. Per-Criterion Leave-One-Out Ablation: Three τ-BENCH disagreements were all G-Eval=PASS over-credits, while RubricForge=FAIL was correct in every case.Each disagreement involved an induced criterion that fired.
  • C. Per-Criterion Leave-One-Out Ablation: 0.115 →0.135 false-pass rate follows removal of Request Fulfillment, showing this criterion most resists over-crediting.The criterion requires explicit confirmation that the requested action was completed.
  • C. Per-Criterion Leave-One-Out Ablation: Removing Tool Usage is mildly redundant: accuracy nudges to 0.790, κ to +0.118, and false-pass to 0.096.Its tool-error signal is partly subsumed by the other criteria.
  • C. Per-Criterion Leave-One-Out Ablation: Accuracy, κ, and false-pass remain exactly unchanged when Authentication, Order Verification, or User Communication is removed.The result holds on the τ-BENCH test set.
  • D. Ranking versus Calibration: Spearman 0.410 versus G-Eval’s 0.370 shows RubricForge ranks graded WebShop outcomes more faithfully.Kendall correlation likewise favors RubricForge, 0.347 versus 0.314.
  • D. Ranking versus Calibration: Mean |ŝ −r⋆| 0.288 versus 0.336 favors G-Eval in absolute calibration, with paired-bootstrap ∆= −0.048, p = 2×10−4.Ranking faithfulness and absolute calibration are distinct quantities.
  • D. Ranking versus Calibration: Agent comparison should use ranking, where RubricForge wins, whereas an absolute score regressor should not rely on it.The distinction follows from the separate ranking and calibration results.

E. Stratification: Where the Advantage Lives · VIII. DISCUSSION · IX. CONCLUSION

RubricForge’s over-crediting advantage is concentrated in difficult, fluent airline trajectories rather than uniform across settings. Its deployment value lies in reducing false passes and improving faithfulness, despite no significant binary-agreement advantage and marginally worse absolute-score calibration.

  • E. Stratification: Where the Advantage Lives: The false-pass advantage concentrates in the harder airline domain rather than appearing uniformly across τ-BENCH strata.The airline false-pass gap is +0.19: G-EVAL 0.438 versus RUBRICFORGE 0.250.
  • E. Stratification: Where the Advantage Lives: Within the airline domain, the largest gap occurs for the fluent self-evolving airline|RSEA policy.Its false-pass gap is +0.25: G-EVAL 0.375 versus RUBRICFORGE 0.125.
  • VIII. DISCUSSION: For reward-free evaluation, false-pass rate is the deployment-relevant metric rather than aggregate agreement.False passes can ship broken agents, whereas false fails primarily impose retry costs.
  • VIII. DISCUSSION: Aggregate agreement is an inadequate headline metric when labels are imbalanced and evaluator errors have asymmetric deployment costs.The discussion contrasts dangerous false passes with inefficient false fails.
  • IX. CONCLUSION: RubricForge induces a human-readable evaluation rubric from ground-truth-labeled trajectories, freezes it, and applies it without environment access.Evaluation uses one frozen-model call as a reward-free metric.
  • IX. CONCLUSION: McNemar p = 0.248: the induced rubric does not significantly out-agree a generic LLM judge on the hard binary success label.The conclusion characterizes this result as deliberately honest and contrarian, while noting marginally worse absolute graded-score calibration.

APPENDIX A INDUCED RUBRICS, SWEEP, AND HYPERPARAMETERS

RubricForge’s appendix specifies deterministic induction settings, validation sweeps, and frozen rubrics for τ-BENCH and WEBSHOP. The selected rubric improves validation agreement over the generic G-Eval seed, while WEBSHOP shows a ranking advantage but worse absolute calibration.

  • Induction sweep and hyperparameters: 200 rollout calls on τ-BENCH and 150 on WEBSHOP produced accepted rubric pools with best validation agreements of 20/26 = 0.769 and 18/24 = 0.750, respectively.The runs used 208 and 156 judging calls, accepting pools of 6 and 4 rubrics.
  • Induction sweep and hyperparameters: Deterministic judge calls used temperature 0 and a maximum of 220 tokens, with GEPA reflective evolution using minibatch 4 and seed 0.The generic G-Eval-style rubric served as the seed candidate, so accepted rubrics had to beat a competent judge.
  • Induction sweep and hyperparameters: +11.5 and +8.3 validation points were the selected rubric’s improvements over the generic seed on τ-BENCH and WEBSHOP, respectively.The selected rubric improved from 0.654 to 0.769 on τ-BENCH and from 0.667 to 0.750 on WEBSHOP.
  • WEBSHOP graded-test summary: WEBSHOP’s graded-test summary reports that RubricForge wins ranking, while G-Eval wins absolute calibration with ∆|err| = −0.048 and p = 2×10−4.The comparison covers n = 56 graded test cases and gives a 95% CI of [−0.083, −0.018].
  • Frozen rubrics: The frozen WEBSHOP rubric checks query alignment, product-attribute matching, price-limit compliance, and completion of Buy Now with matching final options.Failure is indicated if any criterion is unmet.
Loading 2608.13564v1…