PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning

Liang Ding

arXiv:2603.21362v3cs.AIcs.CL

TL;DR

Fixed LLM-judge rubrics apply the same dimensions across tasks, mis-evaluating goal-directed agent trajectories. AdaRubric generates task-adaptive rubrics, scores trajectories per step and dimension with confidence weighting, and synthesizes filtered preference data. Across benchmarks, it reports stronger human correlation, evaluator reliability, task-success gains from DPO pairs, and transfer to unseen domains.

  • Problem

    Fixed LLM-judge dimensions apply uniformly across tasks, producing systematic mis-evaluation when agent quality depends on task-specific criteria.

  • Method

    AdaRubric generates task-specific rubrics from task descriptions, evaluates trajectories per step and dimension with confidence weighting, and synthesizes filtered preference pairs.

  • Results

    Pearson r=0.79 (+0.15 over the best baseline) with α=0.83, while DPO preference pairs improve agent task success by up to +8.5% and generalise to unseen task types.

  • Takeaways & Limitations

    Structured rubric elicitation supports reliable, transferable agent evaluation and reward learning across task types, including zero-shot generalisation and modality-agnostic extension.

  • Takeaways & Limitations

    Rubric quality depends on LLM capability and task-description specificity, while adversarial descriptions degrade r by 0.04–0.06 and confidence scores may require recalibration out of distribution.

Abstract

from arXiv · show

Evaluating LLM agent trajectories is fundamentally task-specific: a code-debugging agent should be judged on Correctness and Error Handling, not on Fluency or Safety. Yet the dominant paradigm -- LLM-as-Judge with a fixed rubric -- applies the same static dimensions regardless of task, producing systematic mis-evaluation. We present AdaRubric, a framework that (i) adaptively generates task-specific evaluation rubrics from task descriptions via LLM, (ii) evaluates agent trajectories step-by-step with confidence-weighted, per-dimension scoring, and (iii) produces dense reward signals for preference learning. Three composable filtering strategies, including the novel DimensionAwareFilter that provably prevents dimension-level quality masking, yield high-quality DPO preference pairs. On WebArena, ToolBench, and AgentBench, AdaRubric achieves Pearson r = 0.79 human correlation (+0.15 over the strongest baseline), with strong reliability (Krippendorff's alpha = 0.83). DPO models trained on AdaRubric-generated pairs improve task success by +6.8-8.5% over the best baseline. AdaRubric also generalises zero-shot to unseen domains (SWE-bench) and extends to multimodal agents (VisualWebArena, OSWorld) without modification. Our code is available at: github.com/alphadl/AdaRubrics

1 Introduction

AdaRubric addresses systematic mis-evaluation from fixed LLM-judge rubrics by generating task-specific criteria and producing structured, dense trajectory rewards. Its design combines adaptive dimensions, confidence-weighted per-step scoring, reliability quantification, and filtering for preference learning.

  • Motivation: Fixed LLM-judge dimensions such as Helpfulness, Fluency, and Safety misalign with goal-directed tool-use tasks and can reward irrelevant style over task success.ToolBench API chaining instead requires dimensions such as API Selection Accuracy, Parameter Correctness, and Error Recovery.
  • AdaRubric: AdaRubric generates orthogonal, task-specific evaluation dimensions with calibrated 5-point criteria from each task description.The framework uses the LLM’s knowledge of task structure, success criteria, and domain conventions to align evaluation with human expert judgment.
  • AdaRubric: Confidence-weighted scoring evaluates trajectories per step and per dimension, producing dense rewards for post-training methods such as RL and DPO.This decomposes trajectory quality instead of relying on undifferentiated judgments.
  • AdaRubric: Krippendorff’s alpha supplies a principled deployment criterion for judging the reliability of LLM-based evaluators.The stated criterion is α ≥0.80.
  • AdaRubric: DimensionAwareFilter prevents per-dimension quality masking within the end-to-end evaluation-to-training pipeline.The filters are composable and support generation of preference pairs for training.

2 Related Work

Prior work provides fixed-dimension judges, task-specific success signals, scalar or step-level rewards, and agreement metrics, but AdaRubric combines adaptive evaluation with trajectory-level reward synthesis. Its pipeline organizes rubric generation, per-step and per-dimension evaluation, filtering, and DPO-pair construction.

  • LLM-as-Judge evaluation: Existing LLM-as-Judge methods use fixed criteria, task-specific training data, fine-grained skills, or large judgment corpora rather than adaptive rubrics for agent tasks.The cited methods include MT-Bench, G-Eval, Prometheus, FLASK, JudgeLM, and RewardBench.
  • LLM agent evaluation: Agent benchmarks provide task-specific success signals, but these signals are binary or coarse-grained, non-transferable across tasks, and lack per-step rewards.The related benchmarks include WebArena, ToolBench, AgentBench, and SWE-bench.
  • Reward signals and RLHF/DPO: RLHF and DPO use preference or scalar rewards, while process reward models assign step-level credit, motivating denser trajectory evaluation for agent post-training.The cited literature establishes the relevance of reward quality and process-level credit assignment.
  • Inter-rater reliability: Krippendorff’s alpha and Fleiss’ kappa provide established agreement measures that AdaRubric applies to quantify LLM evaluator consistency across runs.The framework uses reliability measurement as a deployment criterion.
  • AdaRubric pipeline: AdaRubric’s pipeline synthesizes task-adaptive rubrics, evaluates trajectories per step and dimension, then filters trajectories and generates DPO pairs.This positions the framework as an evaluation-and-reward synthesis pipeline rather than only a benchmark signal.

3 The ADARUBRIC Framework

ADARUBRIC turns task descriptions into structured, task-specific rubrics, evaluates trajectories per step and dimension, and synthesizes scores into quality-filtered preference pairs.

  • 3.1 Problem Formulation: ADARUBRIC defines tasks and trajectories explicitly, replacing scalar evaluation with structured, task-conditioned scores and confidence values.Tasks include instructions, domains, context, and expected tools or modalities, while trajectories contain thought, action, and observation steps.
  • 3.2 Stage 1: Adaptive Rubric Generation: The framework generates N orthogonal rubric dimensions with positive weights and five verbalized scoring levels derived from task success criteria.The generation prompt identifies critical criteria, clusters them into dimensions, assigns relative weights, and specifies concrete behaviors for each level.
  • 3.2 Stage 1: Adaptive Rubric Generation: Rubrics are validated for non-overlap, normalized weights, and complete scoring levels, with one retry and a domain template fallback for persistent failures.Rubrics can also be cached per task type, reducing API cost by more than 95% without loss in evaluation quality.
  • 3.3 Stage 2: Confidence-Weighted Evaluation: Step-level evaluation supplies scores and confidence values, which composable aggregation strategies combine using step importance, recency, or safety requirements.Weighted Mean is the default, Geometric Mean promotes balanced competency, and Min Score suits safety-critical tasks; confidence weighting is motivated as a lower-variance estimator under the stated model.
  • Filtering and preference construction: DimensionAwareFilter prevents scalar-score masking by requiring every rubric dimension to meet its threshold.A trajectory can pass an absolute threshold while one dimension is near zero; the dimension-aware criterion rejects this case by construction.
  • 3.5 Reward Synthesis and 3.6 Reliability Quantification: Filtered trajectories are sorted by global score to form DPO preference pairs, producing quality-assured preferred and informative dispreferred examples.The pipeline includes reliability quantification with Krippendorff’s alpha and recommends deployment when α ≥0.80.

4 Experiments

Across three benchmarks, AdaRubric’s adaptive evaluator correlates better with human rankings, produces more reliable judgments, and improves downstream DPO performance, including on SWE-bench.

  • 4.2 Main Results: Evaluation Quality: +0.15 Pearson r over GPT-4 Direct is attributed primarily to task-adaptive rubrics rather than additional test-time computation.The compute-matched GPT-4 CoT-Decomposed baseline reaches r=0.68, versus r=0.79 for ADARUBRIC-DA on WebArena.
  • 4.2 Main Results: Evaluation Quality: r=0.79/0.74/0.77 across benchmarks, outperforming GPT-4 Direct at r=0.64/0.60/0.62 in human-ranking correlation.ADARUBRIC-DA achieves these Pearson r values across the evaluated benchmarks.
  • 4.2 Main Results: Evaluation Quality: +8.5% is the largest DPO gain on ToolBench, while AgentBench improves by +7.7% and benefits extend across web, API, code, and OS tasks.The gains are reported for ADARUBRIC-DA using Qwen2.5-7B as the shared backbone.
  • 4.2 Main Results: Evaluation Quality: ADARUBRIC-DA achieves α > 0.82 on all benchmarks, exceeding the α ≥0.80 deployment criterion while G-Eval and Prometheus remain below it.G-Eval reaches α=0.63 and Prometheus α=0.70.
  • 4.5 Generalisation to SWE-bench: r=0.77 against the SWE-bench binary oracle, with a 14.7% DPO resolve rate, demonstrates zero-shot generalisation to code-repair tasks.The resolve rate improves by +4.9% over GPT-4 Direct.

5 Analysis

Ablations and extensions show that adaptive rubric generation is the central contributor, while the framework transfers across domains, modalities, and reinforcement-learning settings.

  • 5.1 Ablation Study: +0.14 r from fixed dimensions to domain templates and a further +0.07 from adaptive generation identify task-specific rubric design as the core contribution.Confidence weighting adds +0.03, DimensionAwareFilter adds +0.04, and the full pipeline gains +0.28 r over the generic fixed baseline.
  • 5.4 Cross-Domain Transfer: 31.2 TB and 24.6 WA cross-domain performance exceeds Prometheus in-domain scores of 21.4 and 21.0, indicating transfer across task families.The combined WA+TB→AB setting reaches 32.7%, approaching 34.1% in-domain performance.
  • 5.5 Multimodal Extension: r=0.76/0.73 on VisualWebArena and OSWorld, with a +5.8% DPO success-rate gain, extends AdaRubric to multimodal agents without multimodal-specific engineering.The rubric generator adds visual dimensions such as Visual Grounding and Screenshot Interpretation when task descriptions require them.
  • 5.6 Online RL: 30.2% success at 5K PPO steps, +6.6% over Prometheus, accompanies a +4.3% gap at 1K steps and indicates faster convergence with dense per-dimension rewards.The experiment uses ADARUBRIC-DA as the reward model for Qwen2.5-7B on WebArena.
  • 5.3 Rubric Quality: 4.3/4.2/4.1 human-study scores for relevance, orthogonality, and completeness place generated rubrics within 0.2–0.4 of expert-designed rubrics.Generic fixed rubrics score 1.6 on completeness, while edge-case coverage remains the main gap from expert rubrics.
  • 5.2 Sensitivity Analysis: λ=0.5 provides the best recency balance, whereas λ=2.0 overweights terminal steps and reduces correlation to r=0.72 versus r=0.79.The selected decay emphasizes later completion information without discarding early planning steps.

6 Discussion: Knowledge in LLM Evaluation

AdaRubric externalizes implicit evaluation knowledge by generating task-specific rubrics before judging trajectories, improving reliability and enabling transfer across domains and model scales.

  • r=0.79 versus 0.64 shows that generating an explicit rubric before evaluation outperforms direct evaluation.The two-step process decomposes complex judgment into manageable subtasks and produces more calibrated assessments.
  • 31.2% on ToolBench versus Prometheus’s 29.3% shows that WebArena-trained rubric evaluation transfers across domains.The result suggests rubric-guided evaluation captures generalizable quality concepts rather than only domain-specific heuristics.
  • Rubric-guided Llama-8B evaluation reaches r=0.68, exceeding unstructured GPT-4 evaluation at r=0.64.Structured knowledge scaffolding can partially compensate for model scale, supporting lower-compute evaluation deployments.
  • ADARUBRIC generates reusable task-type-level rubrics from parametric knowledge before training, complementing approaches that co-evolve instance-specific rubrics during reinforcement learning.This positions the framework as a complementary approach to preference-data quality and search-grounded rubric generation.

7 Conclusion

ADARUBRIC adaptively generates evaluation rubrics from task structure and criteria, producing reliable judgments, improved preference learning, and evidence of generalization. Its main constraints are dependence on task-description quality, computational cost, and limited breadth in some evaluations.

  • ADARUBRIC adaptively generates task-specific trajectory-evaluation rubrics by leveraging LLM parametric knowledge of task structures and evaluation criteria.
  • Pearson r=0.79, α=0.83, and up to +8.5% task-success improvement summarize ADARUBRIC’s reliability and preference-learning gains across five benchmarks.The framework also reports r=0.77 and a +4.9% DPO gain on unseen SWE-bench code repair, plus +6.6% PPO improvement at 5K steps.
  • Rubric quality depends on LLM capability and task-description specificity, with possible missing, overlapping, or specialized criteria and sensitivity to adversarial descriptions.In a pilot with 30 perturbed descriptions, correlation degraded by 0.04–0.06; confidence scores may also require recalibration for strongly out-of-distribution tasks.
  • ADARUBRIC costs about 40 evaluator calls per WebArena trajectory and 3–5× GPT-4 Direct wall-clock time, while multimodal, PPO, and SWE-bench results are treated as compact generalization evidence.

A Implementation Details

The default implementation uses five rubric dimensions and specified filtering parameters, while evaluation requires K×N calls per trajectory and incurs 3–5× GPT-4 Direct latency.

  • ADARUBRIC defaults to N=5 dimensions, λ=0.5, δmin=0.5, θj=2.5, and percentile filter p=80.
  • WebArena evaluation uses approximately 40 calls per trajectory, compared with one for GPT-4 Direct, and caching with batching reduces but does not eliminate the 3–5× latency cost.The count follows K≈8 and N=5, plus rubric generation per task type.

B Proof of Proposition 3.1

The proof characterizes confidence-weighted aggregation as a best linear unbiased estimator under an inverse-confidence Gaussian noise model, then tests the model empirically.

  • The construction sets one score to ϵ and adjusts the remaining scores so the weighted aggregate equals the target θ̄, generalizing to any θ′.
  • Under εk,j ∼ N(0, σ2/ck,j), confidence-weighted aggregation is BLUE by the Gauss–Markov theorem.
  • The equality condition in the aggregation bound requires all confidence values ck,j to be equal.
  • On 300 WebArena trajectory pairs with three evaluator runs, residual bias is at most 0.07, but excess kurtosis is approximately 1.4 and residual magnitude correlates negatively with confidence at −0.31.The inverse-confidence direction holds empirically, whereas Gaussian tails do not, so BLUE is only a motivating approximation.

C Calibration and Error Analysis

AdaRubric’s scores closely track human percentile rankings, while disagreement analysis identifies failure modes involving binary goals, implicit conventions, and ambiguous observations.

  • Calibration: Spearman ρ=0.98 (p<0.001) links AdaRubric score buckets to human percentile ranks, with rating-5 trajectories averaging the 91st human percentile.The near-linear relationship indicates that the 1–5 scale is meaningfully calibrated.
  • Error Analysis: Among 50 WebArena disagreements, 32% involved long-horizon binary goals where partial completion received excessive credit despite binary final outcomes.This failure mode concerns multi-hop tasks whose ultimate success condition is binary.
  • Error Analysis: Other disagreements involved implicit domain conventions (28%) and ambiguous observations (24%), including incorrect date formats, HTML artefacts, and API error codes.The authors report that richer task descriptions or observation preprocessing could address these cases.

D Additional Results

Additional experiments show that AdaRubric rewards improve WebArena PPO performance and that generated rubrics receive high expert ratings, alongside reported multimodal evaluation coverage.

  • PPO Training: 30.2% task success at 5K rollouts is achieved by ADARUBRIC-DA, exceeding ADARUBRIC-WM, Prometheus, and GPT-4 Scalar.The corresponding 5K results are 27.1%, 23.6%, and 21.3%, respectively.
  • Multimodal Results: The multimodal evaluation reports Pearson r and DPO SR% for VisualWebArena and OSWorld in Table 12.The supplied passages identify the table’s scope but do not provide its cell values.
  • Rubric Quality Human Study: ADARUBRIC-generated rubrics receive expert ratings of 4.3, 4.2, and 4.1, below expert-designed rubrics but above domain templates and generic rubrics.The three ratings correspond to the study’s reported rubric-quality dimensions; inter-annotator agreement is κ=0.79.
Loading 2603.21362v3…