PaperlayerView exact revision on arXiv

Source-linked AI summary

ComplexConstraints and Beyond: Expert Rubrics for RLVR

Sushant Mehta, Liudas Panavas, Suhaas Garre, Edwin Chen

arXiv:2606.09118v3cs.AI

TL;DR

Benchmarks can miss complex, contextual instruction following and agent workflows when they focus on narrow surface constraints. This paper studies expert-curated rubrics as evaluation criteria and RL rewards across instruction-following and agentic settings, finding training gains that transfer to external benchmarks.

  • Problem

    Programmatic benchmarks cover narrow surface constraints, while real-world instruction following and agentic workflows require judging complex, contextual behavior.

    Source: §1 · p. 1

  • Method

    The paper develops expert-curated rubric design practices and applies rubric-based rewards to training across instruction-following and agentic tasks.

    Source: §1 · pp. 1–2, §5.3 · p. 6

  • Results

    Rubric-reward training transfers across instruction-following and agentic benchmarks, including a +8.4 pp overall improvement on AdvancedIF after training on approximately 1,000 expert-curated examples.

    Source: §5.5 · p. 7, §5.6 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    Expert rubrics can serve both as evaluation instruments and as effective training signals for instruction following and agentic behavior.

    Source: §7 · p. 9

    The comparison with RIFL is suggestive rather than controlled because the base models and training pipelines differ.

    Source: §5.6 · p. 8

Abstract

from arXiv · show

Evaluation protocols can lag behind LLM capabilities. Programmatically verified benchmarks cover narrow surface constraints, whereas real-world instruction following and agentic workflows require judging semantic, contextual, and policy-dependent behavior. We study expert-curated rubric-based evaluation as a unified mechanism for measurement and reinforcement-learning rewards across two settings: complex instruction following and enterprise agentic tasks. We identify rubric-design choices that affect reward quality, including maximum viable atomicity, intent-aware criterion design, and LLM-judge calibration. We introduce ComplexConstraints, an expert-curated instruction-following suite comprising a public 75-prompt benchmark with 1,559 rubric criteria and a disjoint 1,000-prompt training set, with 10-40 atomic criteria per prompt. Empirically, rubric rewards improve training in both fixed task datasets, such as ComplexConstraints, and stateful RL environments, such as CoreCraft. Training a 4B model on ComplexConstraints improves mean criterion pass rate by +15.5 pp on a held-out split, bringing it within 0.5 pp of the untrained baseline of a roughly 60x larger Qwen3 model, and the gains transfer to external benchmarks the model never saw during training: +8.4 pp on AdvancedIF and +10.1 pp on MultiChallenge. In CoreCraft, rubric-reward RL likewise transfers to out-of-distribution benchmarks (+4.5 pp BFCL, +7.4 pp tau^2-Bench, +6.8 pp Toolathlon). These results show that expert-authored rubrics provide effective evaluation targets and scalable reward signals for improving LLM instruction following and agentic behavior.

1 Introduction

Traditional benchmarks can lose the ability to distinguish model capabilities and may measure surface constraints rather than the complex, contextual behavior real-world use requires. This paper studies expert-curated rubrics as a shared approach to evaluation and reinforcement-learning rewards, introducing COMPLEXCONSTRAINTS and reporting transferable training improvements.

1 Introduction

  • Real-world use requires complex instruction following and reliable multi-step workflows, while saturated benchmarks and narrow programmatic checks may not measure these capabilities.IFEval-style checks can reward incoherent responses that satisfy surface constraints.Source: §1 · p. 1
  • Expert-curated rubrics decompose task success into atomic, verifiable criteria that capture explicit requirements and pragmatic user intent.The paper examines this approach in complex instruction following and enterprise agentic task execution.Source: §1 · p. 1
  • COMPLEXCONSTRAINTS provides a public 75-prompt benchmark with 1,559 criteria and a disjoint 1,000-prompt training set, with 10–40 atomic criteria per prompt.The training set was built through the same expert-curation pipeline.Source: §1 · p. 2
  • Rubric-based RL training on modest amounts of data produces substantial, transferable improvements in instruction following and agentic task execution.The paper presents the rubrics as both evaluation instruments and effective training signals.Source: §1 · p. 2

2 Background and Related Work

Prior work expands evaluation from programmatic constraints toward richer rubrics and realistic agent tasks, while research also explores rubrics as reinforcement-learning rewards. COMPLEXCONSTRAINTS combines expert-authored, dense criteria with an explicit dual purpose: evaluation and RL training.

From Programmatic to Rubric-Based Evaluation

  • Instruction-following evaluation progressed from programmatic checks to richer approaches, including multi-turn instance-level rubrics and expert-curated criteria.AdvancedIF found that frontier models achieve approximately 75% against its richer criteria; COMPLEXCONSTRAINTS uses rubrics for training as well as evaluation.Source: §2 · p. 2
  • The paper frames richer evaluation criteria as a second source of value: rubrics expressive enough to evaluate frontier models can also train them.Source: §2 · p. 2

LLM-as-a-Judge

  • LLM-judge reliability remains a concern, motivating clear criteria that reduce evaluator ambiguity.Prior work reports limitations in single-shot evaluations and notes that clear criteria matter for judge reliability.Source: §2 · p. 2

Agentic Evaluation and Rubrics as Training Signals

  • Agent benchmarks cover function calling, customer service, and diverse long-horizon tasks, while prior RL methods have applied rubric or fine-grained rewards to training.Examples include BFCL, τ-bench, τ 2-Bench, Toolathlon, RIFL, VerIF, RLCF, and ToolRL.Source: §2 · pp. 2–3

Concurrent Rubric-Based Work

  • COMPLEXCONSTRAINTS distinguishes itself with expert-authored, high-density instruction-following rubrics designed to serve as both evaluation instruments and RL reward signals.Its design is complementary to the medical focus of HealthBench and the evaluation-only orientation of AdvancedIF’s benchmark component.Source: §2 · p. 3

3 What Makes a Rubric Trainable

Rubrics become useful RL rewards when criteria capture meaningful task success, user intent, and judge-verifiable distinctions. The section identifies design practices for reducing reward noise and producing informative gradients.

3 What Makes a Rubric Trainable

  • Naive rubric rewards can grant credit to confidently wrong answers, reward surface phrasing over intent, or turn ambiguous criteria into judge noise.The section introduces its design decisions as responses to these predictable failure modes.Source: §3 · p. 3

3.1 Maximum Viable Atomicity

  • Criteria should target the smallest meaningful unit rather than literal maximal atomicity, which can degrade evaluation quality and direct RL gradients toward superficially similar wrong answers.Source: §3.1 · p. 3

3.2 Intent over Literalism

  • Authors restate the user’s intent—the “what” and “why”—before writing criteria, so rubric rewards favor contextual understanding over surface phrasing.The passage illustrates this with a request to improve Spanish while reading an economics article in Spanish.Source: §3.2 · p. 3

3.3 Calibrating the Judge

  • Each criterion is checked against a reference response and adversarially edited to verify that the LLM verifier changes its verdict when correctness flips.Ambiguity surfaced in an “avoid alliteration” criterion, which was clarified to specify poetic alliteration.Source: §3.3 · p. 3
  • Calibration matters in training because noisy rewards can be exploited by RL optimizers, making criterion calibration a defense against reward hacking.Source: §3.3 · p. 3

3.4 Optimal Difficulty for RL

  • Difficulty is tuned task by task toward the informative band, since tasks all models pass or all models fail provide no RL gradient.Rubric authors evaluate frontier responses during construction to tune task difficulty.Source: §3.4 · pp. 3–4
  • Per-criterion scoring creates a continuous reward gradient between partial and full task success, even when satisfying every criterion remains rare.A response satisfying 28 of 30 criteria receives more reward than one satisfying 15.Source: §3.4 · p. 4

4 The COMPLEXCONSTRAINTS Dataset

COMPLEXCONSTRAINTS is an expert-curated instruction-following suite built for both frontier-model evaluation and RL training. Its prompts combine layered constraints with dense criteria, and its public benchmark remains challenging even for strong models.

4 The COMPLEXCONSTRAINTS Dataset

  • The suite has two disjoint sets: a public 75-prompt benchmark with 1,559 criteria and a 1,000-prompt training set, with 10–40 expert-authored criteria per prompt.Criteria are labeled as explicit or implicit and objective or subjective.Source: §4 · p. 4
  • Prompts model authentic professional tasks with dependent, conditional, planning, or implicit constraints; the training set spans seven categories led by business writing and scheduling.Business writing accounts for 39% and scheduling 22% of the training set.Source: §4 · p. 4
  • Domain experts authored prompts and reference responses, wrote rubrics, and used independent expert review with senior adjudication when needed.Difficulty was calibrated through a pilot evaluation against a frontier model, and criteria undergo LLM-judge calibration before use.Source: §4 · p. 4
  • Even the strongest frontier model passes barely two-fifths of public benchmark tasks outright, while dense criteria provide partial-credit gradients for RL.The passage attributes this difficulty profile to realistic prompt complexity rather than artificial engineering.Source: §4 · p. 4

5 Expert Rubrics as RL Training Signals: Empirical Evidence

Expert-rubric RL improves instruction following and agentic task execution, while dense criteria expose specific capability gaps and provide localized training feedback. Gains transfer to external benchmarks, and rubric analysis organizes agent failures into distinct capability levels.

5.1 Targeting Frontier Capability Gaps for Training

  • On COMPLEXCONSTRAINTS, the strongest frontier model passes only 40.4% of tasks, while the strongest public CoreCraft result solves 30.80%; dense criteria also show which requirements fail.Because all criteria must be satisfied for a task pass, rubric-level scores provide a gradient between partial and complete success for RL.Source: §5.1 · pp. 4–5, Table 1 · p. 5, Table 2 · p. 5

5.2 Grounding Rubrics in Agentic Failures

  • Analysis of twelve models across 150 CoreCraft tasks groups recurring failures into five levels: tool use, planning, adaptability, groundedness, and common-sense reasoning.The levels span correctly invoking tools, completing plans, recovering from unexpected results, staying grounded in context, and reasoning beyond explicit instructions.Source: §5.2 · pp. 5–6

5.3 Training Methodology: Rubrics as RL Rewards

  • RLVR uses an LLM judge to score required, bonus-only, and penalty-only criteria, combining their satisfaction scores into a reward that supports partial credit.For the instruction-following setup, all criteria are required; paired-rollout inspection found no systematic response-quality degradation alongside reward improvement.Source: §5.3 · p. 6

5.4 Results: Generalization on Instruction Following

  • +15.5 pp in mean per-criterion pass rate for Qwen3-4B after training on approximately 900 expert-curated examples, bringing it within 0.5 pp of the untrained Qwen3-235B-A22B-Instruct baseline.The held-out split reports 73.4% for the trained 4B model versus 73.9% for the roughly 60× larger baseline; the training result is from a single seed.Source: §5.4 · pp. 6–7, Table 3 · p. 7
  • +8.4 pp on AdvancedIF overall and +10.1 pp on MultiChallenge overall after training exclusively on single-turn tasks, with the largest gains on multi-turn dimensions.MultiChallenge instruction retention improves by +22.1 pp, AdvancedIF system steerability by +12.4 pp, and MultiChallenge self-coherence by +10.5 pp; the transfer mechanism remains a hypothesis.Source: Table 4 · p. 7, §5.4 · p. 7

5.5 Results: Generalization on Agentic Tasks

  • +11.4 pp on held-out CoreCraft tasks after one epoch of rubric-based RL, with transfer gains of +4.5 pp on BFCL Parallel function calling, +7.4 pp on τ 2-Bench Retail customer service, and +6.8 pp on Toolathlon long-horizon tool use.Pass3 reliability on CoreCraft nearly doubles from 9.3% to 17.6%.Source: Table 5 · p. 7, §5.5 · p. 7
  • Qualitative trajectory analysis identifies improvements in multi-step workflow execution, constraint handling, and professional response quality.The reported competencies include task decomposition and sequencing, temporal filtering and policy application, and structured communication.Source: §5.5 · p. 8

5.6 Discussion: Why Expert Rubrics Are Effective Training Signals

  • Rubric effectiveness is attributed to localized credit assignment, task difficulty calibration, and data efficiency from expert curation, with gains also transferring across formats and domains.The AdvancedIF comparison with RIFL is suggestive rather than controlled because the base models and training pipelines differ.Source: §5.6 · p. 8

6 Broader Discussion

The discussion argues that expert rubrics improve construct validity across content, predictive, and discriminant dimensions, while recognizing the cost of expert authoring. It also points to hybrid rubric creation and expert-supervised verifiers as possible ways to improve automated rubric quality.

Construct Validity

  • Rubrics extend evaluation to semantic correctness and pragmatic appropriateness, and rubric performance is associated with gains on out-of-distribution benchmarks.The discussion contrasts this with IFEval-style tuning, which can overfit to a small set of verifiable constraints and fail to generalize to unseen ones.Source: §6 · p. 8
  • ComplexConstraints frontier-model pass rates span 0% to 40.4%, exposing capability differences that saturated benchmarks obscure.The discussion frames this spread as discriminant validity: meaningful differences remain visible rather than clustering within a few percentage points.Source: §6 · p. 9

Infrastructure Investment

  • Expert rubric creation costs more, while hybrid expert-seed and synthetic scaling approaches may offer a middle ground.RIFL’s finetuned verifier reaches 0.728 F1 agreement with humans, versus 0.515 for a vanilla LLM judge (He et al., 2025).Source: §6 · p. 9

7 Conclusion

The conclusion presents expert-curated rubrics as improving both LLM measurement and training. It summarizes the ComplexConstraints suite and reports transfer from rubric-based RL across instruction-following and agentic settings, while identifying open questions about transfer mechanisms and reward shaping.

7 Conclusion

  • Four rubric-design decisions—maximum viable atomicity, intent over literalism, adversarial judge calibration, and difficulty/density tuning—guide rubrics as reward signals.The conclusion positions expert-curated rubric evaluation as an approach to improving both measurement and training of large language models.Source: §7 · p. 9
  • ComplexConstraints is an instruction-following suite comprising a public 75-prompt benchmark and a 1,000-prompt training set, with 10–40 atomic criteria per prompt.Source: §7 · p. 9
  • Rubric-based RL yields in-distribution gains and transfers to out-of-distribution benchmarks in both instruction-following and agentic settings.Open questions include isolating which design decisions drive transfer and studying asymmetric reward shaping more broadly.Source: §7 · p. 9
Loading 2606.09118v3…