PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks

Tianze Xu, Yanzhao Zheng, Pengrui Lu, Lyumanshan Ye, Yong Wu, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu

arXiv:2604.02795v1cs.CLcs.AI

TL;DR

Response-level rubric rewards can be sparse or ambiguous, limiting credit assignment in instruction following. RTT predicts which tokens relate to each constraint and jointly optimizes token-level and response-level advantages; experiments report gains across instruction- and rubric-level accuracy.

  • Problem

    Existing rubric-based RL methods predominantly use response-level rewards, which can be sparse or ambiguous and provide coarse credit assignment.

    Source: §1 · p. 1, §4.1 · p. 4

  • Method

    RTT uses a Token-Level Relevance Discriminator to produce constraint-specific token rewards and RTT-GRPO to jointly optimize token-level and response-level advantages.

    Source: §1 · p. 2

  • Results

    RTT consistently outperforms all baselines in instruction-level and rubric-level accuracy across benchmarks.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The authors conclude that RTT addresses limitations of response-level rewards in instruction-following tasks.

    Source: §7 · p. 10

    Inter-Sample Token Group Normalization can be length-biased when response lengths differ because longer responses contribute more tokens to the joint normalization statistics.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models.

1 Introduction

Response-level rubric rewards can be sparse or ambiguous, limiting useful credit assignment. RTT assigns constraint-related rewards to tokens and combines token-level and response-level advantages; experiments report better benchmark performance while preserving general capacities.

1 Introduction

  • Rubric-based RL uses structured constraints to capture multiple quality dimensions and is presented as an alternative to RLVR for open-ended tasks (Huang et al., 2025; Lambert et al., 2024; Guo et al., 2025).Source: §1 · p. 1
  • Binary rewards can leave weak initial models without meaningful gradients, while aggregated rewards can give identical scores to responses satisfying different constraint subsets.The first creates reward sparsity; the second creates reward ambiguity and coarse credit assignment.Source: §1 · p. 1
  • RTT trains a Token-Level Relevance Discriminator to identify tokens responsible for each constraint’s satisfaction or violation, producing token-level reward signals.The framework bridges response-level rubric scores and token-level credit assignment.Source: §1 · pp. 1–2
  • RTT-GRPO jointly optimizes response-level and token-level advantages, while Intra-sample Token Group Normalization addresses normalization boundaries in three-dimensional reward space.The identified Group Partitioning Problem concerns how to define boundaries across reward dimensions.Source: §1 · p. 2
  • RTT consistently outperforms all baselines across instruction-following benchmarks while preserving general capacities.The introduction reports this broad result without giving quantitative values.Source: §1 · p. 2

2 Related Work

Related work covers instruction-following alignment, token-level credit assignment, and rubric-based reinforcement learning. These lines of work address open-ended or multi-constraint tasks, while rubric methods commonly retain scalar response-level rewards.

2.1 Instruction Following Methods

2.2 Token-level Credit Assignment

2.3 Rubric-based Reinforcement Learning

  • Rubric-based RL uses structured rubrics and LLM judges for open-ended tasks, but existing approaches mainly aggregate constraint satisfaction into scalar response rewards, introducing sparsity and ambiguity (Zhou et al., 2025; Zhang et al., 2025).Source: §2.3 · p. 3

3 Preliminaries and Backgrounds

Section 3 defines constraint-based evaluation and explains why response-level GRPO normalization becomes difficult when token-level rewards span responses, token positions, and constraints.

3.1 Instruction Following

  • Hard constraints use deterministic rule-based verification, while soft constraints are evaluated with an LLM-as-a-Judge.The instruction’s constraint set includes both rule-verifiable requirements and requirements needing nuanced semantic evaluation.Source: §3.1 · p. 3
  • All-or-Nothing requires every constraint to be satisfied, while Constraint Satisfaction Rate measures the percentage satisfied; as rewards, they risk sparsity and ambiguity, respectively.AON and CSR provide complementary instruction-following measures but have these distinct reward-signal challenges.Source: §3.1 · p. 3

3.2 GRPO and Group Partitioning Problem

  • Standard GRPO samples G responses to one instruction and applies each response’s normalized scalar advantage uniformly to its tokens.Advantages are normalized along the sample axis only.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • Token-level rubric rewards span sample, token, and constraint dimensions, creating a Group Partitioning Problem; the paper identifies this as a new high-dimensional setting for study.The authors state that prior work studied group size and intra-group sampling, not group partitioning in high-dimensional reward spaces.Source: §3.2 · p. 4
  • Inter-sample token normalization jointly normalizes tokens across responses for each constraint, but unequal response lengths let longer responses dominate its statistics and suppress shorter-response signals.The approach averages per-constraint advantages into a final token-level advantage.Source: §3.2 · p. 4
  • Intra-sample Token Group Normalization instead normalizes within each response, eliminating length-induced bias and decoupling token-level from response-level advantages.The paper proposes it to address the inter-sample strategy’s length bias.Source: §3.2 · p. 4

4 Rubrics To Tokens

RTT projects rubric scores onto token-level relevance signals and optimizes with RTT-GRPO using response- and token-level advantages. Intra-sample normalization distributes token rewards within each response.

4.1 Token-Level Relevance Discriminator

  • The Token-Level Relevance Discriminator takes the prompt, constraint, and response as input and predicts a relevance score for each response token.The score estimates each token’s relevance to the specified constraint.Source: §4.1 · p. 4, §4.2 · p. 5

4.2 Rubric-to-Token RL Training

  • Training data draw prompts and constraints from Tulu-3 and HiR-16K; negative responses come from minimal edits or omitted constraints, with relevant segments labeled by a strong LLM.The annotated segments are mapped to tokens.Source: §4.2 · p. 5
  • A token-level classification head predicts relevance probabilities from final hidden states, and the discriminator is trained with binary cross-entropy on valid tokens and relevance labels.Valid indices exclude prompt tokens and padding.Source: §4.2 · p. 5

4.2.1 Rubric-to-Token GRPO

  • RTT-GRPO adapts GRPO for dense token rewards, sampling grouped outputs and optimizing a clipped objective that combines response-level and token-level advantages.Response-level advantages use AON or CSR rewards; token-level advantages redistribute constraint signals within a response.Source: §4.2.1 · p. 5

4.2.2 Intra-sample Token Group Normalization

  • For each constraint, RTT multiplies the response-level score by predicted token relevance, then standardizes the resulting rewards across the tokens of that response.The normalization group is the full set of tokens in one response.Source: §4.2.2 · p. 6
  • Within-response normalization amplifies advantages for relevant tokens and neutralizes uniformly relevant or irrelevant tokens; advantages are averaged across constraints.Source: §4.2.2 · p. 6

5 Experimental Settings

The experiments use a multi-source instruction-following training set, evaluate both in-domain and out-of-domain benchmarks, and compare RTT with supervised, preference-optimization, and response-reward RL baselines.

5.1 Datasets

  • HiR-16K aggregates MulDimIF, VerIF, IFTrain, and Chatbot Arena to provide broad instruction-following coverage.Source: §5.1 · p. 6

5.2 Benchmarks

  • Evaluation covers IFEval, IFBench, MulDimIF, and AdvancedIF in-domain, plus MATH-500, GPQA, and MMLU-Pro out-of-domain.Source: §5.2 · p. 6

5.3 Models and Configurations

  • Experiments use Qwen3-4B-Instruct, Qwen2.5-7B-Instruct, and Llama3.2-3B-Instruct policies, with Qwen3-4B-Base- and Llama3.2-3B-Base-initialized discriminators and RTT-CSR/RTT-AON variants.For soft constraints, DeepSeek-V3 serves as the judge LLM.Source: §5.3 · p. 6

5.4 Baselines and Evaluation Metrics

  • Baselines are SFT, DPO, RL-AON, and RL-CSR; evaluation reports instruction-level accuracy and rubric-level accuracy for benchmarks other than MulDimIF.Table 1 presents main instruction-following benchmark results, with fine-tuned method scores and deltas relative to the base model.Source: §5.4 · p. 6, Table 1 · p. 7

6 Results

RTT leads the reported instruction-following benchmarks, generally preserves out-of-domain ability, and is examined through ablations, training analyses, and token-attribution cases.

6.1 Main Results

  • 2.50% average gain in instruction-level accuracy and 1.64% in rubric-level accuracy over strongest RL baselines RL-AON and RL-CSR; RTT-CSR and RTT-AON lead both metrics across benchmarks.RTT also generally preserves or improves out-of-domain performance despite training solely on instruction-following data.Source: §6.1 · p. 7, Table 2 · p. 7

6.2.1 A1: Group Partitioning Strategies

  • Intra-sample Token Group Normalization generally performs best across models, with per-response normalization eliminating length-induced bias in advantage estimation.Table 3 compares inter-sample and intra-sample normalization across model configurations.Source: §6.2.1 · p. 7, Table 3 · p. 8

6.2.2 A2: Discriminator Weight Strategy

  • The full-annotation discriminator strategy provides more reliable token-level supervision than Random, Uniform, or Greedy token weighting.The comparison includes four token-weighting strategies for RL training.Source: §6.2.2 · p. 8, Table 4 · p. 8

6.2.3 A3: Sensitivity to the Token-Level Weight (β)

  • β = 0.5 gives the best performance; higher β values gradually reduce performance as the intra-sample token advantage increasingly dominates the joint advantage.The sensitivity study fixes α = 1 and evaluates RTT-AON and RL-AON on Qwen3-4B-Instruct across IFEval, IFBench, and MulDimIF.Source: §6.2.3 · p. 8, Figure 3 · p. 8

6.3.1 Training Dynamics

  • RTT-CSR improves steadily across IFEval, IFBench, and MulDimIF while maintaining lower response entropy than RL-CSR throughout training.Figure 4 compares benchmark performance on the left and response entropy on the right.Source: §6.3.1 · p. 9, Figure 4 · p. 9

6.3.2 Training Stability on Llama3.2-3B-Instruct

  • On Llama3.2-3B-Instruct, RL-CSR shows sharp accuracy drops and rapidly rising entropy, KL loss, and clip fraction, while RTT-CSR remains stable.RL-CSR also performs worse than RL-AON on IFEval and MulDimIF.Source: §6.3.2 · p. 9, Figure 5 · p. 9

6.3.3 Computational Overhead

  • 117.9 GPU hours, or 7.8% of total compute, go to the discriminator forward pass; this adds 8.5% overhead over the RL baseline.Rollout generation dominates at 76.8%; the discriminator needs no gradient computation or optimizer-state storage.Source: Table 5 · p. 9, §6.3.3 · pp. 9–10

6.3.4 Token Attribution Case Studies

  • The discriminator assigns high relevance to compliant-content tokens in satisfying responses and to violation-causing tokens in violating responses, distinguishing responses with the same CSR score.This localizes token-level reward signals instead of treating equal response-level scores as identical token-level cases.Source: §6.3.4 · p. 10

7 Conclusion

RTT maps response-level rubric scores to token-level rewards and combines them with response-level signals in RTT-GRPO, using intra-sample token group normalization. The paper reports gains over baselines in instruction- and rubric-level accuracy and points to agentic tasks as a possible application.

7 Conclusion

  • RTT trains a Token-Level Relevance Discriminator to project coarse response-level rubric scores into fine-grained token-level reward signals.The framework targets limitations of response-level rewards in instruction-following tasks.Source: §7 · p. 10
  • RTT-GRPO integrates per-constraint token-level advantages with response-level signals, and intra-sample token group normalization is proposed as a group-partitioning strategy.The Table 6 visualization uses darker shading for higher discriminator relevance scores across constraint types.Source: §7 · p. 10, Table 6 · p. 11
  • Experiments report that RTT consistently outperforms baselines in instruction-level and rubric-level accuracy; the authors hope to apply it to agentic scenarios with long-horizon, multi-step constraints.The stated motivation for that possible application is the importance of credit assignment in those scenarios.Source: §7 · p. 10

A.1 Limitations

The paper identifies limited exploration of normalization partitions as a limitation, then outlines future work on agent-turn credit assignment, higher-dimensional reward partitioning, and applications beyond instruction following.

A.1 Limitations

  • Only inter-sample and intra-sample partitioning strategies are explored, leaving other ways to set normalization boundaries untested.The authors identify ordering normalization and aggregation across reward dimensions as an open direction.Source: §A.1 · p. 14

A.2 Future Work

  • A Turn-Level Relevance Discriminator is proposed as future work to identify which actions, tool calls, or reasoning steps satisfy sub-goals in agent trajectories.The authors suggest this could enable more precise policy updates for long-horizon tasks with sparse, delayed rewards.Source: §A.2 · p. 15
  • Future work could investigate structured partitioning across reward dimensions, especially as reward spaces grow to include sub-task dimensions.The passage points to group normalization theory and multi-objective optimization as possible theoretical foundations.Source: §A.2 · p. 15
  • Although RTT is evaluated on instruction-following tasks, the paper identifies long-form writing, code generation, and creative writing as possible extensions.A key open challenge is training a discriminator that transfers across task types and constraint categories without domain-specific annotation.Source: §A.2 · p. 15

B.1 Benchmarks and Evaluation Protocols

The evaluation protocol spans instruction-following and out-of-domain reasoning benchmarks, specifies benchmark-dependent accuracy metrics and generation settings, and details training and judging procedures. It also gives the discriminator’s token-relevance prompt.

B.1 Benchmarks and Evaluation Protocols

  • The evaluation uses full official datasets for IFEval, IFBench, MulDimIF, AdvancedIF, MATH-500, GPQA, and MMLU-Pro.The first four are in-domain instruction-following benchmarks; the final three are out-of-domain reasoning benchmarks.Source: §B.1 · p. 15
  • IFEval and IFBench report instruction-level AON pass rate and rubric-level constraint satisfaction rate; AdvancedIF reports Overall and Rubric accuracy, while MulDimIF reports accuracy only.AdvancedIF uses o3-mini as judge, and IFEval and IFBench use strict mode.Source: §B.1 · p. 15
  • Generation uses temperature 0.7, top-p 0.8, and top-k 20, averages five completions per instance, and applies model-default templates for instruction following.Out-of-domain tasks use added prompts; Table 7 is identified as the CoT evaluation prompts for those benchmarks.Source: Table 7 · p. 15, §B.1 · p. 16

B.2 Training and Evaluation Hyperparameters

  • SFT and DPO use LLaMA-Factory, RL uses ROLL, and discriminator training uses OpenRLHF; experiments run on NVIDIA A100-80GB GPUs.Training configurations are provided in Table 8.Source: §B.2 · p. 16, Table 8 · p. 16

B.3 LLM-as-a-Judge for Soft Constraints

  • The soft-constraint judge returns YES only for entirely fulfilled criteria and NO for failures or cases with no information to judge.Even minor inaccuracies exclude a response from YES.Source: §B.3 · p. 16

C.1 Discriminator Prompt

  • The discriminator prompt asks the model to identify which response tokens are relevant to the criteria.The passage identifies this as the prompt template for the token-level relevance discriminator.Source: §C.1 · p. 17

C.2 Data Sources and Construction

The discriminator corpus combines existing instruction-response data with generated responses and constraint-violating negatives. The paper defines relevance labels by constraint scope and polarity, then describes prompts for generating negatives and annotating relevant text.

C.2 Data Sources and Construction

  • The initial corpus draws 22,656 Tulu-3 SFT examples and all 16,969 HiR-16K samples, then adds responses generated by multiple frontier models.The merged positives and negatives form a 62,714-sample training set for token-level annotation and training.Source: §C.2 · p. 17
  • Negative responses are synthesized by minimally modifying a response to violate one constraint or by omitting a constraint.GPT-5.2 performs minimal modification, while Qwen3-4B-Instruct performs constraint omission.Source: §C.2 · p. 17, §C.4 · p. 18

C.3 Annotation Taxonomy

  • Global constraints mark every token relevant; local constraints concern specific spans and are classified by positive or negative polarity.Satisfied positive constraints mark matching text as partially relevant, while unsatisfied positive constraints mark all tokens irrelevant; negative constraints mark violating spans or all tokens irrelevant when satisfied.Source: §C.3 · p. 17
  • The annotator’s extracted spans are mapped to tokenizer indices and assigned binary labels: relevant tokens receive 1 and other tokens 0.All-relevant and all-irrelevant cases assign the same label across the entire response.Source: §C.3 · p. 17

C.4 Annotation Prompts

  • The annotation prompt uses scope and polarity branches, while span extraction requires exact original text segments rather than paraphrases.The negative-generation prompt requests the smallest change needed to break a single rule while preserving the rest of the response.Source: §C.4 · p. 18

D Inter-Sample Token Group Normalization

Inter-sample token group normalization pools token rewards across sampled responses separately for each constraint. Because this weights longer responses more heavily, a short response’s normalization statistics can be governed mainly by the other responses.

D Inter-Sample Token Group Normalization

  • For each constraint, inter-sample normalization groups tokens across all sampled responses, computes a shared mean and standard deviation, and averages per-constraint token advantages.The group contains all tokens across the G responses, and the normalized per-constraint advantages are combined across constraints.Source: D Inter-Sample Token Group Normalization · p. 20
  • The shared mean is a length-weighted average of response means, so longer responses contribute more to the centering statistic.For the shortest response, its contribution to centering and scaling shrinks with its length weight; its normalized advantages are then governed primarily by statistics from other responses.Source: D Inter-Sample Token Group Normalization · pp. 20–21
  • Intra-sample normalization instead computes mean and standard deviation within each response, using that response’s own token rewards and eliminating the stated length-bias effect.The passage contrasts these response-specific statistics with inter-sample statistics induced by the other responses.Source: D Inter-Sample Token Group Normalization · p. 21
Loading 2604.02795v1…