PaperlayerView exact revision on arXiv

Source-linked AI summary

Training LLM Judges from Language Feedback via Position-Selective Self-Distillation

Ilgee Hong, Changlong Yu, Zhenghao Xu, Xin Liu, Yuwei Zhang, Qin Lu, Bing Yin, Tuo Zhao

arXiv:2609.38792v1cs.CLcs.LG

TL;DR

Outcome-supervised RL gives no explicit, separate credit to criterion-choice tokens, although preference rationales often name decisive criteria. The paper uses feedback-conditioned self-distillation and entropy-shift position masking; its judges outperform Dr. GRPO on evaluated subjective subcategories, and masking improves OOD generalization over naive SD.

  • Problem

    Outcome-supervised RL credits tokens only through final-verdict accuracy, providing no explicit guidance on criterion choice despite preference rationales that name decisive criteria.

    Source: §1 · p. 2

  • Method

    The method distills dense token-level distributions from a feedback-conditioned teacher and masks positions using entropy shift, retaining the lower tail of each generation’s distribution.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4

  • Results

    Self-distilled judges outperform Dr. GRPO by 2–9 percentage points on evaluated subjective subcategories, while higher-entropy-shift masking improves OOD generalization over naive SD.

    Source: §1 · p. 2, §4.2 · p. 9

  • Takeaways & Limitations

    The approach performs better than outcome-supervised RL on evaluated subjective tasks while remaining competitive on objective subcategories.

    Source: §1 · p. 2, §5 · p. 11

    Masking may not provide useful supervision when feedback fails to identify a decisive criterion or gives only generic guidance.

    Source: §6 · p. 11

Abstract

from arXiv · show

We study training LLM judges from natural language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation criteria the judge invokes and how it weighs them. The dominant approach, outcome-supervised RL (e.g., GRPO), credits every token in the rollout with a single scalar determined only by the accuracy of the final verdict, providing no separate credit at the criterion-choice tokens and ignoring the rich language feedback (e.g., preference rationales) that naturally accompanies preference labels. Self-Distillation (SD) is one natural way to use this language feedback: the same model, conditioned on this feedback, acts as a teacher providing dense, position-level supervision. However, not all positions carry equally useful signal. Using the per-position entropy shift between teacher and student, we identify two regimes: context sharpening, where the teacher concentrates probability on a particular feedback-aligned criterion expression, and context spreading, where the teacher distributes probability across multiple feedback-aligned alternatives. We interpret these patterns as follows: sharpening encourages memorization of a particular criterion expression, whereas spreading promotes semantic understanding by preserving these alternatives. Motivated by this asymmetry, we introduce position masking based on the entropy shift that retains the lower tail of the entropy-shift distribution. Experiments show that masking higher-entropy-shift positions improves out-of-distribution generalization over naive SD. The resulting self-distilled judges outperform judges trained with outcome-supervised RL by 2-9 percentage points on the evaluated subjective subcategories, while remaining competitive on objective ones.

1 Introduction

Outcome-supervised RL gives limited explicit guidance on criterion choice in subjective judgments, despite preference rationales naming decisive criteria. The introduction motivates feedback-conditioned self-distillation and entropy-shift masking, reporting improved subjective-task performance and OOD generalization.

1 Introduction

  • Outcome-supervised RL credits every rollout token according to final-verdict accuracy, leaving criterion choice and weighting without explicit, separate guidance.This is a limitation for subjective tasks, where choosing or weighting the wrong criterion can yield an incorrect verdict even with rigorous application.Source: §1 · p. 2
  • Preference rationales often name decisive criteria, and feedback-conditioned self-distillation uses them to provide dense, position-level supervision at criterion-choice tokens.The same model acts as teacher when conditioned on feedback.Source: §1 · p. 2
  • Entropy shift distinguishes context sharpening, which concentrates probability on one feedback-aligned criterion expression, from context spreading across multiple alternatives; masking retains the distribution’s lower tail.The paper interprets spreading as promoting semantic understanding and sharpening as encouraging memorization of a particular expression.Source: §1 · p. 2
  • Self-distilled judges outperform Dr. GRPO by 2–9 percentage points on evaluated subjective subcategories, while higher-entropy-shift masking improves OOD generalization over naive SD.Dr. GRPO remains competitive on objective subcategories.Source: §1 · p. 2

2 Related Work

Prior work uses natural-language feedback in refinement and critique pipelines, while on-policy self-distillation conditions a teacher on feedback unavailable to the student. Token-selection research also filters or reweights supervision, and this work studies entropy-shift-based position selection for judge training.

2 Related Work

  • Prior feedback methods include iterative inference-time revisions and training on refinements, self-critiques, principle-based revisions, or critique-conditioned GRPO rollouts.Text2Grad aligns critique phrases with response spans to create per-span differentiable reward signals [40].Source: §2 · pp. 2–3
  • On-policy self-distillation uses the same model as a feedback-conditioned teacher, with prior work applying diverse feedback types; this work uses annotator rationales that expose preference criteria.The rationales are especially relevant to subjective tasks, where generalization depends on selecting and weighting criteria.Source: §2 · p. 3
  • Token-selection work filters or reweights supervision using token importance, entropy, divergence, relevance, trajectory dynamics, or teacher reliability; this work selects positions by student–teacher entropy shift.The paper situates its approach in full-logit on-policy self-distillation for judge training.Source: §2 · p. 3

3 Method

The section moves from outcome-based judge training to feedback-conditioned self-distillation, analyzes how feedback shifts token uncertainty, and masks high-shift positions; SD+mask invokes more distinct criterion clusters than naive SD.

3.1 Preliminary: Outcome-Supervised RL for LLM Judges

  • Outcome-supervised RL scores each rollout by final-verdict correctness and gives no direct supervision where the judge chooses and weighs criteria.Source: §3.1 · p. 3

3.2 Language Feedback Self-Distillation for LLM Judges

  • Self-distillation uses one model as a prompt-only student and a feedback-conditioned teacher, transferring the teacher’s distribution at every token position, including criterion choices.Source: §3.2 · pp. 3–4

3.3 Analysis of Per-Position Self-Distillation Signal

  • Both ∆Ht tails carry approximately 20–28× the per-position KL of the middle 40% of positions.Source: Figure 2 · p. 4, §3.3 · p. 4
  • Criterion-selection spans are 16.7% of response tokens but account for 44.4% of reverse-KL mass, while criterion-name positions alone account for 11.1%.Source: §3.3 · p. 5
  • Under matched versus control rationales, sharpening top-1 similarity is 0.713 versus 0.647 (difference +0.066, 95% CI [0.033, 0.100]), consistent with concentration on one feedback-aligned expression.Source: §3.3 · p. 6
  • Spreading top-10 similarity is 0.646 versus 0.610 under matched versus control rationales (+0.037, 95% CI [0.022, 0.051]), supporting preservation of multiple feedback-aligned alternatives.Source: §3.3 · pp. 6–7

3.4 Position Masking Based on Entropy Shift

  • The per-generation mask ranks positions by ∆Ht, drops the top ρ fraction, and retains the bottom 1 −ρ fraction; ρ = 0 recovers naive SD.Source: §3.4 · p. 7
  • SD+mask invokes more distinct criterion clusters than naive SD at every evaluated threshold, with the gap widening from 4 clusters at t = 0.65 to 26 at t = 0.85.Source: Figure 5 · p. 7, §3.4 · p. 7

4 Experiments

The experiments evaluate self-distilled judges against outcome-supervised RL and other baselines on two OOD benchmarks, then test downstream selection and entropy-shift masking. Self-distillation gains most on subjective categories, and masking higher-shift positions improves overall accuracy and downstream creative-writing selection.

4.1 Setup

  • The experiments train Qwen3-4B and Qwen3-30B judges on cleaned HELPSTEER3-PREFERENCE with human rationales and evaluate OOD accuracy on RM-BENCH and REWARDBENCH V2.The comparisons include Dr. GRPO, naive SD, SD+mask, prompting baselines, and trained reward-model baselines.Source: §4.1 · p. 8

4.2 Main Results

  • On subjective subcategories, both SD variants exceed Dr. GRPO by 2–9 percentage points, while Dr. GRPO remains competitive on objective categories.SD+mask raises Total Avg from 77.70 to 79.04 at 4B and from 80.17 to 82.50 at 30B; the 30B judge also outperforms leading judge baselines and remains comparable to Claude-Sonnet-4.Source: §4.2 · pp. 8–9, Table 1 · p. 9

4.3 Downstream Utility for Policy Optimization

  • In multi-round best-of-eight tournaments, SD+mask improves creative-writing win rate by 7.6 points over Dr. GRPO and 2.0 points over naive SD.The tournament tests repeated pairwise selection but omits the gradient-based policy updates of a full GRPO run.Source: §4.3 · pp. 9–10, Table 2 · p. 10

4.4.1 Mask-fraction (ρ) sweep

  • At 30B, Total Avg rises from 80.17 to 82.50 as ρ increases from 0 to 0.7, with ρ = 0.7 highest among tested fractions.The 4B results also attain their highest Total Avg at ρ = 0.7.Source: Table 3 · p. 10, §4.4.1 · p. 10

4.4.2 Selector ablation

  • At matched ρ = 0.7, the top-∆Ht selector outperforms all five controls on overall RM-BENCH and REWARDBENCH V2 accuracy.The controls test masking direction, absolute entropy shift, random selection, and student- or teacher-entropy selection.Source: Table 4 · p. 10, §4.4.2 · pp. 10–11

5 Conclusion

The conclusion frames the work as dense, feedback-conditioned supervision for subjective LLM judging, with entropy-shift masking to improve generalization.

  • The work targets subjective judging, where verdicts depend on the criteria a judge invokes and how it weighs them.Source: §5 · p. 11
  • Per-position entropy shift distinguishes context sharpening from context spreading, motivating a mask that retains the lower tail of the shift distribution.Source: §5 · p. 11
  • Across model sizes, self-distillation outperforms outcome-supervised RL on evaluated subjective tasks, and masking improves generalization over naive self-distillation.Source: §5 · p. 11

6 Limitations and Scope

The method depends on feedback that identifies a decisive criterion, and the paper leaves long-chain-of-thought reasoning models for future work.

  • Entropy-shift masking does not guarantee useful supervision when feedback omits the decisive criterion or offers only generic guidance.Source: §6 · p. 11
  • The method does not explicitly address known self-distillation limitations, including hallucination and training instability in long-chain-of-thought reasoning models [15].Source: §6 · p. 11
  • The study focuses on instruction-tuned models for efficient judge inference; extending the method to long-chain-of-thought reasoning models remains future work.Source: §6 · p. 11

A Additional Context-Sharpening and Context-Spreading Examples

The appendix adds rollout examples of context sharpening and context spreading, using the same plot layout as the main-body figures.

B Additional Experiments with Rubric Feedback

The rubric-feedback experiments find that entropy-shift masking improves accuracy with both feedback formats, while preference rationales outperform longer generated rubrics.

C Additional Ablation Results

The 4B ablations test the mask fraction and position selector for SD+mask. A 0.7 mask fraction and signed entropy-shift selection give the strongest reported total averages among the tested settings.

C Additional Ablation Results

  • 79.04 total average at ρ = 0.7 is the strongest tested 4B mask-fraction setting, versus 77.70 for naive SD.The ρ = 0.3 and 0.5 settings reach 77.53 and 78.52, respectively.Source: Table 6 · p. 17, C Additional Ablation Results · p. 17
  • 79.04 total average for top-∆Ht masking exceeds 78.35 for the direction-flipped selector and at most 77.80 for the other tested controls.The controls are |∆Ht|, random, student-entropy, and teacher-entropy selection; all selectors use ρ = 0.7.Source: Table 7 · p. 17, C Additional Ablation Results · p. 17

D Implementation Details

The implementation uses a shared on-policy training setup for the 4B and 30B models, with method-specific learning rates and rollout counts. SD uses a top-k approximation for reverse KL, while full-vocabulary entropies inform ∆Ht.

D Implementation Details

  • All six runs use verl with FSDP sharding and vLLM rollouts, and AdamW with a constant learning rate and no warmup.Runs cover Qwen3-4B-Instruct and Qwen3-30B-A3B-Instruct, each trained with Dr. GRPO, naive SD, and SD+mask.Source: D Implementation Details · p. 18
  • Shared settings include batch size 256, three epochs, identical sample order, prompt and response limits of 4096 and 8192 tokens, and per-epoch A/B order swaps.These hyperparameters apply across all six training runs.Source: D Implementation Details · p. 18
  • Dr. GRPO uses 8 rollouts per prompt, while naive SD and SD+mask use 4 and k = 100; their learning rates vary by model scale.Dr. GRPO uses 1e−6 at 4B and 2e−6 at 30B; SD uses 5e−6 and 1e−5, respectively. SD approximates reverse KL with teacher top-k tokens and a tail-remainder bucket, computes ∆Ht from full-vocabulary distributions, and maintains an EMA teacher at update rate 0.01.Source: D Implementation Details · p. 18
  • Evaluation samples one rollout per prompt at temperature 0.7, top-p 0.8, and top-k 20, using the Appendix E prompt template.The decoding follows the official Qwen3-Instruct model cards.Source: D Implementation Details · p. 18

E Prompt Templates

The templates wrap conversation turns and candidate responses in role tags, then ask the judge to compare responses and return an A-or-B verdict. Variants prompt for task-specific quality dimensions or a ranked rubric, and the teacher can receive language feedback unavailable to the student and evaluation prompts.

E Prompt Templates

  • The prompt uses a single user message without a system prompt, wrapping conversation turns and candidate responses in <user> and <assistant> tags.The {context} placeholder holds one user block for a single-turn query or the full alternating dialog for a multi-turn conversation.Source: E Prompt Templates · p. 18
  • The teacher prompt inserts language feedback at {feedback}, while student and evaluation prompts omit that field.The examples show both a single-turn France query and a multi-turn Tokyo-planning conversation, with each candidate reply enclosed in an assistant block.Source: E Prompt Templates · pp. 18–19
  • The standard instruction asks the judge to identify task-relevant quality dimensions, compare responses step by step, and return only an A-or-B verdict.For correctness-sensitive tasks, it also instructs the judge to solve the problem and check each response for errors.Source: E Prompt Templates · p. 19
  • The rubric variant asks the judge to rank hard requirements and discriminative criteria, explain what makes responses better or worse, and compare them against that rubric.It replaces the final instruction paragraph of the shared prompt.Source: E Prompt Templates · p. 19
Loading 2609.38792v1…