PaperlayerView exact revision on arXiv

Source-linked AI summary

RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

Kunwang Zheng, Xinyu Sun, Yu Cao, Shuocheng Li, Lei Cheng, Ying Liu, Qilong Zhang, Jinlong Chen, Hua Yang, Qianglong Chen

arXiv:2610.17623v2cs.LGcs.AIcs.CL

TL;DR

Rubric-based RL collapses instance-specific criteria into a scalar reward, discarding criterion-level detail, and tied groups have no GRPO reward gradient. RELAY distills from a frozen rubric-conditioned teacher on selected positive-advantage and low-scoring zero-advantage rollouts, then uses distillation-loss progress to hand off to GRPO alone. RELAY has the highest in-domain score among compared methods in all four model–domain settings, improving on GRPO by 4.62 and 6.02 points on RaR-Medicine.

  • Problem

    Rubric-based RL collapses instance-specific criteria into a scalar reward, discarding criterion-level detail, and tied groups have no GRPO reward gradient.

    Source: §1 · p. 2

  • Method

    RELAY distills from a frozen rubric-conditioned teacher on selected positive-advantage and low-scoring zero-advantage rollouts, then uses distillation-loss progress to hand off to GRPO alone.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    RELAY has the highest in-domain score among compared methods in all four model–domain settings, improving on GRPO by 4.62 and 6.02 points on RaR-Medicine.

    Source: §6 · p. 9

  • Takeaways & Limitations

    On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening, or refreshing it, while selected-rollout distillation beats all-rollout distillation at a matched average coefficient.

    Source: §6 · p. 9

    The plateau rule was calibrated on one model–domain pair and reused in three others, but conservative settings can miss the trigger, so substantially different dynamics or scales may require retuning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.

1 Introduction

Rubric-based RL reduces criterion-level feedback to scalar rewards, while ties can eliminate GRPO’s reward gradient; RELAY adds selective rubric-conditioned distillation and stops it when its loss plateaus. Across four model–domain settings, RELAY leads in-domain, with a calibrated stopping rule reused across settings.

1 Introduction

  • 10.9% of initial Qwen3-1.7B/RaR-Medicine groups tie, a third at nonzero scores, leaving all tied groups without a GRPO reward gradient.Rubric-based RL collapses multiple written criteria into one scalar reward, discarding which facts to include and errors to avoid.Source: §1 · p. 2
  • RELAY selects positive-advantage and low-scoring zero-advantage rollouts for privileged distillation, covering about half the response tokens and outperforming all-rollout distillation at a matched average coefficient.Selection uses rewards and advantages; the supplementary branch targets rollouts receiving little or no reward-driven gradient.Source: §1 · p. 2
  • 4.62 and 6.02 points: RELAY exceeds GRPO on RaR-Medicine for Qwen3-1.7B and Qwen3-4B, respectively, and has the highest in-domain score in all four settings.It also has the highest five-benchmark aggregate at both model sizes.Source: §1 · p. 2
  • A stopping rule tracks relative decreases in distillation loss and requires no additional evaluation; the plateau detector is calibrated once and reused unchanged across three other settings.The observed divergence falls by about two thirds and flattens as the validation lead over GRPO stops growing; retiring the teacher ends above continuing.Source: §1 · p. 2

2 Related Work

Prior work restores rubric detail through reward modifications, criterion-aware contexts, or privileged-context distillation, and explores several rollout-selection and stopping strategies. RELAY combines rubric-conditioned teacher supervision with rubric-based RL and uses an online loss-based hand-off.

Rubrics as rewards and context

  • Rubric-based RL aggregates criterion scores into one reward; related approaches add token-level reward terms, reweight advantages, expose criteria to the policy, or return unmet criteria for revision.RELAY instead keeps the reward and advantage unchanged and conveys rubric information through the teacher’s token distributions.Source: §2 · pp. 2–3

Rubric-conditioned self-distillation

  • Privileged-context self-distillation has shown early gains followed by degradation, while three rubric-conditioned methods use teacher supervision instead of a judge or verifier.Standalone frozen-teacher distillation scores below GRPO in all four in-domain settings, motivating its use alongside rubric-based RL.Source: §2 · p. 3

Rollout selection

  • Prior hybrids distill selected positive-advantage, incorrect, annotated, or highest-advantage rollouts; graded rubric rewards also permit ties at nonzero scores.RELAY supervises tied groups from the initial model and uses a running cutoff; selection changes the average coefficient, so rules are compared at matched coefficients.Source: §2 · p. 3

Stopping teacher supervision

  • Prior methods stop supervision by fixed decay, staged budgets, or selected switch steps; RELAY switches online from combined distillation and RL to RL alone based on loss change.Unlike RetireOPD, RELAY uses a training statistic without validation-set success rates; RL continues after the distillation term stops.Source: §2 · p. 3

3 Method

RELAY combines GRPO with rubric-conditioned supervision from a frozen copy of the initial model, selecting rollouts by reward and advantage and retiring the teacher when distillation-loss progress slows.

3 Method

  • Stage 1 combines selective teacher distillation with GRPO; after a plateau-triggered hand-off, Stage 2 continues with GRPO alone.The teacher scores sampled tokens by teacher forcing, without decoding.Source: §3 · pp. 3–4, Figure 2 · p. 5

3.1 Preliminaries

  • Rubric-based GRPO scores groups of sampled responses against signed criteria, and every response token shares its rollout’s advantage, which is zero for tied groups.The GRPO objective is averaged over valid response tokens and includes a token-averaged KL penalty against the initial model.Source: §3.1 · p. 4

3.2 The privileged teacher and its objective

  • The frozen teacher is the initial model conditioned on the prompt and full signed rubric, while the student answers without the rubric.The initial model also serves as the GRPO KL reference, so the teacher requires no additional model weights.Source: §3.2 · p. 4

3.3 Which rollouts to distill

  • Distillation targets positive-advantage rollouts and low-scoring near-zero-advantage rollouts, using a moving 30th-percentile reward cutoff; selected tokens average about 49% across settings.The selection rule targets rollouts with little reward-driven correction and controls the fraction of response tokens receiving distillation.Source: §3.3 · pp. 4–5

3.4 When to retire the teacher

  • The hand-off occurs when the relative decrease in selected-token distillation loss stays below tolerance; afterward, the coefficient is zero and GRPO continues without the teacher pass.The detector was calibrated on Qwen3-1.7B/Medicine and reused unchanged in the other three settings.Source: §3.4 · p. 5

4 Experiments

The experiments compare RELAY with GRPO, standalone distillation, and a CriPO reproduction across four model–domain settings and additional benchmarks. RELAY leads in-domain across all four settings, while standalone distillation trails GRPO.

4.1 Setup

  • Evaluation uses separately trained Qwen3-1.7B and Qwen3-4B models on RaR-Medicine and RaR-Science, with corresponding test splits and same-domain benchmarks.GPT-5.1 evaluates responses independently of the training judge; the baseline set includes the base model, OPSD, GRPO, and a CriPO reproduction.Source: §4.1 · p. 6

4.2 Main results

  • RELAY leads in-domain across all four settings, beats GRPO by 4.62 and 6.02 points on RaR-Medicine and 4.21 and 1.44 on RaR-Science, and has the highest five-benchmark aggregate at both sizes.The Medicine and Science deltas are for the 1.7B and 4B models, respectively.Source: §4.2 · p. 6
  • RELAY has the highest score on four of five benchmarks at 1.7B and all five at 4B; HealthBench gains over GRPO are 5.4 and 3.1 points.On LLMEval-Med, RELAY scores 57.51 and 69.14 at the two model sizes.Source: §4.2 · p. 6
  • Standalone OPSD improves on the base model in all four in-domain settings but trails GRPO, with validation scores peaking between steps 20 and 185 before declining.Source: §4.2 · p. 6

5 Analysis

The analysis tests teacher retirement, the plateau trigger, what the monitored loss measures, and rollout selection. Removing the teacher from a shared checkpoint yields larger later gains than continued distillation, while selection covering about half the response tokens outperforms all-rollout distillation at a matched average coefficient.

5.1 Does retiring the teacher help?

  • From a shared step-200 checkpoint, removing the teacher gives the largest average gain in every reported interval from steps 270–300 onward, reaching 5.06 points versus 2.88 for keeping it at full strength over steps 360–390.Source: §5.1 · pp. 6–7, Table 3 · p. 7
  • After hand-off, entropy is 0.32–0.42 by steps 281–300 versus 0.60 under continued distillation, but conditional KL alone does not establish a rise in full-vocabulary entropy.Source: §5.1 · pp. 7–8, Table 4 · p. 7

5.2 Is the trigger well placed, and does it transfer?

  • The tested step-171 switch scores 46.77 on RaR-Medicine versus 45.81 at step 140 and 46.38 at step 200; the unchanged detector fires between steps 122 and 171 across all four settings.The detector was calibrated once and transferred without re-tuning despite divergence scales differing by more than a factor of two.Source: Figure 3 · p. 7, §5.2 · p. 8

5.3 What the plateau measures

  • Conditional teacher–student KL falls from 0.55 to 0.18 by step 150 and plateaus as RELAY’s lead over GRPO reaches 2.9 points by step 110 and stops growing.At step 140, the rubric-conditioned teacher scores 90.5 versus 40.4 for the student; top-1 predictions still differ at 56% of positions by Stage 1’s end.Source: §5.3 · p. 8

5.4 Which rollouts to distill

  • At fixed α = 0.01, selecting rollouts covering at most 0.54 of response tokens yields 7.5–7.9 points of peak gain, while the two variants above 0.9 yield 7.3 and 6.8.The selected-rollout rules covering roughly half the tokens outperform distilling almost all rollouts in these runs.Source: §5.4 · p. 8, Table 5 · p. 8
  • At a matched average coefficient of 0.0049, all-rollout distillation still trails the selection rule by 0.8 peak-gain points and produces longer, more often truncated rollouts.The matched-dose step-200 checkpoints differ by a paired 1.49 points on RaR-Medicine under GPT-5.1.Source: §5.4 · p. 8, Table 5 · p. 8
  • Adding low-scoring zero-advantage rollouts raises selected-token coverage from 0.42 to 0.49 and peak gain from 7.5 to 7.9, while increasing mean response length by 173 tokens.This branch includes tied-group rollouts and those at the mean reward in groups with unequal rewards.Source: §5.4 · pp. 8–9

6 Conclusion

RELAY selects rollouts for privileged self-distillation and retires the teacher when loss progress slows. It leads the compared methods in-domain across four settings, but the plateau does not establish full transfer of rubric-conditioned teacher capability.

6 Conclusion

  • RELAY has the highest in-domain score among compared methods in all four model–domain settings, improving on GRPO by 4.62 and 6.02 points on RaR-Medicine.Source: §6 · p. 9
  • On Qwen3-1.7B/Medicine, stopping distillation from a shared checkpoint beats keeping, weakening, or refreshing the teacher later; selected rollouts covering about half the response tokens also beat all-rollout distillation at a matched average coefficient.Standalone distillation remains below GRPO in all four in-domain settings.Source: §6 · p. 9
  • The plateau does not imply full rubric transfer: the rubric-conditioned teacher still scores substantially higher, and whether another objective could transfer more remains open.Source: §6 · p. 9

Limitations

The reported limitations concern calibration scope, the frozen reference, binary rollout selection, and unmeasured variation across training seeds.

Limitations

  • The plateau rule is calibrated on one model–domain pair; conservative settings can miss its trigger, so substantially different dynamics or scales may need retuning.The rule was reused unchanged in three other settings.Source: Limitations · p. 9
  • The teacher remains frozen, and adapting it could extend the useful distillation window but would complicate a stopping rule that assumes a stationary reference.Source: Limitations · p. 9
  • Binary rollout weights limit supervision granularity; continuous weights could allocate it more finely, but their interaction with the stopping rule remains unstudied.Source: Limitations · p. 9
  • Only one seed was trained per configuration, so training-seed variation is unmeasured; reported standard deviations and confidence intervals instead reflect repeated evaluations and question resampling.Source: Limitations · p. 9

Ethical Considerations

The paper states that its medical-question outputs are not clinically validated and describes implementation and benchmark-data choices.

Ethical Considerations

  • The medical-question outputs have not been validated by clinicians and are unsuitable for clinical decision-making; all datasets are public research benchmarks, with no new human data collected.Source: Ethical Considerations · p. 9

A Implementation Details

  • The teacher prompt gives the signed rubric as private guidance, instructs it to satisfy positive requirements and avoid penalties, and scores student-generated tokens by teacher forcing.Prompts are longer than 5 and at most 1,024 tokens.Source: A Implementation Details · p. 12

B Data

  • HealthBench is subsampled to 500 questions, while the other benchmarks are used in full.Source: B Data · p. 12

C Evaluation

The evaluation uses GPT-5.1 on benchmark test partitions and reports repeated evaluations and question-bootstrap uncertainty. The paired in-domain differences are positive across all eight comparisons, while the intervals do not measure variation across training seeds.

C Evaluation

  • GPT-5.1 grades test responses using the Figure 5 rubric prompt; three of five benchmarks use adapted scoring protocols.ResearchQA uses binary unit-weight criteria; HealthBench averages before clipping; LLMEval-Med uses its own five-point criteria and GPT-5.1 grading.Source: C Evaluation · p. 12
  • Each checkpoint receives two additional evaluations with fresh responses, and Table 2 reports the mean across all three evaluations.Table 7 gives standard deviations across evaluations for each benchmark.Source: C Evaluation · p. 12
  • All eight paired in-domain differences have 95% question-bootstrap confidence intervals excluding zero; these intervals do not estimate training-seed variation.Figure 4 presents in-domain test-score gains from the first GPT-5.1 evaluation, with separate models trained for each domain.Source: C Evaluation · p. 12, Figure 4 · p. 12

D Case Study

The case study shows that rubric-specific explanation requirements distinguish otherwise correct answers, while the baseline descriptions specify how GRPO, OPSD, and the CriPO reproduction are configured.

D Case Study

  • All three runs reach the correct numerical answer, but the rubric distinguishes them on the required explanation of the intercept and blank correction.Source: D Case Study · p. 13
  • With total positive rubric weight 25, RELAY scores 1.00 while CriPO and GRPO each score 0.80 on the case-study question.The separating requirement asks for the intercept’s meaning and its connection to blank correction; omission also triggers a penalty.Source: D Case Study · p. 13, Table 9 · p. 14
  • The scalar reward reports the score without identifying the missing criterion, whereas the teacher has that criterion in context when scoring student tokens.Scoring only positive criteria would omit the additional penalty for the explanation’s absence.Source: D Case Study · p. 14

E Baselines

  • GRPO is RELAY with α = 0 throughout and no teacher; its Medicine trajectory is a single run used for the first 200 steps and full-budget schedule comparison.Source: E Baselines · p. 14
  • Standalone OPSD uses forward KL on all rollouts with coefficient 1 and no reward; its four runs peak and then flatten or decline, stopping between steps 73 and 208.Source: E Baselines · p. 14
  • The CriPO reproduction combines behavior injection with advantage correction; its behavior-injection coefficient is 2sβ, where s is the nonnegative-advantage rollout fraction.The reproduction uses β = 0.01; at the published β = 0.05, entropy exceeds 1.0 within 23–37 steps.Source: E Baselines · pp. 14–15

F The Plateau Rule and Distillation Diagnostic

The plateau rule’s default configuration triggers across all four settings, while the selected-token fraction remains near one half. The diagnostic also clarifies that conditional KL tracks progress on selected distributions, not equality of full distributions.

F The Plateau Rule and Distillation Diagnostic

  • The default plateau configuration fires in all four settings, with window and tolerance choices affecting whether and when the rule triggers.The deployed configuration uses W = 20 and τ = 0.05; shorter windows fire earlier, while longer windows and tighter tolerances often exhaust the available trace.Source: F The Plateau Rule and Distillation… · p. 15, Table 10 · p. 15
  • The selected-token fraction averages 0.476–0.505 across Stage 1 and never falls below 0.37, without per-setting tuning.The fraction is the share of response tokens admitted by the current-reward percentile cutoff.Source: F The Plateau Rule and Distillation… · p. 15, Table 11 · p. 15

G Interpreting the Conditional KL

  • The conditional KL omits set-mass and tail divergences, so zero loss does not imply equality of the full distributions.The plateau is an operational stopping signal, not evidence of full-distribution convergence.Source: G Interpreting the Conditional KL · p. 16

H Hand-off Schedules

  • Table 12 compares hand-off times with GRPO and distillation-to-end schedules, while Table 4 reports their entropy trajectories.The supplied passage identifies the comparisons but does not state their outcomes.Source: H Hand-off Schedules · p. 16

I Gradient Alignment

  • From step 100 onward, estimated GRPO–OPSD gradient cosines are small and have both signs; only the initialization interval excludes zero.Across checkpoints, positive- and negative-advantage OPSD gradients have cosines of 0.97–0.99.Source: I Gradient Alignment · pp. 16–17, Table 13 · p. 17

J Selection and Training Dynamics

Continuous-weighting variants keep the teacher throughout, but none matches the step-200 hand-off’s reported gain. Their end-of-training entropy is also higher than RELAY’s and GRPO’s.

J Selection and Training Dynamics

  • 13.13 points for the step-200 hand-off exceeded gains of 9.72, 9.66, 9.87, and 11.17 points for four continuous-weighting variants over steps 270–300.The variants used recentered or fixed half-Gaussian weights, a uniform per-step token budget, or reduced weight on negative-advantage rollouts.Source: J Selection and Training Dynamics · p. 17
  • At training end, CriPO entropy is 1.5–2.4, versus 0.1–0.4 for RELAY and 0.06–0.15 for GRPO.Source: J Selection and Training Dynamics · p. 17

Related papers

Loading 2610.17623v2…