PaperlayerView exact revision on arXiv

Source-linked AI summary

Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

Jeonghye Kim, Jiwon Jeon, Dongsheng Li, Yuqing Yang

arXiv:2605.10781v1cs.LGcs.CL

TL;DR

RLVR faces sparse credit assignment and reasoning-diversity challenges, while standard self-distillation can suppress successful student reasoning. RLRT reverses the teacher signal on correct rollouts to reinforce self-driven tokens, outperforming self-distillation and exploration baselines across Qwen3 checkpoints and six math benchmarks.

  • Problem

    RLVR provides only sparse trajectory-level rewards, while existing exploration methods treat reasoning diversity uniformly rather than identifying valuable successful alternatives.

  • Method

    RLRT augments GRPO by amplifying tokens on correct rollouts where the student diverged from the teacher yet succeeded.

  • Results

    8.9% average improvement over self-distillation baselines on six math reasoning benchmarks across base, instruction-tuned, and thinking-tuned Qwen3 checkpoints.

  • Takeaways & Limitations

    Information asymmetry can serve as a principled intrinsic source of valuable exploration and a new design axis for RLVR.

  • Takeaways & Limitations

    The experiments are limited to self-distillation with shared teacher–student parameters and mathematical reasoning.

Abstract

from arXiv · show

Self-distillation has emerged as a powerful framework for post-training LLMs, where a teacher conditioned on extra information guides a student without it, both from the same model. While this guidance is useful when the student has failed, on successful rollouts, the same mechanism instead overwrites the student's choices and suppresses it's own reasoning. Therefore, we propose reading the original self-distillation signal in reverse: when the student succeeds along a path the teacher would not have predicted, these tokens reflect its self-driven reasoning. Building on this, we propose RLRT (RLVR with Reversed Teacher), which augments GRPO by reinforcing these tokens on correct rollouts. We interpret this as a new form of exploration in RLVR: not uniform diversity, but valuable exploration grounded in the student's own success. Across base, instruction-tuned, and thinking-tuned Qwen3 checkpoints, RLRT substantially outperforms self-distillation and exploration-based baselines, establishing information asymmetry as a new, principled design axis for RLVR.

1 Introduction

The paper reframes self-distillation’s teacher–student gap on correct rollouts as evidence of the student’s self-driven reasoning rather than an alignment target. RLRT reverses the signal on these successful paths, augmenting GRPO to reinforce such tokens as targeted exploration.

  • Results: RLRT shows faster training-score growth and consistently outperforms self-distillation and other RLVR baselines across base, instruction-tuned, and thinking-tuned Qwen3 checkpoints.The reported evaluations span Qwen3-4B/8B-Base, Qwen3-4B-Instruct, and Qwen3-8B.
  • Motivation: RLVR’s sparse end-of-trajectory reward creates a credit-assignment bottleneck, while self-distillation uses teacher–student information asymmetry to provide dense token-level supervision.The teacher receives additional information, whereas the student does not.
  • Motivation: On failed trajectories, teacher distillation supplies corrective guidance; on already-successful trajectories, it can overwrite the student’s reasoning instead of preserving it.The mechanism’s value therefore depends on whether the rollout was correct.
  • Reversed Teacher Signal: On correct rollouts, tokens that differ sharply from teacher predictions identify self-driven choices because the student reached the correct answer despite disagreeing with the teacher.The paper interprets these disagreements as successful reasoning choices rather than arbitrary deviations.
  • RLRT: RLRT reverses self-distillation on correct rollouts, amplifying tokens where the student reasoned differently from the teacher while still succeeding.The method augments GRPO and treats this targeted reinforcement as exploration grounded in successful student reasoning.

2 Related Works

Prior work uses privileged information within a teacher–student model to align the student toward teacher behavior and addresses RLVR’s reasoning boundary collapse through uniform diversity objectives. RLRT instead reverses this alignment, treating successful student deviations from teacher predictions as evidence for grounded exploration.

  • Prior asymmetric teacher–student methods: Information-asymmetry methods condition a same-model teacher on privileged context, such as reasoning traces, runtime errors, or other signals, to guide the student.Their alignment mechanisms include distribution matching, second-turn distillation, teacher-influence weighting, and failed-rollout-only alignment.
  • RLRT’s reversed alignment: RLRT inverts prior alignment by using student–teacher divergence on correct rollouts as evidence of self-driven reasoning.These are choices made against the teacher’s prediction that nonetheless reached the correct answer.
  • Exploration in RLVR: RLVR’s reasoning boundary collapse concentrates policies on narrow high-reward strategies, motivating token-level entropy regulation and sequence- or outcome-level diversity objectives.Existing remedies operate over individual tokens or full reasoning traces, respectively.
  • RLRT’s grounded exploration: RLRT identifies teacher-departure tokens within correct rollouts, producing exploration grounded in the student’s successful reasoning rather than uniform diversity or heuristic variation proxies.This contrasts with approaches that rely on local stochasticity or heuristic surrogates of variation.

3 Preliminaries

This section defines the response, prefix-history, reward, and vocabulary notation, then introduces self-distillation in RLVR and contrasts distribution matching with magnitude-only teacher-guided credit assignment. RLSD uses the verifiable reward to determine update direction while the teacher modulates token-level magnitude.

  • Notation: The notation defines a prompt x, response y, prefix history h_t, verifiable reward R ∈ {0, 1}, and vocabulary V.The response is generated by policy π_θ, with y_<t and y_>t denoting its prefix and suffix.
  • Self-distillation in RLVR: In self-distillation RLVR, one model acts as student and teacher, with the teacher additionally conditioning on privileged context c hidden from the student.The student conditions only on h_t, while c may be a ground-truth solution or successful rollout.
  • Self-distillation in RLVR: OPSD uses the teacher-derived ∆_t to drive the student distribution directly, whereas RLSD treats the ratio as a magnitude-only credit signal because the student lacks c.RLSD addresses the ill-posedness of matching a target conditioned on information unavailable to the student.
  • Self-distillation in RLVR: RLSD uses sign(A) for direction-aware credit assignment: favored tokens are amplified on correct rollouts and attenuated on incorrect rollouts.The verifiable reward determines update direction, while the teacher modulates magnitude across tokens within a trajectory.

4 Motivation

RLVR should reinforce reasoning tokens that diverge from the teacher on successful rollouts, because these choices identify self-driven paths and meaningful exploration. Token-level information asymmetry determines the update direction, while position-level asymmetry identifies critical positions where token choices affect correctness.

  • Motivation: Self-driven reasoning consists of student tokens that diverge from the teacher’s predictive distribution in ways influential to reaching the correct answer.These tokens are proposed as the targets for detection and reinforcement on correct rollouts.
  • Critical positions: Position-level asymmetry marks critical positions where token choice meaningfully affects the probability of a correct outcome.Most positions are low-asymmetry, while high-asymmetry positions are those where choices strongly affect the outcome.
  • Critical positions: InfS(t)^2 ≤2 ¯Dt, so small position-level asymmetry guarantees an inert position with negligible influence.The bound follows by controlling influence through total variation distance and applying Pinsker’s inequality.
  • Explore/exploit directions: At critical positions, ˆDt(yt) < 0 defines the teacher-aligned exploit direction, whereas ˆDt(yt) > 0 defines the student-driven explore direction.Reinforcing positive-asymmetry tokens moves the student onto a self-driven path consistent with success.
  • Teacher construction: In practice, a known correct solution c conditions the teacher distribution as P t T(·) = πθ(· | ht, c), approximating conditioning on success.The abstract success event R = 1 cannot be conditioned on directly.

5 RLRT: RLVR with Reversed Teacher

RLRT augments standard GRPO with a reversed-teacher token-level credit signal, reinforcing student choices that diverge from the teacher on correct rollouts. This treats teacher–student disagreement as valuable, outcome-conditioned exploration rather than correction.

  • RLRT method: RLRT modifies only GRPO’s token-level credit assignment, leaving rollout, reward, and trust-region machinery unchanged.It uses an informed teacher and amplifies tokens with ˆDt > 0 on correct rollouts.
  • Reversed-teacher signal: On correct rollouts, RLRT up-weights tokens the student chose against the teacher’s prediction, amplifying self-driven choices.The reversed-teacher signal is ˆDt, with positive values identifying these tokens.
  • Reversed-teacher signal: RLRT flips the teacher/student ratio relative to RLSD, treating teacher–student disagreement as valuable exploration rather than a correction.This reversal expresses a different training intent: RLSD aligns the student to the teacher, whereas RLRT reinforces successful disagreement.
  • Reward-gated update: The reverse weight applies only to correct rollouts, with λ ∈ [0, 1] controlling signal strength: λ = 0 recovers vanilla GRPO and λ = 1 yields full reverse weighting.The clip εw bounds the per-token advantage perturbation by λ · εw.

6 Experiments

Across base, instruction-tuned, and thinking-tuned Qwen3 checkpoints, RLRT outperforms GRPO and self-distillation baselines while inducing faster exploration and higher benchmark gains. Experiments further show that RLRT amplifies causally important tokens, reshapes candidate distributions beyond the base policy, and requires reward gating for stable training.

  • Experimental Setup: RLRT is evaluated on base, instruction-tuned, and thinking-tuned Qwen3 models using DAPO-Math-17k, with GRPO, SDPO, SRPO, and RLSD as comparison methods.SDPO is selected because it operates entirely on the model’s own rollouts, matching the self-distillation setup.
  • Performance Comparison: 18.0% (Qwen3-4B-Base), 12.0% (Qwen3-8B-Base), and 3.4% (Qwen3-4B-Instruct) are RLRT’s reported average benchmark gains over the best other methods.Across four backbones, RLRT also shows faster training-score growth and substantially outperforms GRPO and self-distillation baselines.
  • Causal Critical-Position Test: On the untuned checkpoint, max_kl produces twice the flip→R rate of random or min_kl, indicating that high-¯Dt positions causally affect correct outcomes.RLRT increases the max_kl flip→R gain from ∼18% to over 40% by step 100, whereas GRPO’s gain collapses toward random and min_kl.
  • Distributional Shifts: RLRT places more positions in the high-divergence regime and reorganizes the candidate set, pulling base-tail tokens into top positions rather than merely sharpening πbase’s preferences.GRPO and RLSD retain ∼80% of πbase’s candidates at k ≥3 in high-divergence positions.
  • Exploration Baselines: RLRT is compared with GRPO+EB and DIVER on pass@k curves for k ∈ {1, 2, . . . , 256} over AIME24 and AIME26, where GRPO+EB tracks GRPO and DIVER’s improvement is most visible at large k.The supplied passage states that DIVER’s margin remains narratively incomplete in the excerpt.
  • Ablations: Reward gating is necessary: RLRT-all causes response length and entropy to grow unbounded before training collapses around step 40, while εw = 1.0 gives the strongest training score on both base backbones.Tighter clipping keeps the reverse weight near unity and tracks GRPO closely.

7 Conclusion

RLRT reverses self-distillation on correct rollouts by amplifying successful student–teacher divergences as an information-asymmetry-based exploration signal. Experiments across Qwen3 variants show gains over GRPO, self-distillation, and exploration baselines, while broader extensions remain future work.

  • Method: RLRT amplifies tokens where the student diverged from the privileged-context teacher yet still succeeded, rather than pulling the student toward the teacher.The method inverts self-distillation specifically on correct rollouts.
  • Findings: RLRT formalizes this student-driven behavior through information asymmetry and demonstrates it as an exploration signal theoretically and empirically.The conclusion characterizes the signal as both principled and experimentally effective.
  • Findings: Experiments on base, instruction-tuned, and thinking-tuned Qwen3 yield substantial gains over GRPO, self-distillation, and exploration baselines.The reported gains span all three Qwen3 checkpoint categories listed in the conclusion.
  • Future Work: Future work includes extending RLRT to noisier rewards, other asymmetry forms, and broader on-policy distillation beyond self-distillation.The proposed broader distillation setting may draw teacher distributions from diverse sources.

A Limitations and Future Directions · B RLRT Algorithm

RLRT reframes teacher–student information asymmetry as a source of exploration, while its algorithm modifies GRPO only through a reverse per-token weight and reward gate. The paper identifies limitations involving the self-distillation setup and proposes varying teachers, privileged information, and distillation regimes in future work.

  • A Limitations and Future Directions: RLRT is presented as the first method to improve RLVR with distillation by reversing, rather than aligning to, the teacher’s signal.The paper interprets the asymmetric teacher–student signal as exploration rather than imitation and reports consistent gains across diverse model families.
  • A Limitations and Future Directions: The setup is limited because it relies on a self-distillation framework where the teacher is derived from the same model.The supplied passage introduces this limitation but is truncated before stating the remaining details.
  • A Limitations and Future Directions: Future work could replace the self-distillation teacher with a separate stronger reasoning model or, conversely, a weaker teacher.The passage frames teacher variation as one axis for extending RLRT, including on-policy distillation with a stronger model.
  • A Limitations and Future Directions: Future work could vary the privileged information supplied to the teacher, including process feedback, partial hints, or failed attempts instead of a complete successful rollout.This is identified as a second axis for studying RLRT.
  • A Limitations and Future Directions: The paper also proposes characterizing RLRT under off-policy distillation.The supplied passage introduces this as a third future-work axis but is truncated before providing further detail.
  • B RLRT Algorithm: Algorithm 1 changes GRPO structurally only through the per-token reverse weight in Eq. (6) and the reward gate in Eq. (7).Rollout, reward, and trust-region mechanisms remain unchanged.
  • B RLRT Algorithm: For each trajectory, RLRT computes a verifiable binary reward r(y^(k)) ∈ {0, 1} and a group-standardized advantage A^(k).The algorithm applies these computations across trajectories k = 1, . . . , K.
  • B RLRT Algorithm: RLRT uses token-level information asymmetry for its reversed-teacher advantage and falls back to the vanilla GRPO advantage otherwise.The resulting advantages are used in the standard GRPO surrogate update.

C Proofs and Supporting Results · C.1 Proof of Lemma 1 · C.2 Proof of Theorem 2

The supporting results derive Lemma 1 by factorizing the teacher-conditioned distribution and prove Theorem 2 in two steps: total-variation control followed by Pinsker’s inequality.

  • C.1 Proof of Lemma 1: Lemma 1 identifies T(v) with the teacher-conditioned probability πθ(v | h_t, R = 1).The proof begins from the definition of P_t and writes T(v) as πθ(v | h_t, R = 1).
  • C.1 Proof of Lemma 1: The teacher-conditioned probability factorizes as πθ(R = 1 | h_t, y_t = v) · πθ(v | h_t).The numerator and denominator are then simplified using their definitions.
  • C.2 Proof of Theorem 2: Theorem 2’s proof has two steps: express InfS(t) in closed form and bound it by total variation, then apply Pinsker’s inequality.This proof structure explicitly invokes Lemma 1 before the distance inequalities.
  • C.2 Proof of Theorem 2: Step 1 substitutes Lemma 1 into the definition of InfS(t) to obtain a total-variation bound.The proof denotes the relevant quantity by f(v) = ¯f_t before substitution.
  • C.2 Proof of Theorem 2: The total-variation expression follows from the definition TV(P, Q) := 1/2 … .The second-to-last equality is justified using the definition of total variation.
  • C.2 Proof of Theorem 2: Step 2 invokes Pinsker’s inequality for any two probability distributions P and Q on a common space.The inequality relates total variation to KL divergence.

D Marker Statistics for the Explore/Exploit Reading · E Further Examples of Critical Positions and Explore/Exploit Directions · F More Results

The marker analysis identifies reliable explore- and exploit-side tokens through student–teacher asymmetry, while additional rollouts show that high-asymmetry tokens cluster at critical reasoning positions. The supplied passages do not report substantive findings for F More Results.

  • D Marker Statistics for the Explore/Exploit Reading: 200 trajectories from 8 rollouts per problem were analyzed across 100 DAPO-Math-17k problems, retaining one correct and one incorrect trajectory per problem.At each position, the student-favored token enters the explore corpus and the teacher-favored token enters the exploit corpus.
  • D Marker Statistics for the Explore/Exploit Reading: 251 explore-side and 171 exploit-side candidates survived |zv| ≥3 before stopword removal.The filtering used α = 0.5 and excluded NLTK English stopwords plus a domain-specific list of approximately 400 words.
  • D Marker Statistics for the Explore/Exploit Reading: Stopword removal yielded 38 explore-side and 61 exploit-side markers.The domain-specific stoplist covered math vocabulary, Greek letters, LaTeX fragments, tokenizer artifacts, English numerals, and generic non-discourse fillers.
  • D Marker Statistics for the Explore/Exploit Reading: The retained markers include all tokens not mapping to predefined discourse categories, with 13 explore and 32 exploit tokens in the “Other” rows.The table defines positive zv as teacher-suppressed and negative zv as teacher-favored.
  • D Marker Statistics for the Explore/Exploit Reading: Varying |zv| across {2, 3, 5} preserves the qualitative pattern: the same discourse categories dominate, while only category-tail depth changes.Thus, the explore/exploit reading is qualitatively robust to the marker threshold.
  • E Further Examples of Critical Positions and Explore/Exploit Directions: Figure 10 replicates the decomposition on an additional rollout: most tokens have small asymmetry, while a few high-asymmetry tokens identify critical positions.This extends the single-trajectory illustration from Figure 2 and argues that the pattern is not an artifact of one example.
  • E Further Examples of Critical Positions and Explore/Exploit Directions: At critical positions, exploit-leaning candidates such as “Final” and “Conclusion” push toward closing the argument, whereas explore-leaning candidates are identified by D̂_t > 0.Exploit candidates satisfy D̂_t < 0; the displays show the top four candidates on each side.

F.1 Benchmark Results on Qwen3-4B-Instruct … H Full-Trajectory Heatmaps of ¯Dt

RLRT improves Qwen3-4B-Instruct benchmark performance while reversing teacher-guided weighting to reinforce student-favored reasoning on correct rollouts. The experiments also characterize SDPO’s instability, standardize baseline implementation and hyperparameters, and show that information-asymmetry signals are sparse and concentrated at meaningful decision points.

  • F.1 Benchmark Results on Qwen3-4B-Instruct: 3.4% average improvement on avg@16 over the best baseline is achieved by RLRT on Qwen3-4B-Instruct across six mathematical benchmarks.The benchmarks are AIME24/25/26, HMMT26, AMC23, and MATH500.
  • F.2 Behavior of SDPO on Base Models: SDPO can become unstable in math reasoning by excessively suppressing hedging and reflective tokens that support robust reasoning.The collapse is reported as particularly severe on the Base model, while SDPO is otherwise described as improving in-domain performance and shortening responses.
  • F.2 Behavior of SDPO on Base Models: SDPO collapses quickly on Qwen3-8B-Base: training reward drops while response length increases sharply.These trends are shown in the training-reward and response-length curves.
  • G Experimental Details: Experiments implement GRPO with entropy bonus, SRPO, RLSD, and RLRT alongside the Kim et al. implementation, while DIVER uses its official code.All methods train on DAPO-Math-17k with matched hyperparameters, using 2×B200 GPUs.
  • G.1 Details of Baseline Algorithms: All baselines share the GRPO surrogate and differ in privileged teacher context, per-token advantage weight, and trajectory-level gate.Each baseline is run with the primary settings recommended in its original paper.
  • G.1 Details of Baseline Algorithms: RLRT reverses RLSD’s teacher preference on correct rollouts: it up-weights student-favored tokens with ˆDt>0 rather than teacher-favored tokens with ˆDt<0.RLRT additionally gates this weighting on correct rollouts, amplifying self-driven reasoning instead of imitating the teacher.
  • G.2 Hyperparameters: RLSD and RLRT share λinit = 0.5 while sweeping ϵw ∈{0.2, 0.5, 1.0}; RLSD favored ϵw = 0.2, whereas RLRT remained above GRPO across settings.SDPO and SRPO follow their original hyperparameters, with SRPO sweeping β ∈{0, 0.5, 1}.
  • H Full-Trajectory Heatmaps of ¯Dt: Full-trajectory heatmaps show that critical ¯Dt positions are sparse, concentrated, and clustered at semantically meaningful junctions, while most tokens are decision-insensitive.Greener tokens indicate choices that can change correctness, whereas pinker stretches are routine.
Loading 2605.10781v1…