PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation

Karn Tiwari, Varnith Chordia, Prathosh A P

arXiv:2610.04596v1cs.LG

TL;DR

OPD provides dense token-level guidance but is weakly tied to trajectory correctness, while GRPO uses outcome feedback with coarse credit that vanishes on reward-uniform groups. DiffGate combines GRPO with bounded teacher guidance on failed trajectories, scaled by group difficulty. Across the reported Qwen3 mathematics and code settings, it improves pass@8 over GRPO.

  • Problem

    OPD's token-level teacher guidance is not directly aligned with trajectory correctness, while GRPO's coarse group-relative signal vanishes when sampled trajectories receive uniform rewards.

    Source: §1 · pp. 1–2

  • Method

    DiffGate combines GRPO with bounded teacher guidance applied only to failed trajectories and scaled by group difficulty.

    Source: §1 · p. 2

  • Results

    DiffGate improves pass@8 over GRPO in all four model–domain settings, with mathematics gains of 1.1 and 3.9 points and code gains of 1.6 and 5.7 points.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    Across the evaluated Qwen3 mathematics and code settings, DiffGate improves solution coverage while outperforming OPD baselines in avg@8 and pass@8.

    Source: §5.1 · p. 7, §6 · p. 10

    Effectiveness on agentic reasoning and long-horizon decision-making remains unexplored, and computational constraints limit evaluation to students up to 4B parameters.

    Source: §6 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) has emerged as a widely used paradigm for post-training large language models, reducing the train--test mismatch of conventional distillation by supervising the student on its own generated trajectories. However, existing OPD objectives remain largely token-local and outcome-agnostic, optimizing teacher--student agreement at each prefix despite reasoning quality being determined at the trajectory level. Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), provides complementary outcome-level supervision but suffers from sparse rewards and coarse credit assignment. We show that OPD and RLVR exhibit complementary blind spots: teacher signals provide dense local guidance but are weakly aligned with rollout correctness, whereas group-relative rewards capture task success but provide coarse token-level credit and vanish on all-failure groups. We introduce DiffGate, an outcome-gated objective that combines GRPO with selective, bounded teacher guidance. Teacher supervision is applied only to failed trajectories, scaled by group difficulty, and smoothly bounded to prevent extreme teacher--student discrepancies from dominating optimization. The verifier therefore determines \emph{which trajectories} receive teacher guidance, while the teacher provides dense token-level update directions within those trajectories. Across Qwen3-0.6B and Qwen3-1.7B students, DiffGate improves code avg@8 over matched GRPO by $+1.7$ and $+1.8$ points and pass@8 by $+1.6$ and $+5.7$ points, respectively. On mathematics, avg@8 remains within $0.5$ points of GRPO while pass@8 improves by $+1.1$ and $+3.9$ points. Overall, DiffGate improves pass@8 across all four model--domain settings, demonstrating improved solution coverage under our evaluation protocol.

1 Introduction

OPD supplies dense token-level guidance without directly encoding trajectory correctness, while RLVR captures task success with coarse credit that can disappear on reward-uniform groups. DiffGate combines them by applying bounded teacher guidance to failed trajectories according to group difficulty.

1 Introduction

  • Naive combination can conflict: teacher disagreement may not identify incorrect trajectories or decisive errors, and uniform imitation can alter correct solutions or let large discrepancies dominate updates.The paper also notes that strong imitation may restrict exploration and interfere with reward-driven learning.Source: §1 · p. 2
  • DiffGate applies bounded teacher guidance only to failed trajectories, scaling its strength with group difficulty and preserving successful trajectories from additional teacher pressure.Guidance is strongest for all-failure groups, where GRPO supplies no signal, and decreases as the group solve rate rises.Source: §1 · p. 2
  • OPD and GRPO have complementary blind spots: OPD is dense but outcome-agnostic, whereas GRPO is correctness-aware but coarse and silent on reward-uniform groups.The paper also notes sparse rewards and possible entropy collapse during RLVR optimization.Source: §1 · pp. 1–2
  • DiffGate is evaluated against GRPO and distillation baselines on mathematics and code across Qwen3 student sizes, with science transfer and design-factor analyses also included.The evaluation also examines difficulty gating, bounded guidance, rollout group size, and teacher strength.Source: §1 · p. 2

2 Related Work

Related work improves teacher-signal reliability, outcome-based supervision, and combinations of distillation with policy optimization. DiffGate is positioned as combining bounded guidance on failed trajectories with group-difficulty weighting.

2 Related Work

  • OPD research calibrates dense token-level teacher signals, but standard OPD remains token-local and teacher–student disagreement does not directly encode full-trajectory success.The cited work studies uncertainty-adaptive divergence, outcome-calibrated guidance, reasoning alignment, teacher capability, gradient noise, and policy divergence.Source: §2 · p. 2
  • In the distillation-only Qwen3-1.7B analysis, disagreement distinguishes correct from incorrect rollouts at chance on mathematics and weakly on code, and large gaps may occur away from decisive errors.The figure reports mathematics AUROC 0.50 and 0.49, and code AUROC 0.60; aggregate predictiveness also reflects response length.Source: Figure 1 · p. 3
  • GRPO and related RLVR methods optimize verifiable task success, but trajectory-level advantages are coarse and vanish when all responses in a group receive the same reward.Prior work addresses these limits through dynamic sampling, bias correction, and finer-grained credit assignment.Source: §2 · p. 3
  • Existing combinations include joint GRPO and reverse-KL distillation, selective imitation with trust regions, joint outcome-and-distillation rewards, and outcome-calibrated teacher supervision.DiffGate is distinguished by bounded teacher guidance on failed trajectories weighted by group difficulty.Source: §2 · p. 3

3 Background and Preliminaries

The preliminaries set up student-generated trajectories and contrast token-level teacher feedback with verifier-based sequence outcomes. OPD yields dense local guidance, while GRPO compares trajectory rewards within rollout groups.

3 Background and Preliminaries

  • The setup uses a fixed teacher policy and a parameterized student that samples responses autoregressively, with each token conditioned on the prompt and preceding generated tokens.The teacher is queried on contexts visited by the student, and the verifier evaluates the complete response.Source: §3 · p. 3
  • OPD trains on student-generated prefixes and commonly minimizes reverse KL at visited contexts, using sampled student tokens to estimate a dense token-level learning signal.A positive teacher–student log-probability gap increases the sampled token's probability; a negative gap suppresses it.Source: §3 · pp. 3–4
  • RLVR uses a binary verifier reward for complete-response correctness, while GRPO compares rewards among multiple responses sampled for the same prompt.The verifier does not identify which tokens contributed to success or failure, so GRPO applies the same trajectory-level signal across a response.Source: §3 · p. 4
  • When every trajectory in a group receives the same reward, the group-relative GRPO advantages vanish.The paper motivates DiffGate as trajectory-level routing: the verifier selects failed trajectories, while the teacher supplies token-level directions without locating a causal error token.Source: §3 · p. 4

4 Methodology

DiffGate combines the global GRPO outcome signal with bounded local teacher guidance. The verifier selects failed trajectories, group solve rate controls guidance strength, and teacher–student gaps provide local update directions.

4 Methodology

  • The proposed token-level advantage combines GRPO's trajectory-level outcome signal with a bounded teacher signal, addressing the contrast between dense but local OPD and global but coarse GRPO feedback.Teacher–student discrepancies can be unbounded and can disagree with outcome signals, including on correct trajectories.Source: §4 · p. 4, Figure 2 · p. 5
  • The teacher–student log-probability gap is transformed into a bounded signal that preserves its sign, is approximately linear near zero, and smoothly saturates for large discrepancies.This bounds extreme token-level updates while retaining the local direction of the OPD signal.Source: §4 · p. 5
  • Teacher guidance is activated only for failed trajectories and weighted by group difficulty, increasing as the group solve rate decreases.The group solve rate is the proportion of successful trajectories, and γ controls the dependence on difficulty.Source: §4 · p. 5
  • The verifier determines whether guidance is needed, the group solve rate determines its strength, and the teacher supplies local token updates rather than identifying the decisive error token.Successful trajectories receive no additional teacher pressure; guidance is maximal on all-failure groups, where GRPO has no signal.Source: §4 · p. 6
  • The token-level advantage is treated as stop-gradient, and λ = 0 recovers GRPO while γ = 0 gives the additive GRPO+OPD baseline without difficulty weighting.These controls isolate the effects of teacher guidance and difficulty gating.Source: §4 · p. 6
  • The analysis establishes a fixed-prefix policy-gradient interpretation, guidance on all-failure groups that decreases with solve rate, and bounded teacher and combined signals, without claiming full PPO convergence.Proofs are provided in Appendix Section C.Source: §4 · p. 6

5 Experiments

Experiments compare DiffGate with matched GRPO and distillation baselines across two Qwen3 student sizes, then evaluate benchmark performance, design choices, and training dynamics. The results emphasize improved pass@8 and examine when difficulty-gated, bounded teacher guidance contributes.

5 Experiments

  • The evaluation uses Qwen3-0.6B and 1.7B students, a fixed Qwen3-8B teacher, DeepMath and TACO training data, and matched GRPO baselines with identical training budgets.Mathematics and code use separate benchmarks, and science transfer is evaluated without additional science training.Source: §5 · pp. 6–7

5.1 Main Results

  • DiffGate improves pass@8 over GRPO in all four settings: mathematics gains are 1.1 and 3.9 points, while code avg@8 gains are 1.7 and 1.8 and pass@8 gains are 1.6 and 5.7 points.On mathematics, avg@8 changes by −0.5 and −0.3 points; DiffGate also outperforms OPD baselines on both metrics across both student sizes.Source: Table 1 · p. 7, §5.1 · p. 7, Table 2 · p. 8
  • On science benchmarks without additional science training, DiffGate reaches avg@8 scores of 31.2 and 39.2, exceeding the corresponding base models and best OPD baselines.Gains come mainly from GPQA and MMLU-STEM, while OlympiadBench-Physics changes relatively little.Source: §5.1 · p. 7
  • With a Qwen3-4B student, DiffGate exceeds the strongest OPD baseline by 2.9 avg@8 and 2.2 pass@8 points on mathematics, and by 2.5 and 2.8 points on code.The mathematics teacher is Qwen3-14B; the code teacher is Qwen3-8B.Source: §5.1 · p. 8

5.2 Ablation Studies

  • Removing difficulty weighting lowers avg@8 by 1.3–2.4 points across the four settings; gating matters more than gate sharpness, and uniform teacher guidance does not reproduce DiffGate’s gains.The gate-strength sweeps show mathematics nearly flat for λ ∈ {0.25, 0.5, 1} before degrading at larger values; varying γ from 1 to 3 has small, non-monotonic effects.Source: §5.2 · pp. 8–9
  • Replacing bounded teacher guidance with the raw log-probability gap lowers avg@8 and pass@8 across mathematics and code, while increasing median gradient norms by 1.4–3.3×.The raw signal also produces larger gradient spikes during training.Source: §5.2 · p. 8
  • Reducing rollout group size from 16 to 4 makes reward-uniform groups more frequent; GRPO loses 1.9 avg@8 points on average, compared with 1.2 for DiffGate.On 0.6B mathematics, DiffGate shifts from 0.5 points below GRPO at group size 16 to 1.2 points above it at group size 4.Source: §5.2 · p. 8

5.3 Why Does DiffGate Help?

  • All-failure groups persist on code, while DiffGate maintains bounded teacher guidance, higher entropy, and smaller gradient peaks; its code pass@8 advantage over GRPO appears early and persists across evaluated checkpoints.On 0.6B code, the teacher–student discrepancy on failed rollouts falls to 0.37 nats/token under DiffGate versus 0.56 under GRPO; on 1.7B it falls to 0.19 versus 0.30. Higher entropy is consistent with retaining a broader set of plausible solution modes.Source: §5.3 · pp. 9–10, Figure 4 · p. 10

6 Conclusion and Limitations

DiffGate is presented as addressing the local–global mismatch between OPD and RLVR by concentrating bounded, difficulty-gated teacher guidance on failed and difficult trajectories. The conclusion reports improved solution coverage and more robust behavior than OPD and GRPO, while noting that evaluation remains limited in task and model scope.

6 Conclusion and Limitations

  • Across Qwen3 students, DiffGate improves solution coverage and shows more robust behavior than OPD and GRPO.The conclusion describes teacher guidance as bounded and difficulty-gated, concentrated on failed and difficult trajectories.Source: §6 · p. 10
  • Evaluation covers mathematical reasoning and code generation across three student model sizes, but not agentic reasoning or long-horizon decision-making; students are limited to 4B parameters.Source: §6 · p. 10

A Notation and Conventions

B Failure Modes of On-Policy Distillation

Teacher–student disagreement is a weak indicator of trajectory correctness, especially on mathematics, and does not reliably identify where reasoning errors occur. The paper therefore assigns the verifier the role of selecting trajectories for guidance and the teacher the role of providing token-level update directions.

B Failure Modes of On-Policy Distillation

  • On MATH500 and DeepMath, disagreement yields AUROCs of 0.50 and 0.49, while MBPP+ reaches 0.60.On MBPP+, incorrect trajectories have 0.551 nats/token disagreement versus 0.478 for correct trajectories.Source: B Failure Modes of On-Policy… · p. 16
  • Aggregate disagreement is affected by response length, and the largest discrepancies often occur on ordinary words or identifiers rather than error-linked tokens.Thus, large local disagreement does not reliably identify whether a trajectory is wrong or where its error occurs.Source: B Failure Modes of On-Policy… · p. 16
  • DiffGate uses verifier outcomes to select failed trajectories for teacher guidance, which supplies dense token-level update directions.The paper distinguishes the verifier's role in deciding when correction is needed from the teacher's role in how to update the student.Source: B Failure Modes of On-Policy… · p. 16

C Theoretical Analysis

The analysis characterizes DiffGate’s bounded teacher signal, difficulty gate, and combined token advantage, while limiting the guarantees to the signal rather than the full optimization process.

C Theoretical Analysis

  • DiffGate’s analysis establishes a fixed-prefix gradient interpretation, difficulty-sensitive teacher guidance, and bounded teacher and token-level signals, but not convergence of the complete clipped PPO objective.The results characterize the learning signal used by DiffGate.Source: C Theoretical Analysis · p. 16

C.1 Bounded Teacher Guidance as a Policy Gradient

Consider the detached sampled-token surrogate

  • The fixed-prefix interpretation is local at the behavior policy; the implementation evaluates the teacher signal only on sampled student tokens and never computes the full-vocabulary divergence.The bounded signal agrees to first order with standard OPD for small teacher–student gaps and saturates for large discrepancies.Source: Consider the detached sampled-token… · p. 17

C.2 Difficulty Gating and Relative Supervision Strength

  • On all-failure groups, GRPO’s advantages vanish and the teacher gate is maximal; the average teacher weight decreases with solve rate and is zero when all trajectories succeed.For failed trajectories in mixed groups, the upper bound on teacher contribution relative to GRPO is non-increasing as solve rate rises.Source: §C.2 · p. 18

C.3 Bounded Combined Token-Level Advantage

  • The combined scalar token-level advantage is bounded, but that bound does not guarantee bounded parameter updates or improved task reward.Parameter-gradient norms, parameter movement, and KL drift also depend on the policy Jacobian, optimizer, and PPO clipping dynamics.Source: §C.3 · p. 19

D Implementation Details

  • Table 4 reports shared training and evaluation settings, including 128 prompts per step, different mathematics and code dataset sizes and budgets, and GRPO variants differing in teacher guidance.The mathematics and code training sets contain 21,928 and 4,082 prompts, with budgets of approximately 1.00 and 1.94 dataset passes, respectively.Source: Table 4 · p. 20

D.1 Models, Data and Baselines

The experiments compare DiffGate with GRPO and distillation baselines across Qwen3 students, math and code tasks, and additional science benchmarks. They examine training cost, evaluation settings, and how gating, bounded guidance, and rollout size affect results.

D.1 Models, Data and Baselines; D.5 Evaluation Protocol

  • Runs use Qwen3-0.6B and Qwen3-1.7B students, shared training data and optimizer settings, and n=16 rollouts for GRPO-based methods; DiffGate fixes λ=γ=1 across sizes and domains.Mathematics uses 21,928 DeepMath prompts and code uses 4,082 TACO prompts; evaluation generates eight responses per problem and reports avg@8 and pass@8.Source: §D.1 · pp. 20–21, §D.5 · pp. 22–23

D.2 Algorithm

  • DiffGate routes bounded token-level teacher guidance to failed trajectories, scaling it by group difficulty; successful trajectories receive no teacher update.The method uses the verifier to select trajectories and the teacher to supply token-level update directions.Source: §D.2 · p. 21

D.3 Training Cost; D.4 Hyperparameter Sweep

  • DiffGate’s wall-clock training time differs from GRPO by at most 12.5% across four settings, while standard OPD is generally cheaper because it uses fewer rollouts per prompt.The hyperparameter sweep finds that removing the gate costs 3.2 avg@8 on mathematics in one comparison, while stronger teacher settings can reduce avg@8.Source: §D.3 · p. 21, §D.4 · p. 22

E.1 Out-of-Distribution Transfer to Science

  • For 1.7B mathematics-trained students, DiffGate reaches 33.2 and 74.3 avg@8 on GPQA and MMLU-STEM, while every method stays within two points of the base student on OlympiadBench-Physics.These science benchmarks are evaluated without science training.Source: Table 6 · p. 24, §E.1 · p. 24

E.2 Scaling to Qwen3-4B Students

  • On Qwen3-4B, DiffGate achieves the strongest overall student scores, reaching 30.0 avg@8 and 47.2 pass@8 on mathematics and 57.3 avg@8 and 70.5 pass@8 on code.It also outperforms the strongest OPD baseline in both domains.Source: Table 7 · p. 24, §E.2 · p. 24, Table 8 · p. 25

E.3 Rollout Group Size

  • Reducing rollout groups from n=16 to n=4 lowers avg@8 by 1.9 points for GRPO and 1.2 for DiffGate on average across four settings.On 0.6B mathematics, GRPO drops 3.3 points while DiffGate drops 1.6; DiffGate also retains higher pass@8 in all four settings at both group sizes.Source: Table 9 · p. 25, §E.3 · p. 25

E.4 Bounded versus Unbounded Teacher Term

  • Bounding the teacher signal improves avg@8 by 1.0–1.5 points and pass@8 by 0.8–1.3 points across all four settings versus the unbounded variant.The bound caps large token-level teacher signals at 2, and the unbounded variant produces a largest gradient step of 342 versus 4.3 on 0.6B code.Source: §E.4 · pp. 25–26, Table 10 · p. 26, Figure 6 · p. 26

E.5 Difficulty Gating and the Additive GRPO+OPD Baseline

  • Against additive GRPO+OPD, difficulty gating improves avg@8 by 2.4 points on mathematics at both model sizes and by 1.3–1.4 points on code.Pass@8 gains are 2.8 and 4.1 points on mathematics and 0.1 and 1.4 on code.Source: §E.5 · p. 26

E.6 Selector and Additive Controls

The selector and difficulty weight determine where teacher guidance is applied, while training diagnostics examine reward-uniform groups and changing response behavior. The results show that guidance can increase solution coverage, but its effects vary by metric, domain, and training stage.

E.6 Selector and Additive Controls

  • All-rollout bounded teacher guidance raises pass@8 in three of four settings, including 5.6 points on 1.7B code, but lowers mathematics avg@8 by 2.3 and 1.2 points.Code avg@8 changes by at most 0.2 points relative to GRPO.Source: §E.6 · p. 27
  • Removing the failed-only selector lowers 1.7B mathematics avg@8 by 1.3 points and pass@8 by 2.6 points; avg@8 changes by at most 0.6 in the other settings.The selector’s contribution therefore varies across model size and domain.Source: §E.6 · p. 27
  • Difficulty weighting raises code avg@8 by 1.0 and 2.6 points for 0.6B and 1.7B students versus uniform additive guidance, while effects differ on mathematics.It raises 0.6B mathematics by 1.4 points and changes 1.7B mathematics by −0.4 points.Source: §E.6 · p. 27
  • Compared with the raw all-rollout GRPO+reverse-KL control, DiffGate improves code avg@8 by 1.4 and 3.2 points and pass@8 by 2.0 and 2.4 points across student sizes.On mathematics, the objectives differ by at most 0.2 avg@8 and 1.0 pass@8 points.Source: §E.6 · p. 27

E.7 Advantage Dynamics: DiffGate versus OPD and GRPO

  • All-failure groups account for 32–36% of mathematics prompts and 14–35% of code prompts at step 1; roughly 20% of 0.6B code prompts remain all-fail near training’s end.Reward-uniform groups provide no GRPO signal, while DiffGate guides all-failure groups and leaves all-pass groups unchanged.Source: §E.7 · pp. 27–28, Table 13 · p. 28
  • By training’s end, teacher–student disagreement is 1.3–1.7× higher on correct than failed rollouts under DiffGate, and 1.5–3.5× higher under GRPO.The average applied teacher weight decreases from roughly 0.5 to 0.1–0.2 on mathematics as performance improves.Source: §E.7 · p. 29, Figure 8 · p. 29

E.8 Training Reward and Response Length

  • Teacher-guided models shift response length rapidly in the first few training steps, then response lengths become relatively stable.The early shift coincides with decreasing teacher–student disagreement; the authors report no clear evidence of verifier exploitation but note they did not conduct a dedicated audit.Source: Figure 7 · p. 28, §E.8 · p. 30

E.9 Why the Gate, in One View

Training analyses connect DiffGate’s bounded, difficulty-weighted guidance with gradient, entropy, and teacher-contribution patterns. The reported gains vary by problem difficulty, and training curves show that solution-coverage advantages emerge early and persist across observed checkpoints.

E.9 Why the Gate, in One View

  • DiffGate has a lower median gradient norm than GRPO, 0.205 versus 0.311, and a smaller maximum, 0.68 versus 212.9.Replacing the bounded signal with the raw gap raises the median gradient norm to 0.293 and reduces evaluation performance from 21.1/39.1 to 19.9/38.0 avg@8/pass@8.Source: §E.9 · p. 30, Figure 9 · p. 31
  • GRPO entropy falls from approximately 0.30 to 0.048, while DiffGate and reverse-KL OPD end near 0.42–0.44.The authors say the teacher-guided methods’ entropy difference is within normal step-to-step variation and does not establish a causal link to pass@8.Source: §E.9 · pp. 30–31
  • DiffGate’s applied teacher contribution decreases from 0.087 per token in the first five steps to 0.017 in the final ten, as all-failure groups fall from roughly 31% to 3%.This decline follows the difficulty gate rather than an explicit annealing schedule.Source: §E.9 · p. 31

E.10 Where Does DiffGate Help? Difficulty and Teacher Reliability

  • On never-solved code problems, teacher pass@16 is 42–50%, compared with 94–98% in the remaining difficulty bands.The authors note that teacher reliability could further improve the routing mechanism.Source: §E.10 · p. 32
  • For 0.6B code, DiffGate’s largest gain is +1.13 successful samples per problem in the 5–11/16 difficulty band; for 1.7B, its largest gain is +0.52 on never-solved problems.On the easiest 12–16/16 band for 1.7B, DiffGate is 0.63 successful samples below GRPO.Source: §E.10 · p. 32, Table 15 · p. 32

E.11 Training Dynamics and Performance

  • On code, DiffGate leads GRPO in pass@8 from the earliest observed checkpoint and retains that advantage throughout training; on mathematics, it maintains higher pass@8 across checkpoints.The reported trends indicate that solution-coverage gains emerge early and persist rather than appearing only at the final checkpoint.Source: §E.11 · pp. 32–33, Figure 10 · p. 33

F Qualitative Examples

  • The qualitative examples use the same Qwen3-1.7B student, problem, and 16-sample evaluation pool, selecting cases with a large GRPO–DiffGate performance gap.The displayed reasoning and answers are the first sampled response from each arm.Source: F Qualitative Examples · p. 33

F.1 Code Examples

  • In one code example, GRPO includes the endpoint n in a sieve for primes strictly below n, while DiffGate uses indices 0 through n−1.The examples are illustrative, and the authors do not infer errors beyond those visible in the displayed generations.Source: §F.1 · pp. 33–34

F.2 Mathematics Examples

The mathematics examples contrast displayed reasoning and final answers from DiffGate and GRPO. The accompanying selection criterion excludes losing GRPO responses that contain the reference answer string.

F.2 Mathematics Examples

  • The mathematics pairs require that the reference-answer string be absent from the losing GRPO response.This criterion is intended to make the examples illustrate substantive reasoning or final-decision failures rather than answer-extraction failures.Source: §F.2 · p. 35
  • One displayed reasoning excerpt lists the possible integer bases as b = 7, 8, 9, …, 40.A separate excerpt gives the inequality b ≤ 41 < b².Source: §F.2 · p. 36
  • The DiffGate example preserves both solutions, f(x) = x and f(x) = −x, in its final answer.The passage identifies the response as retaining the complete solution set.Source: §F.2 · p. 37

G Limitations and Future Work

The evaluation is limited to Qwen3 students and verifiable mathematics and code tasks, while broader model families, scales, and reasoning domains remain future work. The authors also identify teacher reliability on difficult prompts and compute efficiency as areas for further investigation.

G Limitations and Future Work

  • Experiments focus on Qwen3 students and verifiable mathematics and code tasks, with models up to 1.7B parameters and a limited 4B evaluation.Extending DiffGate to other model families, larger scales, and broader reasoning domains remains future work.Source: G Limitations and Future Work · p. 37
  • On held-out code problems never solved by the base student, the 8B teacher achieves pass@16 of 42–50%, versus 94–98% on remaining difficulty bands.The authors identify online estimates of teacher reliability as a promising direction because reliability may decrease on difficult prompts.Source: G Limitations and Future Work · p. 37
  • Teacher scoring adds overhead beyond GRPO, but the authors report it remains modest and DiffGate remains effective with smaller rollout groups.Future work could explore fewer rollouts and adaptive teacher usage.Source: G Limitations and Future Work · p. 37
Loading 2610.04596v1…