PaperlayerView exact revision on arXiv

Source-linked AI summary

DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation

Ao Yu, Weibo Gao, Heng Zhou, Linan Yue, Rui Li, Suyi Liu, Yu Yan, Yizhong Zhang, Qi Liu

arXiv:2609.33711v1cs.LG

TL;DR

OPD can ignore correctness even as teacher and student success rates vary across tasks. DuoOPD uses student outcomes to set feedback direction and joint outcomes to select teacher support. It leads all five baselines across two model families, improving mean macro accuracy over OPD by 2.58 points on Qwen3 and 5.98 on Llama.

  • Problem

    Teacher and student success vary across tasks, while OPD ignores correctness and can suppress correct student responses or reinforce failures.

    Source: §1 · p. 1

  • Method

    DuoOPD uses student outcomes to set feedback direction and joint teacher-student outcomes to select teacher context and response-level weight allocation.

    Source: §1 · p. 2

  • Results

    DuoOPD leads all five baselines across two model families and three task mixtures, improving mean Macro over OPD by 2.58 points on Qwen3 and 5.98 on Llama.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    The tested task mixtures use the same settings, and the joint-outcome designs supply most of the gain while outcome-based direction alone stays near OPDVR.

    Source: §1 · p. 3, §7 · p. 11

    DuoOPD relies on binary verification; transfer to held-out tasks remains open.

    Source: §6 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) trains a student on its own responses with token-level feedback from a stronger teacher, yet the teacher can fail on questions the student already answers correctly, and how often each model succeeds varies across tasks. OPD ignores these outcomes and, on average, pushes down even the student's correct responses; gating feedback by student correctness fixes the direction but uses the teacher in the same way whether or not it succeeded. We introduce DuoOPD, in which the student's outcome sets the direction of feedback and the joint teacher-student outcome decides how the teacher supports it: when only the teacher succeeds, its verified answer becomes context for scoring the student's failed response, and when only the student succeeds, a weight shared within the task reinforces the whole response. A single rule covers all four outcome combinations without task-specific settings. Across Qwen3 and Llama, DuoOPD outperforms all five baselines in mean macro accuracy, improving over OPD by 2.58 and 5.98 percentage points, and it also leads on two further task mixtures spanning scientific calculation, instruction following, and code generation. Ablations show that outcome-based direction alone stays near the gated baseline, while the joint-outcome designs supply most of the gain.

1. Introduction

DuoOPD addresses task-varying teacher and student success by using student outcomes to set feedback direction and joint outcomes to choose teacher support. It leads the reported baselines across model families and task mixtures, with gains also appearing whether the teacher succeeds or fails.

Motivation

  • Teacher and student success rates vary across tasks, and OPD ignores correctness, so it can suppress correct student responses or reinforce failures.Across training responses, both models fail on 1% of chemistry-understanding cases but 34% of physics-calculation cases.Source: §1 · p. 1

DuoOPD

  • When only the teacher succeeds, its verified answer guides correction; when only the student succeeds, within-task weight sharing reinforces the complete response.When both succeed or both fail, teacher preferences set reinforcement or suppression strength; the same four-outcome rule uses no task-specific settings.Source: §1 · p. 2

Results

  • In every domain for both model families, DuoOPD raises student accuracy over OPD on questions the teacher solves and on questions it fails.Figure 1 splits student responses by fixed teacher outcome; each green arrow marks the rise above OPD in percentage points.Source: §1 · p. 2, Figure 1 · p. 2
  • DuoOPD leads all five baselines for Qwen3 and Llama, improving mean macro accuracy over OPD by 2.58 and 5.98 percentage points.It also leads on two additional mixtures spanning scientific understanding and calculation, instruction following, and code generation.Source: §1 · p. 2

Ablation

  • Outcome-based direction alone stays near OPDVR, while teacher references and within-task weight sharing supply most of DuoOPD’s gain over OPD.The two joint-outcome designs make complementary contributions.Source: §1 · p. 3

2. Problem Setup and Background

The setup jointly trains a student across a fixed task mixture with a frozen teacher, using task-specific verification to record student and teacher outcomes. The background defines sampled-token OPD feedback and a student-outcome gate that masks feedback opposing verified correctness.

Multi-task setting

  • For each task, questions have ground-truth answers, and a student is jointly trained on a fixed task mixture with a frozen teacher.Source: §2 · p. 3

On-policy distillation

  • Sampled-token OPD scores student-generated responses under both teacher and student at each prefix, producing a scalar feedback weight for each sampled token.The formulation is identified as the OPD baseline.Source: §2 · pp. 3–4

Outcome verification and gated feedback

  • OPDVR uses the task verifier’s binary student outcome to mask feedback opposing correctness and retain aligned weights.Source: §2 · p. 4

Outcome verification

  • An independently generated teacher response is also verified, yielding a joint teacher-student outcome whose distribution varies across tasks and as the student learns.Source: §2 · p. 4

3. DuoOPD: Distillation Guided by Joint Outcomes

DuoOPD uses verified teacher responses and fresh student responses to apply one outcome-conditioned feedback rule across tasks. Student correctness sets whether a response is reinforced or suppressed, while joint outcomes determine how feedback is allocated.

3.1. Principle: Direction from the Student, Support from the Pair

  • Student outcomes set feedback direction, while teacher-student outcomes determine its magnitude and support; the same rule applies across tasks without task-specific settings.When models disagree, support comes from the model that succeeded.Source: §3.1 · pp. 4–5

3.2. A Shared Feedback-Direction Rule

  • Softplus gives student successes positive weights and failures negative weights, preserving gated feedback while supplying nonzero feedback when gating returns zero.The gated weights on verified successes average 0.08–0.09, versus 0.57–0.65 under Softplus for Qwen3 and Llama training responses.Source: §3.2 · p. 5

3.3. Support When Teacher and Student Disagree

  • When only the teacher succeeds, its verified answer conditions teacher scoring, allocating negative correction across the unchanged failed student response.The student’s erroneous prefix can bias teacher scoring, so the teacher reference is added to its context.Source: §3.3 · p. 5
  • When only the student succeeds, all valid response tokens receive one positive weight shared within the task, preventing the failing teacher from weakening token-level feedback.The weight is computed once per rollout batch and preserves the task’s mean reinforcement scale because log-ratio magnitudes differ across tasks.Source: §3.3 · pp. 5–6

3.4. Student Update

  • The student update replaces each log-ratio weight with the joint-outcome weight; teacher references affect that weight, while gradients apply only to student log probabilities.Teacher references are cached and verified before training, while the student generates fresh responses at each update.Source: §3 · p. 4, §3.4 · p. 6

4. Experiments

Across three progressively heterogeneous task mixtures, DuoOPD leads the baselines; ablations attribute most of its gain to teacher references and within-task weight sharing, beyond outcome-based feedback direction alone.

4.1. Experimental Setup

  • The experiments compare Qwen3 and Llama models, five distillation baselines, and three increasingly heterogeneous task mixtures under shared data and verifiers.Final-checkpoint performance uses avg@8 task accuracy and equal-weight Macro; methods process the same fixed-order question stream, and all results average three training runs.Source: §4.1 · pp. 6–7

4.2.1. Comparisons Across Model Families

  • DuoOPD leads all five baselines on biology, chemistry, and physics for Qwen3 and Llama, exceeding OPD in mean Macro by 2.58 and 5.98 points.It also exceeds the strongest baselines, EOPD on Qwen3 and OPDVR on Llama, by 1.16 and 4.58 points.Source: Table 1 · p. 7, §4.2.1 · p. 7

4.2.2. Results on Additional Task Mixtures

  • On materials knowledge, chemistry understanding, and physics calculation, DuoOPD reaches 63.48% mean Macro, 0.86 points above ExOPD.Its largest task-level advantage is 3.25 points over the strongest baseline in physics calculation.Source: §4.2.2 · p. 7, Table 2 · p. 8
  • On physics knowledge, instruction following, and code generation, DuoOPD reaches 49.24% mean Macro, 1.41 points above OPDVR, and scores highest on every task.It best preserves IFEval performance alongside physics and code: 53.30% versus 50.95% for OPD.Source: Table 2 · p. 8, §4.2.2 · p. 8

4.3. Ablation Study

  • Replacing any one outcome rule with OPD feedback lowers mean Macro by 0.76–1.53 points, including the student-only-success rule for 6.9% of training responses.All four outcome rules contribute to performance.Source: Table 3 · p. 8, §4.3 · p. 8
  • Removing teacher references costs 1.49 mean Macro points and removing weight sharing costs 0.71; removing both drops accuracy to 64.56%, near OPDVR’s 64.83%.Within-task weight sharing outperforms sharing across tasks in all three mixtures, with larger margins in mixtures whose task scales differ more.Source: Table 4 · p. 9, §4.3 · p. 9, Table 5 · p. 9

4.4. Analysis by Joint Outcome

  • DuoOPD adds student successes in every domain for both model families, whether the teacher succeeds or fails; teacher references strengthen correction on teacher-only-success responses.Gains over OPD are 1.90 and 3.86 points where the teacher succeeds, and 0.67 and 2.12 where it fails, for Qwen3 and Llama respectively.Source: §4.4 · p. 9
  • OPD’s feedback is negative on average even for correct student responses, while setting direction by student outcome prevents this but alone stays near OPDVR’s accuracy.Correct responses make up 64–69% of training responses; the analysis attributes the additional gain to how DuoOPD uses teacher support after direction is fixed.Source: §4.4 · p. 9

5. Related Work

Prior multi-task methods adjust supervision at the task level, while DuoOPD changes feedback within each response. It combines outcome-based direction with teacher verification to choose scoring context and weight allocation.

Multi-task learning and distillation

  • Multi-task tuning, specialist distillation, and task-weight adaptation operate at task level; DuoOPD instead acts within responses of a fixed mixture and can compose with them.The cited approaches include task supervision pooling (Sanh et al., 2022), specialist distillation (Liu et al., 2019; Ma et al., 2026a,b), and task-weight adaptation (Ramesh et al., 2026).Source: §5 · p. 10

On-policy distillation and feedback calibration

  • DuoOPD uses verified teacher–student outcomes to set feedback direction and allocation, rather than entropy adjustment, teacher extrapolation, or confidence-based filtering.The contrasted methods are EOPD, ExOPD, and FiRe-OPD (Jin et al., 2026; Yang et al., 2026; Li et al., 2026).Source: §5 · p. 10

Verified outcomes and supervision routing

  • Student-correctness gating alone stays near OPDVR; DuoOPD also verifies the teacher to select scoring context and weight sharing.OPDVR masks feedback conflicting with student correctness (Lin et al., 2026).Source: §5 · p. 10

Privileged context and outcome-dependent teacher use

  • DuoOPD supplies a verified teacher reference only when the teacher alone succeeds, leaving student context unchanged and using the reference only to allocate negative feedback.Unlike H2SD’s hint-conditioned self-distillation, DuoOPD verifies an independent frozen teacher (Cai et al., 2026).Source: §5 · p. 10

6. Limitations and Future Work

DuoOPD’s evaluation and design have several stated boundaries. The authors identify verifier uncertainty, teacher computation, longer reasoning, and transfer to held-out tasks as open concerns.

Limitations and Future Work

  • Binary verification constrains DuoOPD; open-ended writing tasks need feedback rules for graded or uncertain judgments.The authors also identify longer reasoning trajectories as a setting where exploration and self-correction warrant study.Source: §6 · p. 10
  • Teacher computation could be reduced through shared caches, compressed references, or selective reference use.These are proposed directions, not demonstrated reductions.Source: §6 · p. 10
  • Because each mixture uses the same tasks for training and evaluation, transfer to held-out tasks remains unknown.Whether task-level outcome statistics ρ_k can guide task scheduling is also open.Source: §6 · p. 10

7. Conclusion

DuoOPD combines student-outcome feedback direction with joint-outcome choices about teacher support, and leads the reported baselines across the tested mixtures. The paper’s repository provides code, data-processing materials, cached responses, verifiers, and experiment configurations.

7. Conclusion

  • DuoOPD leads all five baselines across two model families and three task mixtures, improving mean Macro over OPD by 2.58 points on Qwen3 and 5.98 on Llama.It also leads on the hardest task of every setting.Source: §7 · p. 11
  • Outcome-based direction is necessary but insufficient; ablations attribute most of the gain to the two disagreement designs.Outcome-agnostic OPD pushes down many responses the student already answers correctly.Source: §7 · p. 11

Reproducibility statement

  • The repository includes code and setup instructions, processed data splits and rebuilding scripts, verifiers, cached teacher responses with generation settings, and five representative experiment configurations.The processed splits are provided alongside scripts to rebuild them from source datasets.Source: Reproducibility statement · p. 11

A. Data and Evaluation Protocols

The appendices describe three task mixtures, their data partitions and verifiers, and the teacher-reference and token-scoring procedure. The protocols specify how each dataset is constructed and evaluated.

A.1. Biology, Chemistry, and Physics

  • The biology, chemistry, and physics mixture uses SciKnowEval V2 questions, fixed shared partitions, and 960 training, 100 development, and 200 test questions per domain.The appendix reports no exact cross-partition stem matches or pairs above its 5-gram similarity threshold.Source: §A.1 · p. 14
  • The first mixture retains biology and chemistry multiple-choice questions and four-option physics questions; student answers are scored by the final boxed option, with malformed or invalid answers receiving zero.The ground-truth answer and teacher reference are withheld from student inputs.Source: §A.1 · p. 14

A.2. Materials, Chemistry, and Physics

  • The second mixture combines materials knowledge, chemistry understanding, and physics calculation, with 100 development and 200 test questions per task and a fixed 60-update training stream.Each update samples 16 questions per task, cycling and reshuffling smaller pools as needed.Source: §A.2 · p. 14

A.3. Physics, Instruction Following, and Code Generation

  • The third mixture combines physics, instruction following, and MBPP code generation, using official IFEval prompt-level checks and isolated execution against three original code assertions.Its test set contains 200 physics questions, 541 IFEval prompts, and 496 MBPP problems after duplicate removal.Source: §A.3 · pp. 14–15

B.1. Teacher Cache and Reference Prompt

  • DuoOPD caches one labeled teacher response per training prompt; when only the teacher passes verification, its complete response becomes a reference, while other outcomes use original-context scoring.The teacher prompt limit is 5,632 tokens.Source: §B.1 · p. 15
  • Teacher and student score the same student-response token IDs at aligned positions; loss covers response and actual EOS tokens, excluding prompts, references, and padding.Source: §B.1 · p. 15

B.2. Optimization and Implementation

The implementation uses a shared on-policy update setup with method-specific feedback and auxiliary-loss settings. Runtime comparisons include training-loop costs and teacher-cache costs relative to OPD.

B.2. Optimization and Implementation

  • OPD and DuoOPD share the verl scoring and update implementation, while EOPD retains an additional teacher-entropy-gated KL loss.ExOPD uses reward extrapolation, and FiRe-OPD uses trajectory filtering; both retain synchronous rollouts and token-level rollout importance sampling.Source: §B.2 · pp. 15–16
  • The implemented policy-ratio loss has the same gradient as Eq.(8), with clipping inactive because its on-policy ratio is one.The scalar loss values differ.Source: §B.2 · p. 16

B.3. Teacher Computation and Runtime

  • Student generation accounts for most of Llama's runtime increase, with DuoOPD responses averaging 177 tokens versus 152 for OPD.DuoOPD references only responses where the teacher alone succeeds.Source: §B.3 · p. 16
  • Table 7 compares DuoOPD training and teacher-cache costs with OPD, with cache GPU-time amortized over R runs.Runtime averages cover generation, scoring, and updates over 60 updates, excluding initialization, checkpoint saving, evaluation, and queue time.Source: §B.3 · p. 16, Table 7 · p. 17

C.1. Variability Across Training Runs

The supplied results describe training-run variability, development trajectories, and how DuoOPD changes feedback magnitude and allocation across teacher-student outcomes.

C.1. Variability Across Training Runs

  • Table 8 reports the Macro mean and sample standard deviation for the results in Tables 1 and 2.The supplied table caption identifies its metric as Macro test avg@8 (%) and mean ± standard deviation.Source: §C.1 · p. 17, Table 8 · p. 17

C.2. Development Trajectories

  • DuoOPD leads all baselines at every displayed Llama checkpoint from update 24 onward and stays above 70%; on Qwen3, it has the highest development Macro at seven of ten checkpoints, including the last three.Figure A1 tracks one run per method on 100 development questions per task, with four sampled answers per question, every six updates from the shared initial student.Source: §C.2 · p. 17, Figure A1 · p. 18

D. How Joint Outcomes Change Token Feedback

  • The analysis compares OPDVR's gate and DuoOPD's Softplus over training responses, then illustrates reference use, weight sharing, and feedback sign on fixed student responses.The gate and Softplus means are compared on the same responses using the original-context log-ratio d0; statistics pool three DuoOPD runs per family.Source: D. How Joint Outcomes Change Token… · p. 17, §D.1 · p. 17

D.1. Gated versus Softplus Feedback Magnitudes

  • The gate keeps little positive feedback on verified successes, while Table 9 compares its mean token weights with Softplus weights derived from d0.The gated means follow from recorded means of d0 and |d0|.Source: §D.1 · p. 17, Table 9 · p. 18

D.2. Reference-Guided Correction and Shared Reinforcement

  • With teacher-only success, a reference strengthens negative feedback on the student's incorrect phrase; with student-only success, sharing raises near-zero positive feedback on the correct option.Figure A2's examples hold the student response and negative Softplus mapping fixed for the reference comparison; the shared weight reinforces the correct selection.Source: §D.2 · p. 18, Figure A2 · p. 19

D.3. Feedback Direction on Shared Success and Failure

The feedback-direction examples show how Softplus treats responses when teacher and student agree on success or failure.

D.3. Feedback Direction on Shared Success and Failure

  • When both models are correct, positive Softplus changes an OPD weight of −3.83 on “Heating pads” to +0.0215.The positive weight aligns feedback with the correct final answer.Source: §D.3 · p. 20, Figure A3 · p. 20
  • When both models are incorrect, negative Softplus makes all displayed weights negative while preserving their ordering.The student falsely asserts that the first law is violated, and OPD assigns mixed-sign weights to that assertion.Source: §D.3 · p. 20, Figure A3 · p. 20
  • Both examples come from the first Qwen3 training rollout batch, using the initial teacher and student to compute token weights.Source: §D.3 · p. 20
Loading 2609.33711v1…