MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate

1 Introduction

• OPD is limited by single-teacher capability, underexplored agentic tasks with compounding per-step errors, and task-agnostic divergence selection.The introduction notes that existing stabilization attempts address individual failure modes without connecting divergence choice to task structure.Source: §1 · pp. 1–2
• Existing multi-teacher distillation is off-policy, so students consume precomputed teacher signals rather than live reactions to their own rollouts.Prior work motivates collaboration and debate as sources of collective intelligence, including confidence-weighted consensus that can match or exceed the strongest individual model.Source: §1 · p. 2
• MAD-OPD uses confidence-weighted debate for token-level on-policy supervision; OPAD adds step-level interaction, and the divergence principle selects JSD for agentic tasks and reverse KL for code.The proposed framework brings teacher debate into the OPD loop; OPAD is compatible with both single-teacher OPD and MAD-OPD.Source: §1 · p. 2

On-Policy Distillation

• OPD is presented as dense on-policy distillation, later adopted in frontier-model post-training and extended with privileged information and objective-level stabilization.The cited prior methods rely on a single teacher and task-agnostic divergence, unlike MAD-OPD’s debate-generated privileged information and task-adaptive divergence.Source: §2 · pp. 2–3

Multi-Agent Systems

• Multi-agent debate uses iterative argumentation to produce collective intelligence, but prior debate methods consume outputs at inference rather than train with them.MAD-OPD brings debate inside the on-policy training loop as token-level supervision.Source: §2 · p. 3

Multi-Teacher Distillation

• Prior multi-teacher distillation aggregates independent teacher outputs or interaction traces without inter-teacher interaction; MAD-OPD adds on-policy debate as the supervision source..Source: §2 · p. 3

3.1 On-Policy Distillation as Reward Design

• OPD minimizes divergence between teacher and student distributions along student-generated trajectories, and that divergence determines the dense token-level reward.The teacher may condition on privileged information unavailable to the student; in this setting, that information is the debate transcript.Source: §3.1 · p. 3

3.2 Theoretical Properties of Divergences

• For symmetric JSD, loss is bounded by log 2 and the student-logit gradient is uniformly bounded by 2; reverse-KL gradient descent concentrates on the dominant mode of a disjoint-support mixture.The reverse-KL result gives the stationary-point cost as -log α_j∗ for the dominant mode with mixture weight α_j∗.Source: §3.2 · p. 4

3.3 Task-Adaptive Divergence Principle

• Under either single-turn mean or OPAD’s sum-of-per-step-means, JSD’s per-token gradient bound remains 2, independent of trajectory length and teacher–student gap.The analysis frames agentic trajectories as vulnerable to cascading step errors and identifies the privileged p–q gap as especially sharp in this setting.Source: §3.3 · p. 4

2. Reverse KL: ∥∇ztD←

• When the teacher assigns near-zero probability to a student-sampled token, the stated KL gradient term diverges.The passage identifies this as the regime created by privileged supervision.Source: §2 · p. 4

• Code generation can yield a mixture of multiple correct implementations.Source: §3 · p. 5
• The analysis assumes disjoint-support code modes and a dominant mode consistent across positions.Source: §3 · p. 5
• Under that assumption, reverse KL concentrates the student on the dominant mode, avoiding incoherent splicing.Source: §3 · p. 5

• Forward KL averages incompatible code structures across modes.Source: §2 · p. 5
• JSD partially concentrates on the dominant mode but retains non-negligible probability on secondary modes.Source: §2 · p. 5
• The task-adaptive principle selects JSD for multi-step agentic tasks to bound per-position gradients and support stable long-trajectory training.Source: §2 · p. 5
• For code generation, the principle selects reverse KL to concentrate probability on a single coherent implementation path.Source: §2 · p. 5
• The paper reports that empirical evaluation validates both divergence predictions.Source: §2 · p. 5

4 Method: MAD-OPD

• MAD-OPD translates task-adaptive divergence selection into a pipeline that generates privileged information, weights teachers by post-debate confidence, and applies the resulting loss.Source: §4 · p. 5

4.1 Multi-Agent Debate as Privileged Information Generation

• Teachers independently respond at the first debate round, then revise after reading earlier responses; the resulting debate history is visible to teachers but hidden from the student.Source: §4.1 · p. 5

4.2 Confidence-Based Dynamic Teacher Weighting

• Each teacher self-reports confidence after the debate, and softmax-normalized scores weight its contribution according to post-deliberation certainty.Source: §4.2 · p. 6

4.3 Token-Level Distillation Objective

• Teachers force-decode the student’s sampled tokens with the debate transcript, while the student processes those tokens without it; confidence-weighted divergences update only the student.Source: §4.3 · p. 6

4.4 On-Policy Agentic Distillation (OPAD)

• OPAD sums per-step losses over the student’s on-policy trajectory, with debate conditioned on actual prior observations so supervision adapts to the generated trajectory.Source: §4.4 · pp. 6–7

5.1 Experimental Setup

• The evaluation covers Qwen3 and Qwen3.5 configurations, with students from 1.7B to 14B, across three agentic and two code benchmarks.Source: §5.1 · p. 7, §5 · p. 7

5.2 Main Results

• MAD-OPD ranks first on overall average across all six configurations, outperforming the strongest single-teacher OPD.Source: §5.2 · p. 7

5.3 Scaling Behavior

• MAD-OPD extends multi-teacher gains on agentic average to all six configurations; gains also depend on student–teacher capability gaps and teacher strength.Source: §5.3 · p. 8, Figure 3 · p. 9

5.4 Component and Divergence Analysis

• Ablations show each component contributes independently; R=2 is optimal, while R=3 overshoots due to prompt-context bloat.Source: §5.4 · pp. 8–9, Figure 4 · p. 9
• A controlled ablation finds JSD leads on agentic tasks and reverse KL on code, while forward KL trails on both.Source: §5.4 · p. 9

6 Discussion and Conclusion

• MAD-OPD ranks first in all six configurations; the 4B student exceeds its 14B teacher on LCB-v6 (+4.26% pass@1, +10.29% BoN@16).The authors suggest teacher-pool diversity is a bottleneck relative to single-teacher capability, and identify richer debate, cross-vocabulary distillation, and longer-horizon OPAD as future work. The method requires teacher token-level distributions and a shared vocabulary; training cost scales with K × R teacher forward passes per step.Source: §6 · pp. 9–10

A Algorithm Pseudocode

• The algorithm samples student actions on-policy, has teachers deliberate on the current state, then evaluates the sampled action through privileged force-decoding.It covers single-step code generation (M=1) and multi-step OPAD agentic tasks (M>1).Source: A Algorithm Pseudocode · p. 12

B Experimental Details

• The appendix outlines the training pipeline in Sections B.1–B.7 and per-benchmark evaluation details for five benchmarks in Sections B.8–B.12.The appendix describes the complete training and evaluation setup.Source: B Experimental Details · p. 12

B.1 Training Configuration

• All configurations use non-thinking mode, AdamW, a 1×10−5 learning rate, batch size 128, and one training epoch, with best checkpoints selected by overall benchmark performance.Agentic and code generations have maximum sequence lengths of 4096 and 16384 tokens, respectively.Source: §B.1 · p. 12

B.2 Infrastructure

• Experiments use 8× NVIDIA H20 GPUs; teacher serving separates debate-text generation from full-vocabulary logit extraction.A 4B-student agentic run takes approximately 16 hours, while code-task training takes approximately 32 hours.Source: §B.2 · p. 12

B.3 Training Data

• Agentic training uses approximately 16K step-level ToolACE samples, while code training uses 30K OpenThoughts3 problems and separate task checkpoints.For agentic tasks, teachers debate on each student state and force-decode its sampled action; code examples are single on-policy completions.Source: §B.3 · p. 13

B.4 Baseline Setup

• The baselines include the undistilled student, stronger-teacher single-teacher OPD, uniform-weight independent multi-teacher OPD, and 7:3 routed multi-teacher sequence distillation.MT-OPD has no inter-teacher debate; MT-SeqKD trains with sequence-level cross-entropy on routed teacher outputs.Source: §B.4 · p. 13

B.5 Debate Configuration

• Debates use two teachers and two rounds, with round-two revisions informed by round-one responses and final confidence scores converted into teacher weights.The debate temperature is 0.7, confidence softmax temperature is 1.0, and the appendix reports two rounds as the optimal tested setting.Source: §B.5 · p. 13

B.6 Evaluation Protocol

• Checkpoints are selected on BFCL-v4 for agentic tasks and LCB-v6 for code, then evaluated once on all five benchmarks; τ 2-Bench, VitaBench, and MBPP+ are held out from selection.The same protocol applies to all methods; agentic checkpoints are checked every 10 steps and code checkpoints every 5 steps.Source: §B.6 · p. 13

B.7 Hyperparameter Summary

• The task-dependent hyperparameters use JSD0.5 for agentic tasks, reverse KL for code, and maximum sequence lengths of 4096 and 16384 tokens, respectively.Evaluation decoding uses temperature 0.7, top-p 0.8, and presence penalty 1.2.Source: §B.7 · p. 14

B.8–B.10 Benchmark Details

• BFCL-v4 averages AST-checked accuracy across eight subtasks, τ 2-Bench averages pass@1 across three domains, and VitaBench averages success across three scenarios.τ 2-Bench scores exact-match database state; VitaBench uses a rubric-based sliding-window scorer.Source: §B.8 · p. 14, §B.9 · p. 15, §B.10 · p. 15

B.11–B.12 Code Benchmarks

• LCB-v6 uses post-cutoff problems with hidden tests and reports pass@1 and BoN@16, while MBPP+ augments 378 sanitized problems with approximately 5× more tests per problem.The augmented MBPP+ tests are intended to catch correctness failures missed by the original test suite.Source: §B.11 · p. 15, §B.12 · p. 16

C.1 Component Analysis: Full Numbers (RQ3)

• +1.74% Ag-Avg and +1.52% Co-Avg over uniform weighting accompany confidence weighting, which depends on relative confidence ordering rather than calibrated absolute scores.If confidence is uninformative, the weighting reduces to uniform MT-OPD; a systematic per-step calibration study is left for future work.Source: §C.1 · p. 16

C.2 Divergence Ablation: Full Numbers (RQ4)

• JSD leads on agentic tasks and reverse KL leads on code, matching the task-adaptive divergence prediction in the 14B+8B→4B ablation.The default JSD-agentic and reverse-KL-code rows match the main MAD-OPD configuration; other rows isolate non-default divergences.Source: §C.2 · p. 16, Table 4 · p. 17

C.3 Student Outperforms Teachers on LCB-v6 (RQ1)

• 29.83% pass@1 exceeds the 25.57% 14B-teacher ceiling by +4.26%, and 43.43% BoN@16 exceeds 33.14% by +10.29%.The 4B student uses 3.5× fewer parameters than the 14B teacher, with token cost on par with MT-OPD and below MT-SeqKD’s regime.Source: Figure 5 · p. 17, §C.3 · p. 17
• The 4B MAD-OPD student also exceeds both teachers under pass@1 and BoN@16, while other distillation baselines do not cross the 14B-teacher pass@1 ceiling.On pass@1, OPD scores 25.00%, MT-OPD 23.29%, and MT-SeqKD 23.18%.Source: §C.3 · p. 17, Figure 5 · p. 17

C.4 Best-of-N Results for Code Benchmarks (RQ1)

• MAD-OPD achieves best or tied-best LCB-v6 BoN@16 in 5 of 6 configurations and ranks first on MBPP+ in all six.MT-SeqKD leads only on 32B+30B→8B and ties MAD-OPD on Qwen3.5 27B+9B→4B for LCB-v6.Source: Table 5 · p. 18, §C.4 · p. 18

C.5 Training Loss Stability under Different Divergences (RQ4)

• Agentic reverse KL shows high-variance loss spikes, whereas JSD remains stable; for code, reverse KL converges smoothly while JSD can retain secondary modes.The paper relates the agentic spikes to unbounded per-token logit gradients and code behavior to reverse KL’s mode-seeking behavior.Source: §C.5 · p. 18, Figure 6 · p. 19

C.7 Teacher Base Performance Reference

• The teacher-base table supplies the single-teacher reference across all five benchmarks, and the Qwen3.5 family leaves smaller per-method headroom on its configuration.These undistilled teacher scores serve as the single-teacher upper bound that OPD attains and MAD-OPD aims to exceed.Source: §C.7 · p. 18, Table 6 · p. 19

C.6 Effect of Debate Rounds R (RQ3)

• R=2 is the empirical optimum across all five benchmarks, while R=3 lowers Ag-Avg by −8.27% relative to R=2 as extra prompt context introduces noise.R=0 removes teacher generation; R=1 corresponds to independent-only MT-OPD; R≥2 activates inter-teacher revision.Source: §C.6 · p. 18, Figure 7 · p. 20

D.1 OPD Gradient Derivation

• Under teacher-forcing, the OPD gradient depends on student probabilities while the privileged teacher distribution is stop-gradient, removing the REINFORCE score-function term.The resulting standard cross-entropy gradient is q−p.Source: §D.1 · p. 20

D.2 Proof of Lemma 1 (JSD Bounded Loss and Bounded Logit Gradient)

• For any distributions p and q, JSDβ is bounded between 0 and H(β), with per-logit gradient infinity norm at most 2.The appendix also states this per-position bound carries over to the OPAD loss regardless of trajectory length.Source: §D.2 · p. 20, §D.4 · p. 23

D.3 Proof of Lemma 2 (Reverse KL Mode Concentration)

• With disjoint-support modes, reverse-KL minimization concentrates on the dominant mixture mode, and gradients to recover secondary modes vanish as their student probabilities approach zero.The optimal cost is −log αj* for the dominant mode j* = arg maxj αj.Source: §D.3 · p. 22

D.4 Proof of Proposition 1 (Per-Token Gradient Stability)

• JSD has a uniform per-token gradient bound, reverse KL is unbounded under privileged p–q gaps, and forward KL is bounded but mode-covering.The reverse-KL gradient can diverge when the privileged teacher assigns near-zero probability to a student-sampled token.Source: §D.4 · pp. 22–23

D.5 Proof of Proposition 2 (Reverse KL for Coherent Code Generation)

• Reverse KL concentrates probability on the dominant implementation mode, avoiding incoherent splices between distinct code paths.The analysis models each teacher mode as a distinct correct implementation with disjoint support.Source: §D.5 · p. 23

D.6 Supplementary: Forward KL Mode-Covering Instability

• Forward KL’s infinite penalty for missing teacher modes can make sequential token sampling interpolate across implementation paths and produce incoherent code.For multi-step tool use, mixing strategies can pair a tool name from one strategy with another strategy’s argument schema, derailing trajectories.Source: §D.5 · p. 23, §D.6 · p. 24

D.5 Proof of Proposition 2 (Reverse KL for Coherent Code Generation)

• JSD’s mixture removes reverse KL’s infinite inter-mode penalty and retains secondary-mode mass, which suits tasks with multiple valid action sequences but not single-path code generation.The passage identifies this distinction as adequate for agentic tasks and suboptimal when code requires commitment to one implementation.Source: §D.5 · p. 23

D.6 Supplementary: Forward KL Mode-Covering Instability

• Forward KL is bounded but mode-covering can produce incompatible rollouts, whereas reverse KL’s code-side mode concentration avoids splicing.The comparison distinguishes geometric instability from numerical gradient spikes.Source: §D.5 · p. 23, §D.6 · p. 24

E Case Study: Emergent Error Correction via Multi-Agent Debate

• The case study presents a worked ToolACE example in which a single 14B teacher fails but a two-round debate with an 8B teacher yields the correct answer.The authors describe it as illustrative rather than calibration and state that the complementary-error pattern is observed across the ToolACE training set.Source: E Case Study: Emergent Error… · p. 24
• The two teachers correct complementary date and duration errors through debate, with both converging to the ground truth after revision.The 14B teacher adopts the correct November dates while retaining its correct 14-day duration; the final confidence scores are 95 and 90.Source: E Case Study: Emergent Error… · pp. 25–26, Table 7 · p. 26
• MAD-OPD concentrates high divergence on the student’s erroneous month token, providing a corrective signal absent when the single teacher shares the student’s error.The figure encodes divergence by color intensity, from low agreement to high corrective gradient.Source: E Case Study: Emergent Error… · p. 26, Figure 8 · p. 27
• The debate trace provides richer training signal than either teacher’s first-round output alone, including the reasoning pattern of error detection and self-correction.The paper attributes this transfer to privileged force-decoding.Source: E Case Study: Emergent Error… · p. 26

F Debate Prompt Templates

• The prompt protocol uses independent first-round responses, then gives each teacher both responses as shared context for revision without requiring agreement.Prompts are instantiated for agentic and code tasks; the revised debate transcript serves as privileged context during force-decoding.Source: F Debate Prompt Templates · p. 26, §F.1 · p. 27, §F.2 · p. 27

F.3 Confidence Scoring Suffix

• When dynamic confidence weighting is enabled, prompts request a confidence score, and final-round scores are normalized into teacher weights.In the case study, the 14B teacher’s confidence falls from 90 to 85 in Round 1 and rises to 95 after correcting its error.Source: E Case Study: Emergent Error… · p. 26, §F.3 · p. 28

G Broader Impact

• MAD-OPD is presented as reducing the parameter count needed for a given agentic and code performance level, with corresponding reductions in deployment GPU and energy footprint.The broader-impact discussion also describes a 4B student exceeding its 14B teacher on LCB-v6 at competitive token cost.Source: G Broader Impact · p. 28