SimOPD: Relational Knowledge Transfer for On-Policy Distillation

1 Introduction

• On-policy distillation uses student-generated responses with dense teacher feedback, whereas supervised fine-tuning relies on fixed demonstrations and reinforcement learning commonly uses sparse outcome rewards.The introduction notes that fixed demonstrations can create training–inference mismatch.Source: §1 · p. 1
• Token-probability supervision may not convey all teacher knowledge, while direct hidden-state matching complicates cross-architecture transfer and constrains student feature encoding.OPRD-Bridge addresses incompatible hidden spaces with a learned projection and separate calibration stage.Source: §1 · pp. 1–2
• SimOPD treats teacher-defined relationships among student-generated responses as supervision, allowing the student to preserve those similarities in its own representation space.This relational perspective draws on similarities and differences between instances in contrastive learning.Source: §1 · p. 2
• Paired queues store teacher and student representations of the same past responses, enabling each model to form similarity distributions in its own feature space for matching.The student adds each current representation pair to the queues for later comparisons.Source: §1 · p. 2
• 2.6 and 3.1 percentage points: SimOPD improves overall average over the strongest baseline in same- and cross-architecture settings, respectively.The reported experiments cover mathematical reasoning and code generation.Source: §1 · p. 2

2 Preliminary

• Given a prompt, the student samples a response from its policy, and teacher supervision is evaluated on that student-generated trajectory.The setup assumes teacher and student share a vocabulary.Source: §2 · p. 2

2.1 Output-Space On-Policy Distillation

• Output-space OPD supervises next-token predictions on the student’s generated prefixes, using reverse-KL training or token-level log-probability-difference rewards in an RL-style implementation.The policy gradient uses the corresponding token-level rewards.Source: §2.1 · p. 2

2.2 On-Policy Representation Distillation

• OPRD supervises hidden states on the same trajectories by comparing normalized student and teacher representations across supervised positions and layers.When hidden spaces are incompatible, OPRD-Bridge adds a calibration stage to make representations comparable.Source: §2.2 · pp. 2–3
• SimOPD instead computes teacher and student similarity distributions over paired historical responses, then minimizes their KL divergence.The paired memory banks let each model compute similarities in its own representation space.Source: Figure 2 · p. 3

3.1 Dual Memory Bank Construction

• Each response representation aggregates normalized hidden states across selected layers and response positions into one vector for historical comparison.The paired banks retain representations of these same responses for each model.Source: §3.1 · p. 3
• Paired banks store teacher and student representations of the same historical responses in the same order, enabling comparisons within each model’s own feature space without cross-model projection.The banks are initialized from the first training batch and updated together in first-in, first-out order.Source: §3.1 · p. 3

3.2 Similarity-Based Distillation

• For each student-generated response, SimOPD computes cosine similarities to stored representations in each model’s bank and converts them into distributions over the shared historical responses.The temperature-scaled distributions are comparable because corresponding bank entries refer to the same responses.Source: §3.2 · pp. 3–4
• Training initializes paired banks, generates on-policy responses, matches their similarity distributions when at least two bank entries exist, and then updates the banks.The stored representations remain fixed during each training update.Source: §3.2 · p. 4
• The student minimizes KL divergence from the teacher’s similarity distribution using prompts and responses sampled from its current policy.Teacher and stored representations remain fixed during optimization, while gradients flow through the current student representation.Source: §3.2 · p. 4

4.1 Setup

• The evaluation uses separate math and code training runs, with Avg@16 for mathematical reasoning and pass@1 for code generation.The models are Qwen3-4B → Qwen3-4B-Base for same-architecture distillation and Qwen3-8B → Qwen3-1.7B-Base for cross-architecture distillation.Source: §4.1 · p. 4

4.2 Main Result

• SimOPD ranks among the top two standalone methods in all four domain settings, leads in three, and achieves the highest standalone Overall score for both model pairs.Same-architecture Overall is 45.0, 2.0 points above ExOPD; cross-architecture Overall is 33.7, 1.5 points above OPRD-Bridge+OPD.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 7

4.3.1 Ablation Study

• Historical banks improve average scores over in-batch matching by 1.9 and 5.5 points, while similarity regression trails SimOPD by 1.5 and 0.7 points in same- and cross-architecture settings.A shared teacher bank also lowers the same-architecture average to 22.8.Source: §4.3.1 · p. 5, Table 3 · p. 8

4.2 Main Result

• Joint SimOPD+OPD exceeds the strongest baseline in Overall score by 2.6 points in same-architecture and 3.1 points in cross-architecture settings.The corresponding Overall scores are 45.6 and 35.3.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 7

4.3.2 Sensitivity Analysis

• Deeper teacher layers improve averages by 6.5 and 14.3 points from 25% to 100% depth; capacities of 1,024 remain within 0.9 and 0.5 points of 2,048.Scores vary by only 0.5 and 1.3 points across tested temperatures; expanding coverage from 512 to 2,000 tokens improves performance by 0.7 and 5.7 points, with 2,000-token pooling within 0.3 points of full-response pooling.Source: §4.3.2 · p. 6, Figure 3 · p. 7

4.3.3 Out-of-Domain Generalization

• On ARC-C, GPQA, and MMLU-Pro, SimOPD raises the average from 25.9 to 48.8 and exceeds PowerOPD by 1.2 points on average.It gains 4.1 points on ARC-C, matches PowerOPD on GPQA, and trails by 0.6 points on MMLU-Pro.Source: §4.3.3 · pp. 7–8, Table 4 · p. 8

5.1 On-Policy Distillation

• ExOPD, PowerOPD, and OPD² refine token-level supervision, but their supervision still comes from next-token probabilities.OPRD extends on-policy distillation to normalized hidden-state matching, with OPRD-Bridge adding calibrated projections for heterogeneous models.Source: §5.1 · p. 8
• SimOPD transfers similarities between responses through paired memory banks, comparing responses within each model’s own representation space without a learned cross-model projection.This contrasts with OPRD’s hidden-state matching and OPRD-Bridge’s calibrated projections for heterogeneous models.Source: §5.1 · p. 8

5.2 Representations Research in LLMs

• SimOPD applies similarity-based distillation to on-policy language generation, alongside representation-based methods for distillation, reinforcement learning, instance selection, and hallucination detection.Related examples include Similarity-Preserving Knowledge Distillation and CompRess (Tung and Mori, 2019; Abbasi Koohpayegani et al., 2020).Source: §5.2 · p. 8

6 Conclusion

• SimOPD’s paired memory banks match response-similarity distributions within each model’s own representation space, preserving student representational freedom and supporting different hidden dimensions without cross-model projection.Experiments show strong performance in same- and cross-architecture settings, with further gains from combining SimOPD with token-level OPD.Source: §6 · p. 8
• Ablations support the memory-bank design and distillation objective as components of the approach.The paper concludes that response similarities provide an effective supervision source for using teacher internal knowledge beyond token probabilities.Source: §6 · p. 8

• Evaluation primarily covers mathematical reasoning and code generation; applicability to open-ended generation and other modalities remains to be investigated.Source: Limitations · p. 9
• Representation-based relational distillation adds hidden-state processing and cross-response similarity computation, limiting training efficiency in the current implementation.Source: Limitations · p. 9
• Memory-bank initialization, warm-up schedules, update strategies, and additional regularization have not been systematically examined.Source: Limitations · p. 9

A Additional Implementation Details

• Across both domains, training runs for 200 steps on eight NVIDIA A100 80GB GPUs using AdamW at a learning rate of 10−5.Each batch has eight prompts with two responses per prompt, and the global batch size is 16 responses.Source: A Additional Implementation Details · p. 11
• Prompts longer than 3,072 tokens are filtered, and rollout generation uses top-p = 1.0 with a maximum response length of 4,096 tokens.Responses are sampled at temperature 1.0 using non-thinking prompting.Source: A Additional Implementation Details · p. 11
• Baseline methods follow their original formulations and model-role definitions, adapted to common teacher–student pairs, and all train for 200 steps without learning-rate warm-up.The reported configurations include ExOPD λ = 1.25, PowerOPD α = 5.0, and rank-8 projection for OPRD-Bridge.Source: A Additional Implementation Details · p. 11

B MBPP Error Analysis

• The error breakdown separates incorrect MBPP responses into format errors and execution failures, with execution failures covering remaining incorrect responses.Source: B MBPP Error Analysis · p. 11
• Compared with OPD, SimOPD incurs 9.1 percentage points more format errors but 4.8 points fewer execution failures, resulting in a 4.3-point accuracy deficit.Source: B MBPP Error Analysis · p. 11
• Adding OPD reduces MBPP format errors from 23.6% to 13.2% and raises accuracy by 7.5 points.Source: B MBPP Error Analysis · p. 11

C Training Efficiency.

• SimOPD improves the mathematical reasoning average over OPRD by 4.4 percentage points, with 87.20 seconds per step and a 5.0% increase in training time.Source: C Training Efficiency · p. 11
• Peak memory reaches 78.77 GB per GPU, within the baseline range of 76.39–79.69 GB.Measurements use matched hardware, batch sizes, and response-length limits, excluding five warm-up steps, evaluation, and checkpoint saving.Source: C Training Efficiency · p. 11
• Memory banks are used only during training, leaving the student’s architecture and inference procedure uninfluenced.Source: C Training Efficiency · p. 11