PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Policy Delta Distillation for Multilingual Math Reasoning

Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han

arXiv:2608.05802v1cs.CLcs.LG

TL;DR

Multilingual effectiveness of on-policy distillation, especially for Korean and Japanese reasoning, remains underexplored. This paper evaluates OPD and OPD² for multilingual mathematical reasoning and finds that OPD² consistently outperforms OPD while generally narrowing the English–Korean gap, although English-only training often shifts responses toward English.

  • Problem

    OPD effectiveness for multilingual reasoning, particularly in Korean and Japanese with a strong teacher model, remains largely unexplored.

  • Method

    The paper evaluates OPD and OPD² for mathematical reasoning across English, Korean, and Japanese, using the teacher–base-model probability gap as OPD²’s learning signal.

  • Results

    OPD² consistently outperforms OPD across multilingual settings, with particularly strong Korean and Japanese gains, and generally reduces the English–Korean performance gap.

  • Takeaways & Limitations

    English-only OPD can transfer reasoning accuracy to Korean and Japanese, but multilingual data remains important for preserving target-language responses.

Abstract

from arXiv · show

On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD$^2$), for mathematical reasoning in English, Korean, and Japanese. OPD$^2$ improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD$^2$ consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrows the English-Korean performance gap. We further find that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.

1 Introduction

This paper extends On-Policy Delta Distillation to multilingual mathematical reasoning in Korean and Japanese, testing whether it outperforms OPD and narrows language gaps. It also examines whether English-only OPD improves non-English benchmarks while preserving target-language reasoning and generation.

  • Motivation: OPD replaces RL’s sequence-level score with token-level generation probabilities obtained by evaluating the student’s rollout with a teacher model.This provides token-level supervision rather than a single score for the whole response.
  • Method: OPD2 uses the gap between the teacher and its base model as the learning signal, rather than the teacher–student logit-probability difference in OPD.This modification substantially improves performance across mathematical, scientific, and coding reasoning tasks in the Qwen3 model family.
  • Motivation: OPD remains underexplored with strong teacher models in multilingual reasoning, motivating evaluation beyond the predominantly English-focused literature.The paper specifically studies multilingual mathematical reasoning in Korean and Japanese.
  • Results: OPD2 consistently outperforms OPD in multilingual settings and tends to reduce English–non-English performance gaps, though not across every model and setting.The study also evaluates how English-only OPD affects non-English performance and target-language generation.

2 On-Policy Distillation

OPD replaces RL-style post-training with on-policy, token-level teacher supervision over responses sampled from the student’s current policy. OPD^2 modifies OPD’s reward using the gap between a post-trained teacher and its corresponding base model, with additional centering and conditioning components.

  • On-Policy Distillation: OPD samples a response from the student policy, then uses the teacher’s token-level probabilities to supervise the generated question–response pair.This preserves OPD’s on-policy nature while replacing the RL post-training process.
  • On-Policy Distillation: OPD computes KL-divergence only for sampled tokens, providing token-level supervision while limiting deviation from the student’s current generation distribution.The method adjusts relative probabilities among candidate tokens within that distribution.
  • On-Policy Distillation: Rt = log π∗(yt | x, y<t) − log πθ(yt | x, y<t), so OPD rewards tokens preferred more by the teacher and penalizes relatively less-preferred tokens.The reward compares teacher and student probabilities for each sampled token yt.
  • On-Policy Delta Distillation: OPD^2 defines its reward from the difference between the post-trained teacher and its corresponding base model, replacing the student term in the OPD reward.This delta signal targets changes associated with post-training and brings substantial improvements.
  • On-Policy Delta Distillation: OPD^2 adds reward centering to obtain the advantage and a conditioning criterion based on the original OPD signal to address the delta signal’s convergence-point issue.Both components are incorporated in all experiments, while their detailed formulations are omitted.

3 Experiments

Experiments with Qwen3 models show that OPD and especially OPD^2 improve multilingual mathematical reasoning across languages and model scales. OPD^2 narrows English–Korean performance gaps and transfers reasoning to Korean and Japanese, but multilingual training better preserves target-language responses than English-only training.

  • Experimental Setup: The study evaluates OPD and OPD^2 with Qwen3-1.7B and Qwen3-8B students, a Qwen3-30B-A3B-2507 teacher, and multilingual mathematical reasoning benchmarks.Benchmarks include PolyMath and Global-MGSM in English, Korean, and Japanese; HRM8K in English and Korean; and MAWPS in Japanese.
  • Multilingual Reasoning Performance: OPD^2 raises Qwen3-1.7B average scores from 55.1 to 63.6 in English, 40.9 to 51.9 in Korean, and 37.2 to 52.0 in Japanese.These averages are reported over PolyMath and Global-MGSM in non-thinking mode.
  • Multilingual Reasoning Performance: OPD^2 exceeds OPD by 3.1 and 4.0 points for Qwen3-1.7B in Korean and Japanese, versus 2.1 points in English.For Qwen3-8B, the corresponding Korean and Japanese gains are 3.3 and 3.1 points, while the English gain is 0.7 points.
  • Language Gap: OPD^2 reduces the English–Korean gap on six of seven reported benchmarks and preserves the remaining gap at 1.1 points.It reduces PolyMath and Global-MGSM gaps to 5.0 and 9.3 points, and the HRM8K-GSM8K gap from 12.1 to 9.2 points.
  • English-only versus Multilingual Training: English-only OPD^2 improves Korean and Japanese reasoning averages to 52.6 and 53.4 in non-thinking mode, comparable to multilingual OPD^2 scores of 51.9 and 52.0.However, target-language response rates fall from 90.5% to 48.3% in Korean and from 90.9% to 29.6% in Japanese.

4 Conclusion

OPD and OPD^2 improve multilingual mathematical reasoning across English, Korean, and Japanese, with OPD^2 consistently surpassing OPD. English-only training can transfer reasoning accuracy across languages, although the supplied passage truncates the conclusion’s qualification of these gains.

  • Both OPD methods improve multilingual mathematical reasoning across model sizes and generation modes.
  • OPD^2 consistently outperforms original OPD and generally reduces the English–Korean performance gap.
  • English-only training transfers reasoning accuracy to Korean and Japanese, indicating cross-language generalization from English supervision.

A Full Experimental Results

This section reports comprehensive multilingual benchmark results for Qwen3-1.7B and Qwen3-8B across thinking and non-thinking modes, with averages computed over each table’s listed benchmarks. Unless noted otherwise, OPD and OPD2 use balanced English-Korean-Japanese training, alongside an English-only ablation for Qwen3-1.7B.

  • Benchmark coverage: Complete results cover Qwen3-1.7B and Qwen3-8B in both thinking and non-thinking modes across language-specific benchmark tables.English and Korean use PolyMath, Global-MGSM, and five HRM8K subsets; Japanese uses PolyMath, Global-MGSM, and MAWPS.
  • Evaluation protocol: Average scores are computed over all benchmarks presented in each table.
  • Training setup: Unless otherwise specified, OPD and OPD2 train on balanced English, Korean, and Japanese questions at a 1:1:1 ratio.An English-only training ablation is additionally reported for Qwen3-1.7B.
  • Training setup: Base rows represent the original instruction-tuned models without additional OPD post-training.

A.1 Qwen3-1.7B with Multilingual Training

On multilingual Qwen3-1.7B training, OPD² consistently outperforms OPD, substantially improving non-thinking performance and raising overall averages across English, Korean, and Japanese.

  • Non-thinking performance: OPD² achieves the highest average non-thinking score in all three languages, improving base model averages from 47.6 to 65.7 in English, 37.4 to 56.5 in Korean, and 55.5 to 65.4 in Japanese.Both OPD and OPD² substantially improve non-thinking performance across all three languages.
  • Overall performance: 73.4 in English, 68.2 in Korean, and 71.2 in Japanese are OPD²’s overall averages, exceeding OPD’s 71.4, 66.2, and 70.5, respectively.Thinking-mode improvements are smaller because the base model already has strong reasoning performance.
  • Overall performance: OPD²’s consistent advantage over OPD is supported by its higher overall averages in English, Korean, and Japanese.The results show that OPD² improves the overall average across all three languages.

A.2 Qwen3-1.7B with English-Only Training

English-only training transfers effectively to Korean and Japanese in non-thinking mode, but its thinking-mode effects depend on the distillation objective. Despite strong benchmark gains, English-only models frequently respond in English on target-language inputs.

  • A.2 Qwen3-1.7B with English-Only Training: English-only OPD^2 raises Korean average score from 37.4 to 59.3 and Japanese average score from 55.5 to 66.5 in non-thinking mode.These scores are comparable to, and sometimes higher than, multilingual-training results.
  • A.2 Qwen3-1.7B with English-Only Training: In thinking mode, English-only OPD lowers Korean average score from 63.7 to 48.5, while OPD^2 reaches 65.1 and neither method improves English average performance.The results show that English-only post-training depends on both the distillation objective and generation mode.
  • A.2 Qwen3-1.7B with English-Only Training: English-only models frequently generate English text for Korean and Japanese PolyMath inputs, particularly after OPD^2 training.Strong benchmark accuracy under English-only training therefore does not necessarily imply target-language reasoning or responses.

A.3 Qwen3-8B with Multilingual Training

For Qwen3-8B with multilingual training, OPD^2 consistently outperforms original OPD across English, Korean, and Japanese. Its advantages appear in both non-thinking and thinking modes, including cases where original OPD reduces performance.

  • Non-thinking mode: In non-thinking mode, OPD^2 raises average scores from 58.7 to 75.3 in English, 55.4 to 72.2 in Korean, and 70.2 to 75.2 in Japanese.OPD^2 consistently outperforms the original OPD in all three languages.
  • Overall comparison: Across both modes, OPD^2 provides a more reliable post-training signal than original OPD, especially when the initial model already has strong reasoning capability.The results cover English, Korean, and Japanese under multilingual training.
  • Thinking mode: In thinking mode, original OPD lowers average scores from 80.9 to 79.9 in English and from 78.0 to 75.7 in Korean.These decreases occur relative to the strong base model.
  • Thinking mode: In thinking mode, OPD^2 improves the corresponding averages to 83.2 in English and 80.4 in Korean, while achieving the best Japanese average of 78.9.This contrasts with the occasional performance reductions from original OPD.
Loading 2608.05802v1…