PaperlayerView exact revision on arXiv

Source-linked AI summary

Crosslingual On-Policy Self-Distillation for Multilingual Reasoning

Yihong Liu, Raoyuan Zhao, Michael A. Hedderich, Hinrich Schütze

arXiv:2605.09548v1cs.CL

TL;DR

Low-resource languages receive less reasoning supervision, limiting access to models’ mathematical problem-solving ability. COPSD transfers high-resource reasoning through privileged crosslingual teacher context and consistently improves performance across 17 African languages and model sizes.

  • Problem

    Low-resource languages receive limited high-quality reasoning supervision, restricting access to models’ latent mathematical problem-solving ability.

  • Method

    COPSD uses a shared student-teacher model, giving the student only the low-resource problem while the teacher receives English translations and reference solutions for dense supervision.

  • Results

    COPSD consistently improves low-resource reasoning across model sizes and languages, including Qwen3-1.7B average Pass@12 rising from 9.11 to 15.53.

  • Takeaways & Limitations

    Across 17 African languages, COPSD outperforms GRPO and improves format adherence, test-time scaling, and generalization to harder multilingual benchmarks.

  • Takeaways & Limitations

    COPSD depends on English reference solutions, limiting applicability where high-quality English supervision is unavailable or another language would provide better reasoning signals.

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable progress in mathematical reasoning, but this ability is not equally accessible across languages. Especially low-resource languages exhibit much lower reasoning performance. To address this, we propose Crosslingual On-Policy Self-Distillation (COPSD), which transfers a model's own high-resource reasoning behavior to low-resource languages. COPSD uses the same model as student and teacher: the student sees only the low-resource problem, while the teacher receives privileged crosslingual context, including the problem translation and reference solution in English. Training minimizes full-distribution token-level divergence on the student's own rollouts, providing dense supervision while avoiding the sparsity and instability of outcome-only reinforcement learning (RL). Experiments on 17 low-resource African languages show that COPSD consistently improves low-resource mathematical reasoning across model sizes and substantially outperforms Group Relative Policy Optimization (GRPO). Further analyses show that COPSD improves answer-format adherence, strengthens test-time scaling, and generalizes to harder multilingual reasoning benchmarks, with especially large gains for lower-resource languages. We make our code and data available at: https://github.com/cisnlp/COPSD.

1 Introduction

Low-resource languages remain disadvantaged in mathematical reasoning because models have limited exposure to multilingual reasoning supervision. COPSD addresses this gap through crosslingual on-policy self-distillation, consistently improving performance over base models and substantially outperforming GRPO across languages and model sizes.

  • 1 Introduction: COPSD consistently outperforms base and GRPO-trained models across 17 low-resource African languages, with gains generalizing to harder multilingual reasoning settings and especially lower-resource languages.The evaluation covers Qwen3 models at 1.7B, 4B, and 8B scales on AfriMGSM, plus 8 languages from PolyMath.
  • 1 Introduction: Models often fail to access latent problem-solving ability in low-resource languages because those languages receive limited pretraining exposure and high-quality reasoning supervision.This creates a multilingual gap even when the model may possess the underlying ability to solve the problem.
  • 1 Introduction: Directly translating English reasoning for supervised fine-tuning can introduce mathematical and logical errors while creating train-inference distribution mismatch.Outcome-based reinforcement learning is presented as another alternative, motivating denser supervision from the model’s own trajectories.
  • 1 Introduction: COPSD uses one model as student and teacher under different contexts, transferring high-resource reasoning behavior to low-resource languages through on-policy self-distillation.The framework treats high-resource language context as privileged information while learning from the model’s generated trajectories.
  • 1 Introduction: COPSD converges rapidly, improves answer-format adherence, and helps models exploit larger test-time generation budgets.The method’s reported benefits extend beyond aggregate accuracy to training dynamics and inference-time scaling.

2 Related Work

Prior work positions on-policy distillation as an alternative to supervised fine-tuning and outcome-based reinforcement learning for improving LLM reasoning, while multilingual reasoning research documents substantial crosslingual gaps. This work extends OPSD to transfer English-accessible reasoning behavior to low-resource languages.

  • On-Policy Distillation: OPD has emerged as an effective alternative to SFT and outcome-based RL for improving LLM reasoning (Gu et al., 2024; Agarwal et al., 2024; Lu and Lab, 2025; Yang et al., 2026).
  • On-Policy Distillation: COPSD extends OPSD to multilingual settings by transferring English-accessible reasoning behavior to low-resource languages.The approach is presented as an effective, novel method for improving low-resource reasoning.
  • Multilingual Reasoning: Multilingual reasoning evaluates whether language models solve reasoning problems consistently across languages rather than relying primarily on English or other high-resource languages (Ghosh et al., 2025).Prior work reports substantial crosslingual performance gaps, especially in low-resource languages, and inconsistent or language-mixed reasoning traces (Tam et al., 2025; Zhao et al., 2026a; Liu et al., 2026; Ki et al., 2026).

3 Preliminary: On-Policy Self-Distillation

On-Policy Self-Distillation (OPSD) uses one model as both student and teacher under different conditioning contexts, with the student trained on its own trajectories against privileged teacher distributions. This reduces train-test mismatch and provides dense intermediate-step supervision without an external teacher or outcome-only rewards.

  • Framework: OPSD uses the same model as student and teacher, with the student seeing only the problem and the teacher additionally receiving privileged information such as a reference solution.The teacher’s privileged context is intended to provide a stronger learning signal by enabling better problem rationalization. OPSD avoids requiring a separate teacher model (Zhao et al., 2026b; Zhang et al., 2026a).
  • On-policy training: OPSD samples on-policy trajectories from the student, then has both policies evaluate the same student-generated prefixes at each decoding step.This preserves inference-time student behavior during training while allowing teacher-conditioned evaluation of the rollout.
  • Objective: Training minimizes trajectory-averaged token-level divergence between teacher and student distributions, with gradients flowing only through the student.The divergence can be instantiated as KL divergence, while the teacher remains a fixed distributional target conditioned on privileged information.
  • Motivation: OPSD reduces train-test mismatch relative to SFT or off-policy distillation and supplies dense intermediate-step feedback instead of sparse final-answer rewards from outcome-based RL.Its advantages come from combining on-policy student rollouts, privileged information, and teacher supervision throughout reasoning trajectories.
  • COPSD overview: COPSD applies this framework by giving the student a low-resource translation while the teacher evaluates the rollout using privileged English context and a reference solution.Per-token divergence transfers English-accessible reasoning behavior to low-resource-language reasoning.

4 Methodology

COPSD extends on-policy self-distillation to multilingual reasoning by using English context as privileged information for a teacher while the student sees only the low-resource problem. It trains the student on its own rollouts by matching the teacher’s token-level distributions, preserving the inference-time condition.

  • COPSD overview: COPSD transfers high-resource reasoning behavior by giving the teacher English context while restricting the student to the low-resource problem.Both policies are instantiated from the same language model, but only the teacher receives the translated problem and reference solution.
  • Problem formulation: English serves as the high-resource language because common LLM post-training is English-centric (Shaham et al., 2024; Dang et al., 2024).The formulation pairs each low-resource problem x(L) with its English counterpart x(H) and reference solution y∗.
  • Training objective: COPSD minimizes token-level divergence between teacher and student distributions on the student’s own on-policy reasoning trajectory.The policies evaluate the same student-generated prefix at each step, with D instantiated as a distributional divergence such as KL divergence.
  • Training objective: Gradients flow only through the student policy, improving reasoning in the low-resource language while the teacher supplies privileged supervision.The student therefore matches the inference-time condition, whereas the teacher can induce more reliable reasoning behavior from additional crosslingual information.
  • Training objective: The explicit reasoning language of both policies is controlled to match the student input language during training.This language-control condition is referenced in §5.2.

5 Experiments

Experiments on Qwen3 models across three sizes and 17 African languages show that COPSD consistently improves low-resource mathematical reasoning, outperforming GRPO and benefiting smaller models most. Evaluation uses AfriMGSM Pass@12, with PolyMath included as a harder multilingual benchmark for further evaluation.

  • Setup: The experiments use Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, whose post-training data are dominated by high-resource languages such as English.Qwen3 models are pretrained on multilingual corpora and further trained with supervised fine-tuning and reinforcement learning.
  • Data and benchmarks: Training uses 0.5K OpenThoughts math problems translated into 17 low-resource African languages, with English questions and solutions as teacher context and translations as student inputs.The primary benchmark is AfriMGSM, covering 17 African languages with 250 problems per language; PolyMath provides a harder follow-up evaluation with 125 questions per language.
  • Evaluation: Performance is measured with Pass@12 by sampling 12 responses per problem, extracting answers enclosed in \boxed{}, and verifying them against gold answers with Math-Verify.The study controls reasoning language by inserting a language-specific prefix after the <think> token during training and inference.
  • Results: COPSD achieves the best average Pass@12 for every Qwen3 size, improving Qwen3-1.7B from 9.11 to 15.53, Qwen3-4B from 19.20 to 20.61, and Qwen3-8B from 19.41 to 23.55.The smaller model gains over 70% relative improvement in average Pass@12 over its base model, with improvements on nearly every language.
  • Results: GRPO provides only marginal low-resource gains, while COPSD offers denser and more reliable supervision when correct reasoning trajectories are rarely sampled.For Qwen3-1.7B, GRPO increases Pass@12 from 9.11 to 9.18, and it underperforms the base model in several languages.

6 Complementary Analysis

Complementary analyses show that COPSD learns rapidly, improves format adherence and test-time scaling, reduces repetitive degeneration, and generalizes to harder multilingual reasoning tasks, with especially large gains for lower-resource languages.

  • Training dynamics: COPSD improves Pass@12 and format rate within early training steps, whereas GRPO shows no clear improvement trend.Qwen3-4B and Qwen3-8B reach their best performance within only a few gradient updates before gradually declining, consistent with rapid absorption of the dense teacher signal.
  • Format adherence: Pass@12 is strongly associated with answer-format adherence, with mean per-language Pearson correlations of 0.628, 0.838, and 0.728 for Qwen3-1.7B, Qwen3-4B, and Qwen3-8B.The pooled correlations are lower but remain positive.
  • Test-time scaling: COPSD achieves the highest average Pass@12 across model sizes and generation budgets, amplifying longer-budget gains most strongly for Qwen3-8B.Qwen3-8B COPSD improves 30.0% from 1,024 to 4,096 tokens, compared with 13.8% for GRPO.
  • Repetitive degeneration: COPSD consistently maintains the lowest 4-gram repeat rate during training, reducing repetitive loops and redundant reasoning fragments relative to the base model and GRPO.Repetition is a recognized multilingual reasoning failure mode, particularly in low-resource languages (Barua et al., 2026; Tran et al., 2025).
  • Generalization to harder reasoning: COPSD generalizes to harder PolyMath reasoning tasks, improving over the base model across almost all languages and yielding especially large gains for lower-resource languages.On medium difficulty, gains reach +32.0 Pass@12 points for Swahili and +32.8 for Telugu, compared with +15.2 for Bengali.

7 Conclusion

The conclusion presents COPSD as a framework for improving multilingual mathematical reasoning, especially in low-resource languages, by using privileged English context for dense token-level supervision. It further reports improved format adherence, rapid convergence, stronger test-time scaling, and generalization to harder multilingual benchmarks, suggesting that language access and expression partly constrain latent reasoning ability.

  • Conclusion: COPSD improves multilingual mathematical reasoning with a focus on low-resource languages by using privileged English questions and reference solutions during teacher supervision.The student reasons from the low-resource problem alone, while a same-model teacher provides dense token-level supervision.
  • Conclusion: COPSD improves format adherence, converges rapidly, strengthens test-time scaling, and generalizes to harder multilingual reasoning benchmarks.
  • Conclusion: The results suggest that low-resource reasoning failures partly reflect difficulty accessing and expressing latent reasoning ability through underrepresented languages, positioning COPSD as an effective path toward more multilingual reasoning models.

Limitations

Although COPSD consistently improves over baselines across languages, it remains limited by its reliance on English supervision, translated problem statements, and an imperfect same-model teacher for weaker target languages.

  • Despite these limitations, COPSD consistently improves over baselines across languages, motivating future work to address its remaining constraints.
  • COPSD relies on English privileged context and reference solutions, limiting applicability where English supervision is unavailable or another high-resource language would be better.The method assumes access to English reasoning supervision rather than allowing an arbitrary privileged language.
  • Translated training questions may contain artifacts that affect training quality and downstream performance, even though COPSD does not require translated reasoning traces.
  • Using the same model as teacher can yield imperfect guidance for languages where its competence is limited, causing saturation or degradation with continued training.This behavior was observed for some languages and model sizes, despite English context and reference solutions.

Ethical Considerations

The authors used ChatGPT for language polishing and minor code implementation support, while retaining responsibility for all technical contributions, experimental design choices, and final decisions.

  • The authors used ChatGPT for language polishing and minor code implementation support [7], but made all technical contributions, experimental design choices, and final decisions themselves.

A Experimental Details · A.1 Language coverage · A.2 Language Control

The experiments cover all 17 AfriMGSM African languages using ISO 639-3 identifiers across diverse language families and writing systems. Language-specific instructions and prompt-hacking prefixes steer explicit reasoning toward the target language and require boxed final answers.

  • A.1 Language coverage: Experiments cover all 17 African languages in AfriMGSM, spanning multiple language families and writing systems.This scope tests COPSD across substantially different linguistic and orthographic conditions.
  • A.1 Language coverage: ISO 639-3 codes serve as language identifiers throughout training, evaluation, and result reporting.
  • A.2 Language Control: The experiments use complementary prompting strategies to encourage explicit reasoning traces in each target low-resource language.This approach follows Qi et al. (2025) and Zhao et al. (2026a).
  • A.2 Language Control: Target-language prompts ask models to reason step by step and place the final answer inside \boxed{}.The language-specific instructions are provided for each target language.
  • A.2 Language Control: Explicit language instructions may still permit English switching or mixed-language reasoning, complicating comparisons of reasoning behavior.This limitation is documented in prior multilingual reasoning and language-mixing work (Wang et al., 2025a; Qi et al., 2025; Zhao et al., 2026a).
  • A.2 Language Control: Prompt-hacking prefixes inserted immediately after <think> help anchor reasoning in the target language until </think>.The prefixes include target-language statements such as beginning to think in Swahili.

B Complete Results · C Prompt Template · D Environment and Hyperparameters

Complete results show that COPSD generally improves reasoning, formatting, test-time scaling, and repetition across languages, while prompt templates and training configurations define its crosslingual supervision and evaluation setup. Effects vary with language and model scale, with larger models benefiting more consistently from increased generation budgets.

  • B Complete Results: COPSD typically improves Pass@12 and format rate early in training, whereas GRPO often has flatter or more unstable trajectories across languages and model sizes.Language-level effectiveness and saturation depend on model scale and target-language generation quality.
  • B Complete Results: COPSD consistently achieves lower repeat rates than both the base model and GRPO across model scales and n-gram granularities from n = 1 to 6.This indicates a robust reduction in repetition in low-resource reasoning.
  • C.1 Translation Prompt: The translation prompt converts English mathematical problems into 17 low-resource African languages while preserving mathematical content, numbers, and LATEX expressions.Gemini-3-Flash translates only the problem text.
  • C.2 Student-Policy Prompt: The student-policy prompt provides only the low-resource problem and instructs the model to reason step by step in the target language.The prompt uses language-specific instructions and is illustrated with Swahili.
  • C.3 Teacher-Policy Prompt: The teacher-policy prompt supplies the target-language problem, its English translation, and an English reference solution before asking for a target-language solution.The same structure is adapted across languages with language-specific instructions, labels, and reasoning prefixes.
  • D Environment and Hyperparameters: Training uses separate models for each of 17 languages and three scales, totaling 51 models, with LoRA, AdamW, bfloat16, fixed-teacher full-vocabulary distillation, and 8 A100 or 4 H200 GPUs.COPSD uses a 2,048-token maximum completion length, thinking mode for both policies, and checkpoints every 5 training steps.
  • D Environment and Hyperparameters: Evaluation uses identical decoding across methods, sampling 12 responses per problem with thinking mode, temperature 1.0, and top-p = 0.95.AfriMGSM uses 1,024-, 2,048-, and 4,096-token budgets; checkpoints are selected by 1,024-token performance before reporting other budgets.
Loading 2605.09548v1…