PaperlayerView exact revision on arXiv

Source-linked AI summary

GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings

Konstantin Dobler, Federico Scozzafava, Jonathan Janke, Mohamed Ali, Simon Lehnerer

arXiv:2608.13698v1cs.CLcs.LG

TL;DR

GRPO/RLVR post-training remains predominantly English-focused despite multilingual pretraining and concerns about reasoning-language effects. This study evaluates multilingual and non-English GRPO broadly, finding small average native-language gaps and strong crosslingual transfer alongside model- and language-specific regressions.

  • Problem

    GRPO/RLVR post-training is studied primarily in English despite multilingual pretraining, limiting evidence about multilingual reasoning-language effects.

  • Method

    The paper conducts a large-scale controlled study across nine models, 11 training languages, multilingual and monolingual settings, and English- or native-language reasoning rewards.

  • Results

    Native-language rewards usually incur only a small performance gap versus English rewards, while crosslingual transfer is strong but some model-language combinations cause severe regressions on unseen languages or tasks.

  • Takeaways & Limitations

    Multilingual and non-English RLVR can yield broad crosslingual gains, but direct target-language training and broad evaluation remain important for performance and regression detection.

  • Takeaways & Limitations

    Template-based training data lacks the variability of real-world queries, and shared rather than individually tuned recipes may leave some negative results reducible through hyperparameter tuning.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR), often optimized with Group Relative Policy Optimization (GRPO), has become a central recipe for improving the reasoning capabilities of pretrained language models but current studies remain heavily English-centric. We conduct a large-scale empirical study of multilingual and non-English GRPO across a wide range of base models, training languages, and different reasoning language rewards. We find that training to reason in the native language often leaves only a small gap to training for English reasoning. We further observe strong crosslingual transfer: training in one language often improves performance in many others. However, specific trends are highly model- and language-dependent. In some cases, training in a particular language induces severe regressions on out-of-domain capabilities in other languages. Our analysis shows that RLVR beyond English can provide broad crosslingual gains, but also requires broad evaluation to detect language-specific regressions.

1 Introduction

This work examines GRPO/RLVR beyond English through broad multilingual and non-English experiments across models, languages, and training settings. It finds generally strong crosslingual benefits and a small average gap between non-English and English reasoning rewards, while identifying language-dependent regressions.

  • Background: RLVR optimizes diverse reasoning trajectories using automatically checkable outcome feedback, with GRPO emerging as an effective training method.Examples of verifiable feedback include exact-match numeric answers.
  • Motivation: Despite multilingual pretraining, GRPO/RLVR post-training remains primarily English-focused, and reasoning traces may drift toward a dominant language.This drift can occur even when prompts are posed in other languages.
  • Study scope: The study covers three model families, nine models, 11 languages, multiple model sizes, and both multilingual and monolingual training settings.The experiments include both “-Base” and post-trained variants, with monolingual training spanning 11 languages.
  • Main findings: Non-English reasoning rewards leave only a small average gap to English rewards, while GRPO produces substantial crosslingual transfer across many models and languages.The reported findings apply across a large range of models and languages.
  • Main findings: Specific model-language combinations can trigger large regressions on unseen languages or tasks despite improvements elsewhere.This language-dependent failure mode occurs with non-English prompt or rewarded reasoning languages.

2 Related Work

Prior work finds a tension between readable native-language reasoning and the higher accuracy often retained by English or mixed-language reasoning. Another line of work transfers English reasoning capabilities to other languages using varied crosslingual supervision signals.

  • Language of reasoning: Native-language reasoning improves readability, whereas English or mixed-language reasoning often retains higher task accuracy.This tension is documented in multilingual reasoning through inference-time interventions and analyses.
  • Language of reasoning: A similar tension also appears during RLVR, where reasoning may drift across languages.
  • Crosslingual transfer: Existing work transfers English reasoning to other languages through translations, reference traces, judges, or teacher distributions.These approaches differ operationally but use crosslingual supervision signals to bridge English and non-English reasoning.

3 Experimental Setup

The study uses procedurally generated multilingual RLVR data, broad model coverage, controlled monolingual and multilingual training runs, and evaluation across in-distribution, held-out, and established benchmarks. Training spans 11 languages with standardized GRPO settings and binary correctness rewards.

  • Datasets: The Multilingual Reasoning Gym provides procedurally generated data with adjustable difficulty across 14 languages for multilingual RLVR training.Difficulty matching is important because GRPO is sensitive to the relationship between problem difficulty and model capability.
  • Datasets: After filtering tasks with zero Qwen3-14B-Base pass@8 and excluding heavy-English tasks from training, the setup contains 62 training and 30 unseen evaluation tasks.Held-out tasks are sampled across domains including arithmetic, graphs, logic, and geometry.
  • Models: Experiments cover 9 base models from the Qwen, Gemma, and SmolLM families, spanning different sizes, post-training levels, and pretrained language coverage.The listed models range up to 8 billion parameters.
  • Training: Each language run lasts 500 steps with learning rate 1e−6, 64 prompts per step, 8 rollouts per prompt, and batch size 512, using DAPO without a KL penalty.The reward is binary when the extracted answer is verified correct.
  • Training: Training includes monolingual runs in 11 MGSM languages and one uniformly sampled multilingual run, reserving three languages for evaluation.The monolingual languages are English, Chinese, German, French, Spanish, Russian, Japanese, Thai, Swahili, Telugu, and Bengali.
  • Evaluation: Evaluation uses unseen samples from seen tasks, 30 unseen Multilingual Reasoning Gym tasks, MGSM, and PolyMath across four difficulty levels.The lowest PolyMath difficulty is roughly comparable to MGSM, whose performance is often similar to the low-difficulty PolyMath split.

4 Results

Results show strong control of reasoning language and substantial crosslingual transfer, with English rewards often outperforming native rewards by modest average margins. However, transfer and regressions are highly model-, language-, task-, and reward-dependent, requiring broad multilingual evaluation.

  • Reasoning language: 90.7% of monolingual English-reward rollouts use English, versus 94.9% of native-reward rollouts using the training language on seen evaluation languages.On unseen languages, English-reward models reason in English in 89.7% of monolingual and multilingual cases, while monolingual native-reward models use the training language 51.0%, the prompt language 35.0%, and English 9.0%.
  • Reasoning language rewards: English reasoning rewards outperform native-language rewards by 1.3pp–4.1pp on average across models and seen training languages, with larger gaps for Bengali, Telugu, and Swahili.Specific model–language combinations can show double-digit English-reward advantages, while native-reward models may switch languages on unseen prompts.
  • Crosslingual transfer: Substantial in-domain crosslingual transfer occurs, although training in the evaluation language generally performs best and English is often not the strongest transfer language.Lower-resource training languages can transfer strongly, but transfer is especially effective for high- or medium-resource languages.
  • Crosslingual transfer: 32.2pp from Swahili training improves Swahili evaluation on average, versus 20.5pp from Telugu, while Spanish-only training nearly recovers French’s 25.6pp target-language gain at -1.0pp.Target-language data is more beneficial for Swahili and Telugu, where transfer from other languages is less effective.
  • Multilingual training: Multilingual training is competitive with per-language source selection for Qwen3-Base at only 0.3pp lower, but averages 1.3pp below it for Qwen3.Despite using all languages, multilingual training still underperforms monolingual same-language training in some settings.
  • Training language and reasoning language: Training can cause extreme, task-specific regressions on unseen difficult-math and English-content tasks, especially when English prompts are absent; English training mostly alleviates them, while multilingual training may not.Regressions are highly model- and language-specific, and native-language reasoning rewards can also degrade previously learned capabilities in certain combinations.

5 Conclusion

Across nine base models and 11 training languages, multilingual GRPO shows strong crosslingual transfer. Native-language reasoning rewards usually incur only a small performance gap versus English reasoning, though English reasoning benefits lower-resource targets and direct target-language training remains important for in-domain performance.

  • Study scope: The study evaluates multilingual GRPO across nine base models, 11 training languages, multilingual training, and English- versus native-reasoning rewards.These settings define the study’s broad empirical scope.
  • Reasoning language: Native-reasoning rewards usually produce only a small performance drop compared with English-reasoning rewards.The comparison concerns performance under different reasoning-language rewards.
  • Reasoning language: English reasoning yields larger benefits for lower-resource target languages than native-language reasoning.The advantage is specifically reported for lower-resource target languages.
  • Crosslingual transfer: Training in one language often substantially improves performance in other languages, demonstrating strong crosslingual transfer.Despite this transfer, direct target-language training remains important for in-domain performance.

Limitations

The study relies on procedurally generated, template-based RLVR tasks, which enable controlled multilingual training at scale but do not match the variability of real-world queries. It also applies a shared training recipe across many model, language, and reward configurations.

  • The Multilingual Reasoning Gym enables controlled multilingual training at scale but lacks the variability of in-the-wild user queries and other real-world scenarios.Its tasks are procedurally generated and template-based, with a dynamic difficulty curriculum.
  • The study uses a shared training recipe across many model, language, and reward configurations.

A Training Details

The study uses a standardized DAPO-style GRPO training setup across languages, with fixed rollout, optimization, and hardware configurations. Evaluation combines avg@8 scoring on unseen samples with heldout, English-content, and difficult reasoning tasks.

  • Training configuration: RL training runs for 500 steps per language with 512 samples per step, corresponding to 64 unique prompts from 8 rollouts per prompt.The optimization uses a learning rate of 1e-6, maximum gradient norm of 0.1, and AdamW.
  • Training configuration: The training objective uses DAPO-style loss without group-standard-deviation reward scaling or a KL penalty, while multilingual language selection is uniform.Completions exceeding the maximum rollout length are masked.
  • Evaluation setup: Evaluation generates 8 rollouts per sample and reports avg@8 using publisher-recommended sampling parameters, a 16384-token maximum, and 25 unseen samples per task.The test set uses a different random seed from training; MGSM and PolyMath use official evaluation procedures.
  • Multilingual Reasoning Gym setup: The Multilingual Reasoning Gym excludes 22 of 92 tasks for English content or excessive difficulty, then trains on 62 of the remaining 70 tasks and holds out 8.The 22 filtered tasks serve as additional unseen evaluation tasks.
  • Evaluation setup: Additional evaluation covers 9 English-content tasks and 13 tasks deemed too difficult for Qwen3-14B-Base at the default difficulty level.The heldout task set contains 8 tasks that passed the difficulty and English-content filters.

B Sensitivity to language-control filtering · C Additional Related Work · D Full Cross-Lingual Transfer Matrices

The section shows that English-reward gains remain positive without language-control filtering, situates multilingual GRPO among prior multilingual reasoning methods, and defines the full cross-lingual transfer matrices and their macro-averaged metrics.

  • B Sensitivity to language-control filtering: 2.8 to 2.4pp: the English-reward advantage on seen languages decreases only modestly when all 50 paired non-English configurations are included.The sensitivity analysis covers five models and ignores post-training language control.
  • C Additional Related Work: Multilingual benchmarks including MGSM, MathOctopus, MMATH, and PolyMath evaluate reasoning across languages and reveal substantial performance disparities.MGSM also shows that chain-of-thought prompting can transfer across languages.
  • C Additional Related Work: Prior methods improve multilingual reasoning through crosslingual prompting, instruction tuning, multilingual chain-of-thought training, and language-relevant layer alignment.The cited approaches include CLP, xCoT, mCoT, LangBridge, and SLAM.
  • C Additional Related Work: Ranaldi and Pucci propose self-training through a more language-agnostic intermediate reasoning space.These methods provide context for studying crosslingual transfer under GRPO rather than prompting or supervised alignment.
  • D Full Cross-Lingual Transfer Matrices: Each matrix cell reports Avg@8 percentage-point improvement over the base model, with rows encoding training-language and reasoning-reward settings and columns encoding evaluation languages.Color indicates relative performance, bold marks the larger reasoning-reward value, and underlining marks the best value.
  • D Full Cross-Lingual Transfer Matrices: Easy Math averages MGSM and PolyMath-low, while Hard Math averages PolyMath-medium, PolyMath-high, and PolyMath-top.The main paper averages Easy Math only over languages supported by both MGSM and PolyMath-low; full Appendix results are provided for completeness.

D.1 Qwen3-1.7B-Base

This section evaluates Qwen3-1.7B-Base on multilingual reasoning after training, reporting Avg@8 percentage-point improvements over the base model across seen and multiple categories of unseen tasks.

  • Seen tasks: Training performance is assessed on tasks seen during training from the Multilingual Reasoning Gym.Results are reported as Avg@8 percentage-point improvement over Qwen3-1.7B-Base.
  • Unseen tasks: Generalization is evaluated across all unseen tasks using both overall and macro-average summaries.Both summaries report Avg@8 percentage-point improvement over Qwen3-1.7B-Base.
  • Heldout tasks: The evaluation separately reports results for main heldout tasks and unseen English-content tasks.These categories use Avg@8 percentage-point improvement over Qwen3-1.7B-Base.
  • Easy Math: Easy Math results combine MGSM and low-difficulty PolyMath, with some languages using only PolyMath low when MGSM is unavailable.The metric is Avg@8 percentage-point improvement over Qwen3-1.7B-Base.

D.2 Qwen3-4B-Base · D.3 Qwen3-8B-Base · D.4 Qwen3-1.7B

Across Qwen3-4B-Base, Qwen3-8B-Base, and Qwen3-1.7B, the appendices report Avg@8 percentage-point improvements over each model’s base-model baseline across seen, unseen, held-out, English-content, difficulty-stratified, and math tasks. For Qwen3-4B-Base, English rewards were ineffective for Chinese and Japanese training despite strong shifts toward the target reasoning language.

  • D.2 Qwen3-4B-Base: For Qwen3-4B-Base, English reward is ineffective when training in Chinese (zh).Ineffective means the base model had at least 2.5% support for the target reasoning language, but the trained model did not converge to that language.
  • D.2 Qwen3-4B-Base: For Chinese-prompt rollouts, Chinese output rises from 51.1% in the base model to 90.2% in the trained model, while English falls from 24.0% to 2.8%.Other-language output changes from 24.9% to 7.0%.
  • D.2 Qwen3-4B-Base: For Qwen3-4B-Base, English reward is ineffective when training in Japanese (ja).The same ineffectiveness criterion applies: the base model had at least 2.5% target-language support, but the trained model did not converge to that language.
  • D.2 Qwen3-4B-Base: For Japanese-prompt rollouts, Japanese output rises from 49.5% in the base model to 96.4% in the trained model, while English falls from 23.8% to 1.7%.Other-language output changes from 26.7% to 1.8%.
  • D.3 Qwen3-8B-Base: Qwen3-8B-Base is evaluated with Avg@8 percentage-point improvements over its base model on seen and unseen Multilingual Reasoning Gym tasks, macro averages, held-out tasks, English-content tasks, and easy math.The appendix also includes hard-math and very-difficult-task evaluations for this model.

D.5 Qwen3-4B · D.6 Qwen3-8B

The Qwen3-4B and Qwen3-8B evaluations compare Avg@8 percentage-point improvements over their respective base models across seen, heldout, unseen, difficult, English-content, and math tasks. For Qwen3-4B, Chinese training with an English reward is ineffective because the trained model does not converge to English reasoning.

  • D.5 Qwen3-4B: For Qwen3-4B, an ineffective reward means the base model had at least 2.5% support for the target reasoning language but the trained model did not converge to it.This criterion is used to mark some reward rows as ineffective.
  • D.5 Qwen3-4B: Chinese training with an English reward is ineffective for Qwen3-4B: Chinese-prompt rollouts remain 93.0% Chinese, 2.7% English, and 4.3% other languages after training.The base model produced 93.7% Chinese, 2.7% English, and 3.6% other languages on the same rollouts.
  • D.5 Qwen3-4B: Qwen3-4B is evaluated against base model Qwen3-4B on tasks seen during training and across multiple unseen-task aggregates.The unseen evaluations include average, macro-average, main heldout, and very difficult tasks from Multilingual Reasoning Gym.
  • D.5 Qwen3-4B: Qwen3-4B evaluation also isolates unseen English-content tasks and Easy Math, with Easy Math combining MGSM and PolyMath low.For languages without MGSM, Easy Math includes only PolyMath low.
  • D.5 Qwen3-4B: Qwen3-4B is further assessed on Hard Math spanning PolyMath medium, high, and top difficulty levels.The reported metric is Avg@8 percentage-point improvement over the base model Qwen3-4B.
  • D.6 Qwen3-8B: Qwen3-8B is evaluated against base model Qwen3-8B on seen tasks and several unseen-task aggregates from Multilingual Reasoning Gym.The unseen aggregates include average, macro-average, main heldout, and very difficult tasks.
  • D.6 Qwen3-8B: Qwen3-8B evaluation separately reports improvements on unseen English-content tasks from Multilingual Reasoning Gym.This preserves the distinction between English-content evaluation and the broader unseen-task aggregates.
  • D.6 Qwen3-8B: Qwen3-8B is also evaluated on Easy Math, combining MGSM with PolyMath low and using only PolyMath low where MGSM is unavailable.The metric is Avg@8 percentage-point improvement over the base model Qwen3-8B.

D.7 gemma-3-1b-it … E Base Model Performance

The appendix reports language-control findings for gemma-3-1b-it and organizes comparative Avg@8 analyses for gemma-3-4b-it and SmolLM3-3B. It also defines the base-model evaluation aggregates and presents corresponding baseline-performance tables.

  • D.7 gemma-3-1b-it: German, French, Japanese, Bengali, and Telugu training similarly increase the target language in their prompt rollouts while reducing English output.The reported target-language changes are German 25.6% to 92.2%, French 26.1% to 89.4%, Japanese 12.6% to 84.7%, Bengali 24.2% to 81.3%, and Telugu 7.2% to 86.3%.
  • D.8 gemma-3-4b-it: For gemma-3-4b-it, the appendix reports Avg@8 percentage-point improvements over the base model across seen, unseen, heldout, English-content, easy-math, and hard-math task groups.The tables cover Multilingual Reasoning Gym tasks seen during training, macro and average aggregates over unseen tasks, main heldout tasks, English-content unseen tasks, very difficult unseen tasks, Easy Math, and Hard Math.
  • D.9 SmolLM3-3B: For SmolLM3-3B, the appendix reports Avg@8 percentage-point improvements over the base model for the same broad multilingual reasoning and math task categories.Reported categories include seen training tasks, multiple unseen-task aggregates, heldout and English-content tasks, very difficult tasks, Easy Math, and Hard Math.
  • E Base Model Performance: Base-model tables report untrained avg@8 accuracy for each task group and evaluation language in percentage points, with a final Avg column averaging displayed evaluation-language columns.Cells are colored relative to the base models in each table, and bold entries mark the best value in each column.
  • E Base Model Performance: Easy Math averages MGSM and PolyMath-low, while Hard Math averages PolyMath-medium, PolyMath-high, and PolyMath-top.The appendix also defines unseen-task aggregates as equal-weight macro averages over specified Multilingual Reasoning Gym, Easy Math, and Hard Math groups.
Loading 2608.13698v1…