PaperlayerView exact revision on arXiv

Source-linked AI summary

R3S: Refining and Recovering Reinforcement Signals for Multilingual Understanding and Reasoning

Junxiao Liu, Zhijun Wang, Yixiao Li, Zhejian Lai, Liqian Huang, Xin Huang, Xue Han, Junlan Feng, Shujian Huang

arXiv:2602.05940v2cs.CL

TL;DR

Multilingual reasoning models struggle both to understand non-English questions and to reason in the question language, while existing methods typically optimize only one capability. R3S disentangles these objectives with filtered translation rewards and self-generated English hints, improving language-consistent accuracy across benchmarks while maintaining near-perfect language consistency.

  • Problem

    Multilingual reasoning has separate bottlenecks in target-language question understanding and target-language reasoning, which existing methods typically optimize individually.

  • Method

    R3S refines downstream translation rewards through English-solvability filtering and recovers target-language RLVR signals with self-generated English reasoning hints.

  • Results

    10.3 percentage points: R3S improves MMATH language-consistent accuracy over the target-language RLVR baseline across three backbones and five languages, with similar gains on MMLU-ProX.

  • Takeaways & Limitations

    R3S improves multilingual question understanding and target-language reasoning while maintaining 99.9–100.0% language consistency and not compromising English reasoning ability.

  • Takeaways & Limitations

    The evaluation excludes extremely low-resource languages, and the largest model tested has 4B parameters, leaving effectiveness on larger reasoning models unexamined.

Abstract

from arXiv · show

Large reasoning models often default to English reasoning when processing non-English questions, yet their performance drops substantially when reasoning in the question language. Even with the same reasoning language, semantically equivalent English and non-English questions still exhibit a clear performance gap. Together, these phenomena reveal two distinct bottlenecks: target-language question understanding and target-language reasoning. Existing methods typically optimize only one of these capabilities. However, simply combining them may not be sufficient to optimize both effectively, as answer correctness alone cannot distinguish failures in question understanding from those in reasoning. We propose R3S, a reinforcement learning framework that disentangles the optimization of the two capabilities. R3S refines translation rewards derived from downstream reasoning accuracy through English-solvability filtering and recovers target-language RLVR signals using self-generated English hints. Together, these designs require neither external model feedback nor external multilingual training data. Experiments across three backbone models and five languages show that R3S improves language-consistent accuracy over the target-language RLVR baseline on MMATH by an average of 10.3 percentage points, while maintaining near-perfect language consistency. Consistent gains on MMLU-ProX further demonstrate its generalization beyond math problems.

Introduction

R3S addresses two multilingual reasoning bottlenecks—understanding target-language questions and reasoning in the target language—using capability-specific reinforcement signals. Across three backbones and five languages, it improves language-consistent accuracy while preserving near-perfect language consistency.

  • Motivation: Non-English questions expose separate limitations in target-language question understanding and target-language reasoning.Models may default to English reasoning, lose performance when reasoning in the question language, and perform worse on semantically equivalent non-English questions even with reasoning language fixed.
  • Motivation: Existing methods generally optimize either multilingual question understanding or target-language reasoning rather than both.Translation-based approaches improve semantic correspondence but retain primarily English reasoning, whereas reasoning-alignment methods target-language reasoning trajectories.
  • R3S: R3S refines translation rewards by filtering for questions the model can reliably solve in English, reducing failures attributed to problem-solving rather than translation.It uses downstream reasoning accuracy as the translation reward and applies English-solvability filtering to make that signal more specific.
  • R3S: R3S recovers target-language RLVR signals with self-generated English reasoning hints when direct target-language sampling produces no correct response.It also performs English RLVR on source questions and requires no external annotation or model feedback.
  • Results: 10.3 percentage points: R3S improves MMATH language-consistent accuracy over the target-language RLVR baseline while maintaining near-perfect language consistency.Gains across five non-English languages and three models, together with improvements on MMLU-ProX, support generalization beyond math problems.
  • Contribution: R3S combines refined translation rewards and recovered target-language signals into a framework that improves both multilingual capabilities without external model feedback.The method starts from English problems and constructs multilingual training instances and reasoning hints from model generations.

Related Work

Prior multilingual reasoning work primarily targets either question understanding or target-language reasoning. R3S instead uses distinct reinforcement signals to optimize both capabilities within one framework.

  • Question Understanding: Multilingual question-understanding studies show performance gaps across semantically equivalent questions expressed in different languages.This motivates improving semantic correspondence between questions across languages.
  • Target-Language Reasoning: Target-language reasoning methods use translated chain-of-thought data, trajectory alignment, preference optimization, or reinforcement learning to control reasoning language and transfer reasoning ability.These approaches focus on generating or aligning reasoning trajectories in the question language.
  • R3S: R3S separates the two objectives: English-solvability filtering refines translation rewards, while self-generated English hints recover RLVR signals when direct target-language responses all fail.Both signals are generated by the model itself, avoiding external model feedback.

Methods

R3S builds capability-specific training signals by filtering translation rewards through English solvability and recovering target-language reasoning signals with English hints. These signals are optimized with multilingual data and GRPO.

  • Framework: R3S uses downstream reasoning accuracy to reward target-language translations and restricts target-language RLVR to problems already solved in English.This reduces interference from question misunderstanding and insufficient problem-solving ability.
  • Data Construction: R3S generates target-language translations from English problems and uses English-verified translations as target-language problems for subsequent training.Correct English responses generated during translation verification are retained to construct reasoning hints.
  • Framework: Figure 1 organizes R3S into translation-signal refinement followed by target-language signal refinement and recovery, with the resulting data used for GRPO optimization.Stage 2 uses English hints when direct target-language sampling fails.
  • Translation Reward: COMET can score an imprecise “parallelogram”→“quadrilateral” translation higher than the precise translation, despite losing a mathematical constraint.This motivates evaluating translations through whether downstream reasoning remains solvable.
  • Translation Reward: R3S assigns a positive binary translation reward when at least one English response to a candidate translation is correct and otherwise assigns zero.The reward tests whether the translation preserves enough task-critical information for solvability rather than ranking surface similarity.
  • Translation Reward: English-solvability filtering reduces the false-negative rate of translation rewards from 31.3% to 5.2%.Only source problems meeting the solvability threshold are used for translation training, reducing zero rewards caused by insufficient problem-solving ability.
  • Signal Recovery: R3S uses direct target-language sampling first, then constructs approximately 200-token English reasoning hints from correct English responses when all direct responses are incorrect.Hint-assisted responses are retained for RLVR only if at least one is correct.
  • Optimization: GRPO updates the model with English reasoning, translation, and target-language reasoning data, while the framework uses no external model feedback.Its target-language reward accounts for correctness, language consistency, output format, and degenerate repetition.

Experiments

Experiments evaluate R3S across three backbones, five target languages, and complementary math and STEM benchmarks. R3S achieves the strongest multilingual results, preserves language consistency, outperforms direct combination and alignment-based training, and generalizes beyond mathematical reasoning.

  • Experimental setup: Experiments use three backbones spanning different scales and multilingual capabilities, five target languages, and MMATH plus MMLU-ProX STEM benchmarks.MMATH covers four mathematical reasoning subsets, while MMLU-ProX evaluates five STEM domains.
  • Experimental setup: Evaluation reports avg@4 Language Consistency, Accuracy, and LC&Acc, with results macro-averaged across benchmark subsets or domains.LC&Acc measures responses that are both correct and language-consistent.
  • MMATH results: 36.1%, 47.4%, and 63.7% LC&Acc are R3S’s highest average scores across the three backbones on MMATH, exceeding SoftLC-RL by 10.3 points on average.Gains are consistent across all five target languages.
  • MMATH results: 99.9–100.0% language consistency lets R3S retain comparable or higher accuracy than Naive RL while avoiding its fallback to English reasoning.R3S trails Naive RL by 0.7 points on Qwen3-1.7B and exceeds it by 0.7 points on Qwen3-4B.
  • Comparisons: R3S exceeds alignment-based training and Direct Combination, averaging 4.2 points over Direct Combination and up to 6.0 points on DeepSeek-Distill-Qwen-1.5B.The results indicate that simply combining translation training with target-language RLVR is insufficient.
  • Generalization: 14.7 percentage points is R3S’s average LC&Acc gain over SoftLC-RL on MMLU-ProX STEM despite training exclusively on mathematical problems.The gains transfer across broader multilingual STEM problems requiring diverse domain knowledge.

Analysis

Ablations show that reward refinement and signal recovery provide complementary benefits, while filtering improves translation-reward reliability and hint-assisted sampling recovers otherwise unusable reasoning signals.

  • Ablation Study: 47.4% average LC&Acc is achieved when reward refinement and signal recovery are combined, outperforming Direct Combination by 3.3 points.Reward refinement alone reaches 45.8%, and signal recovery alone reaches 46.5%.
  • Ablation Study: Removing translation training lowers average LC&Acc from 47.4% to 44.6%, showing that translation optimization contributes beyond target-language reasoning training.The ablation retains refined translation rewards for data construction but excludes translation samples from policy optimization.
  • Effectiveness of Translation Reward Refinement: English-solvability filtering reduces false negatives from 31.3% to 5.2% and false positives from 5.3% to 2.2% in translation rewards.The filtering keeps source problems the model can reliably solve in English, reducing ambiguity between translation errors and problem-solving failures.
  • Effectiveness of Translation Reward Refinement: Reward refinement is preferred over Direct Combination in 56.3% of translation judgments versus 15.6% preferring Direct Combination, and raises FLORES-200 COMET from 82.1 to 84.7.The remaining 28.1% of judgments are ties.
  • Effectiveness of Target-Language Signal Recovery: Signal recovery remains useful across training stages because hint-assisted sampling recovers correct target-language responses for approximately 70–80% of problems whose direct samples all fail.The all-failure rate first increases and then declines on Qwen3-1.7B and Qwen3-4B, while recovery success remains stable.
  • Effectiveness of Target-Language Signal Recovery: Adding signal recovery raises average MMATH LC&Acc from 44.1% to 46.5%, despite evaluation using no reasoning hints.The result indicates that hint-assisted training transfers to unassisted target-language inference.

Conclusion

R3S improves target-language question understanding and reasoning through capability-specific reinforcement signals, with both mechanisms contributing to gains in language-consistent accuracy across benchmarks.

  • Conclusion: R3S improves language-consistent accuracy on MMATH and MMLU-ProX across three backbones while maintaining near-perfect language consistency.Ablations confirm that reward refinement and signal recovery both contribute.

Limitations

The evaluation covers five target languages and models up to 4B parameters, leaving extremely low-resource languages and larger reasoning models untested.

  • Limitations: R3S is not evaluated on extremely low-resource languages or reasoning models larger than 4B parameters.Future work is proposed across broader language-resource levels, language families, and larger models.

A Complete Training Procedure

R3S uses a two-stage procedure that first identifies reliably English-solvable questions and refines translation rewards, then trains on verified target-language translations with recovered signals when direct sampling fails.

  • A Complete Training Procedure: Stage 1 samples English responses, identifies reliably solvable questions, and generates and verifies target-language translations only for that subset.English reasoning is used to refine the translation rewards.
  • A Complete Training Procedure: Stage 2 operates only on positively verified translations to reduce interference from question misunderstanding.

B.1 Training Data

R3S builds multilingual training instances from English problems across five target languages, while using online generations and specified optimization settings. Baseline data may instead rely on externally generated translations and responses.

  • Training Data: R3S uses DAPO-MATH-17K English problems to construct training instances for French, Portuguese, Japanese, Korean, and Thai.The RL set contains 3,000 English source problems per target language, with difficulty stratification based on initial English reasoning accuracy.
  • Training Data: For each target language, R3S generates target-language translations and English reasoning hints online from English source problems.The supplied passage states that these generations are produced by R3S itself.
  • Training Data: Some baselines use externally generated translations and target-language responses, with translation fidelity verified through multiple evaluation passes.These resources are used only by the corresponding baselines.
  • Training Data: DeepSeek-Distill-Qwen-1.5B receives a cold-start stage teaching target-language reasoning before reinforcement learning.Qwen3-8B generates correct, language-consistent target-language responses to English questions for this stage.
  • Training Data: R3S training uses AdamW with global RL batch size 512, mini-batch size 64, learning rate 1 × 10^-6, and KL coefficient 0.001.Unless otherwise stated, the procedure samples G = 6 responses and K = 4 candidate translations.

C Detailed Results on Mathematical Reasoning

R3S combines a two-stage training procedure with detailed mathematical evaluations across difficulty levels. On AIME24/25 and MATH500, it achieves strong language-consistent accuracy while reducing repetition relative to Direct Combination.

  • Evaluation: The evaluation reports Accuracy, Language Consistency, and LC&Acc across high- and medium-difficulty mathematical problems.Degenerate repetition rates are also reported to characterize a common failure of target-language reasoning.
  • Results on difficult problems: 28.1% average LC&Acc on AIME24/25 accompanies 100.0% language consistency.R3S outperforms SoftLC-RL and M-Thinker by 9.7 and 11.2 percentage points, respectively, and improves over Direct Combination by 3.8 points.
  • Results on medium-difficulty problems: 87.6% average LC&Acc on MATH500 is reported with 99.9% language consistency.
  • R3S training procedure: Stage 1 filters source problems using English solvability, then samples target-language translations and English verification responses.The algorithm initializes translation-related datasets and retains stable source questions before translation sampling.
  • R3S training procedure: Stage 2 samples direct target-language responses and hint-assisted responses before updating the policy with GRPO.The target-language dataset is populated from hint-assisted responses with final rewards, and the next policy uses D_en, D_trans, and D_tgt.
  • Generation stability: R3S reduces repetition from 12.0% to 9.8% on AIME24/25 and from 2.6% to 1.6% on MATH500 versus Direct Combination.

D.1 Why General-Purpose Translation Metrics Are Insufficient

General-purpose translation metrics can miss task-critical mathematical meaning, whereas downstream reasoning accuracy provides a more task-sensitive translation signal. R3S therefore refines rewards with English-solvability filtering, balancing reliability against training coverage.

  • Why general-purpose metrics fail: Mathematical translations can remain fluent while altering terminology, quantities, or logical constraints that change the problem.A Korean translation weakening “parallelogram” to “quadrilateral” received a higher COMET score despite omitting a task-critical constraint.
  • Why general-purpose metrics fail: COMET agrees with independent judgments on 59% of general-domain pairs but only 46% of mathematical pairs.The comparison used 200 FLORES-200 pairs and 200 DAPO-MATH pairs.
  • Reasoning accuracy as a translation signal: 64.3% of pairs with reasoning-accuracy differences above 0.2 favor the higher-accuracy translation, versus 30.4% favoring the lower-accuracy translation.For critical pairs, the nonzero-accuracy translation is preferred in 75.9% of cases and loses in 16.1%.
  • Binary versus graded rewards: The graded reward reaches 46.8% average MMATH accuracy, below the binary reward’s 47.4% under the same Qwen3-1.7B setting.The paper attributes the difference plausibly to variance from estimating reward magnitudes with only G sampled responses.
  • Reward refinement and training coverage: At θ = 1/3, false-negative reward noise falls to 5.2% and MMATH LC&Acc peaks at 47.4%, while stricter filtering lowers coverage and performance.LC&Acc is 44.1% without filtering, 46.6% at θ = 1/6, and 45.3% at θ = 1/2.
  • Reward refinement and training coverage: The eligible training pool expands during training, reaching about 85% for Qwen3-1.7B and 90–91% for Qwen3-4B.English RLVR improves source-problem solvability, allowing previously filtered problems to enter translation training.

F Reward Modeling and Implementation

R3S combines correctness, language consistency, repetition, and format signals for target-language reasoning. Its translation prompt explicitly requires semantically faithful translations that preserve mathematical notation and formatting, while repetition checks target sampled responses.

  • Reward modeling: Target-language reasoning uses accuracy, language consistency, repetition, and format as four reward components.Correctness is determined by the benchmark’s verifiable answer, while language detection checks both reasoning traces and final answers.
  • Reward modeling: Correct answers receive reward only when outputs are well-formed, language-consistent, and free of repetition.Well-formed, language-consistent, non-repetitive incorrect answers receive a smaller reward.
  • Translation implementation: The translation prompt asks for zero loss of mathematical or logical meaning while preserving mathematical notation and output formatting.The template frames the model as a professional translator and specifies the target language.
  • Repetition handling: Repetition is detected through repeated 20-grams or lines of at least 20 tokens occurring six or more times.These checks target local phrase loops and longer repeated reasoning blocks, excluding the English hint.
Loading 2602.05940v2…