Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning

1 INTRODUCTION

• Prior multilingual benchmarks primarily measure final-answer accuracy, while models often revert to English reasoning under non-English prompts.The paper argues that answer correctness alone can conceal a mismatch between the question language and reasoning language.Source: §1 · pp. 1–2
• M2A combines multi-scale reasoning alignment with language-consistency rewards, using supervised fine-tuning and GRPO to encourage reasoning in the question language.The reward signals are intended to support learning without ground-truth reasoning traces in that language.Source: §1 · p. 2
• GeoFact-X provides culturally grounded factual questions localized to five countries and their predominant languages for evaluating multilingual reasoning.The paper also describes an automatic protocol for assessing reasoning-language consistency and reasoning correctness.Source: §1 · p. 2, Figure 1 · p. 2
• M2A significantly improves multilingual reasoning in mathematical and factual tasks while reasoning in the question languages.The models are trained on s1K-1.1 for mathematical reasoning and the GeoFact-X training set for factual reasoning.Source: §1 · p. 2

2.1 MULTILINGUALISM IN LARGE LANGUAGE MODELS

• Scaling and fine-tuning approaches face multilinguality dilution and forgetting risks, while prior analysis of English intermediate reasoning focused on internal representations rather than generated traces.The cited work also describes test-time strategies and multilingual post-training alignment techniques.Source: §2.1 · pp. 2–3
• For a Telugu example, base and supervised fine-tuned models are correct in either answer or language, whereas M2A is correct in both.The figure illustrates the distinction between answer correctness and reasoning in the prompt language.Source: Figure 2 · p. 3

2.2 EVALUATION OF MULTILINGUAL REASONING CAPABILITIES

• Existing multilingual benchmarks often emphasize task accuracy or multiple-choice answers, whereas this work targets free-form step-by-step reasoning and language alignment.The paper presents this design as a more faithful assessment of reasoning quality and language alignment.Source: §2.2 · p. 3

3 M2A: MULTI-SCALE MULTILINGUAL ALIGNMENT

• M2A combines English supervised fine-tuning with test-time reinforcement learning to encourage multilingual reasoning.The approach defines rewards intended to keep reasoning consistent with the question language.Source: §3 · pp. 3–4
• Questions are translated into a target language, and GRPO generates multiple outputs whose rewards are calculated individually.The setup uses machine-translated questions and a group of generated responses.Source: §3 · p. 4
• Context alignment compares whole-response embeddings with reference responses and uses shuffled negatives to discourage trivial matches.The margin-based reward encourages similarity for positive pairs while reducing similarity for shuffled pairs.Source: §3 · p. 4, Figure 3 · p. 4
• Reasoning-step alignment uses dynamic programming to match output sentences to reference sentences in order, allowing their counts to differ.The figure describes this as finer-grained supervision than whole-response alignment.Source: §3 · p. 4, Figure 3 · p. 4
• A language-consistency reward is 1 when a detector identifies the output as the target language and 0 otherwise; the final reward sums individual rewards.The approach combines this direct language incentive with its alignment rewards.Source: §3 · pp. 4–5

4 GEOFACT-X: GEOGRAPHY-BASED FACTUAL REASONING BENCHMARK

• GEOFACT-X contains 3,000 culturally grounded questions, about 600 per country, spanning five subject areas across the USA, India, Japan, Kenya, and Thailand.The questions are localized to English, Hindi, Japanese, Swahili, and Thai.Source: §4 · p. 6
• Dataset construction includes cross-language consistency checks, generated reasoning traces, and native or C1-level review of all test samples.The 85/15 train-test split is designed to avoid semantic overlap, including across languages.Source: §4 · p. 6
• Evaluation measures answer accuracy and reasoning quality, assigning zero reasoning score when a trace is not in the question’s language.Qwen2.5-72B-Instruct judges reasoning by comparing model traces with human-verified references.Source: §4 · p. 6

5 REVISITING MULTILINGUAL MATHEMATICAL REASONING BENCHMARK

• MGSM evaluates multilingual mathematical reasoning in ten languages with chain-of-thought prompts, but reports only mathematical accuracy.That reporting implicitly assumes high accuracy implies language-consistent reasoning.Source: §5 · pp. 6–7
• Language accuracy measures whether generated reasoning matches the intended question language.It uses a language identifier to assess the generated output against the target language.Source: §5 · p. 7
• Joint accuracy combines mathematical correctness with reasoning in the target language.The definition uses the number of samples, an indicator function, generated output, target language, and predicted versus ground-truth answers.Source: §5 · p. 7
• Several MGSM models, including Qwen2.5-72B-Math-Instruct and Llama-3-70B-Instruct, show large gaps between mathematical and joint accuracy, indicating reasoning in the wrong language.The s1 models improve mathematical performance while degrading language accuracy; mathematical accuracy alone therefore overestimates multilingual reasoning ability.Source: §5 · p. 7, Figure 6 · p. 7

6.1 DATASET

• Experiments use Qwen-2.5-7B-Instruct; math training uses s1K-1.1 and translated s1K-X baselines, while GEOFACT-X supplies five-country factual train and test splits.The factual dataset covers English, Hindi, Japanese, Swahili, and Thai.Source: §6 · p. 8, §6.1 · p. 8

6.2 MATHEMATICAL REASONING

• M2A outperforms baselines on all GSM8K metrics and gains substantially in MGSM joint accuracy versus SFT while preserving query-language reasoning.SFT improves mathematical reasoning but degrades MGSM multilingual performance; translated-data SFT preserves language accuracy but reduces mathematical accuracy, while GRPO changes little.Source: §6.2 · p. 8

6.3 FACTUAL REASONING

• On GEOFACT-X, M2A has the strongest reasoning performance and improves both country-associated and non-associated settings by 4–6% in reasoning score and 20–21% in answer accuracy.The Swahili example shows all systems reasoning in the question language, but only M2A predicting the correct answer.Source: Table 3 · p. 9, §6.3 · p. 9
• All pretrained models perform better when the language is associated with the country, a gap attributed likely to more paired examples in pretraining corpora.For example, Thai is associated with Thailand but not the USA.Source: §6.3 · p. 9
• Google Translate output translation offers no substantial improvement over direct GEOFACT-X evaluation, suggesting post-hoc translation is at most a superficial fix.The authors relate this result to weaker multilingual alignment.Source: §6.3 · p. 9, Table 4 · p. 9

6.4 ABLATION STUDY

• The full reward model achieves the best results on both math benchmarks; context alignment supports global alignment, while reasoning-step alignment supports stepwise fidelity.Context alignment improves MGSM multilingual performance but slightly lowers GSM8K accuracy; reasoning-step alignment boosts multilingual performance and mitigates that degradation.Source: Table 5 · p. 9, §6.4 · p. 10
• The margin-based hinge formulation achieves the best results across all metrics, while vanilla cosine is weaker and adding a negative-sample term helps MGSM but slightly reduces GSM8K.The formulation combines negative samples with a margin to stabilize alignment.Source: §6.4 · p. 10, Table 6 · p. 10

• Many LLMs predominantly reason in English or Chinese even when prompted in other languages, limiting their applicability in culturally and linguistically diverse settings.Source: §7 · p. 10
• M2A combines multi-scale alignment rewards with a language-consistency objective to align reasoning traces with references at context and step levels.Source: §7 · p. 10
• GEOFACT-X pairs geography-based factual questions across five languages with step-by-step traces and evaluation of logic, factual correctness, and language consistency.Source: §7 · p. 10
• M2A consistently improves multilingual mathematical and factual reasoning while maintaining strong English performance.Source: §7 · p. 10

• The work aims to encourage research on multilingual reasoning limitations and equitable access to high-performing LLMs regardless of user language.Source: ETHICS AND REPRODUCIBLE STATEMENT · p. 10
• The paper notes societal concerns shared with standard language-model research, including high-resource-language bias and hallucination.Source: ETHICS AND REPRODUCIBLE STATEMENT · pp. 10–11
• The experimental setting is specified in Appendix B, and the codebase is supplied as supplementary material.Source: ETHICS AND REPRODUCIBLE STATEMENT · p. 11

A.1 DATASET COLLECTION

• The dataset contains 3,000 QA pairs: 600 for each of five country-language pairs, with questions and answers in English, Hindi, Japanese, Swahili, and Thai.Source: §A.1 · p. 14, Figure 7 · p. 15
• Questions span ten topic domains and subcategories, whose allocated proportions are represented hierarchically and applied across five countries.Source: §A.1 · p. 14, Figure 8 · p. 16
• The data are split 85% for training and 15% for testing; authors verified 10% of training data and all test data, which native or C1-level speakers also reviewed.Source: §A.1 · p. 16

A.2 EVALUATION PROTOCOL

• Answer accuracy checks predictions against reference answers, while reasoning scores compare traces with human-verified references and are zeroed for language mismatches.Source: §A.2 · p. 16

B.1 TRAINING

• Training uses the s1 configuration except for batch size 8, runs for five epochs, and translates s1K-1.1 into ten MGSM languages.Source: §B.1 · p. 17
• Qwen2.5-72B Instruct judges reasoning traces against ground truth for logic, insights, factual correctness, and conclusion validity, with mismatch detection and answer validation.Source: Figure 10 · p. 18

B.2 EVALUATION

• Mathematical reasoning is evaluated on MGSM Native-CoT and MMLU-ProX Math with five-shot chain-of-thought prompts, with langid checking language correctness.Source: §B.2 · p. 18

C M2A WITH DIFFERENT LANGUAGES

• Randomized translation across ten languages yields the strongest overall results; fixed Japanese or Swahili training incurs only minor accuracy decreases.Source: Table 8 · p. 19, C M2A WITH DIFFERENT LANGUAGES · p. 19

B.3 MODELS EVALUATED ON MGSM

• The evaluated model set includes Qwen2.5, s1, Llama, Gemma, and DeepSeek-R1 across a range of sizes.Source: §B.3 · p. 19, Table 7 · p. 19

B.2 EVALUATION

• Table 9 compares MGSM accuracy for Qwen2.5-7B-Instruct and post-training methods, while omitting standard errors for readability.Source: Table 9 · p. 20

D MGSM EVALUATION IN EACH LANGUAGE

• SFT improves math but loses language performance; GRPO rarely changes either, while M2A generally maintains language performance, except lower German performance possibly tied to the translation language.Source: D MGSM EVALUATION IN EACH LANGUAGE · p. 20
• All models answer the Russian question correctly, but SFT reasons in English while the other models reason in Russian.GRPO's reasoning is almost identical to the base model's, consistent with its near-unchanged metrics.Source: D MGSM EVALUATION IN EACH LANGUAGE · p. 20

E GEOFACT-X EVALUATION IN EACH LANGUAGE AND COUNTRY PAIR

• Associative country-language pairs score higher across all metrics; English generally outperforms other languages, and U.S. non-associative cases exceed Kenya and Thailand.Source: E GEOFACT-X EVALUATION IN EACH… · p. 20
• Including all five languages in training substantially increases language accuracy, but reasoning scores and answer accuracy show no consistent trend across language-country pairs.Source: E GEOFACT-X EVALUATION IN EACH… · p. 20