PaperlayerView exact revision on arXiv

Source-linked AI summary

When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy

Jirui Qi, Shan Chen, Zidi Xiong, Raquel Fernández, Danielle S. Bitterman, Arianna Bisazza

arXiv:2505.22888v2cs.CL

TL;DR

The paper addresses the limited evidence about whether LRMs can think in users’ languages while maintaining accurate answers. It evaluates six LRMs with XReasoning and tests prompting and post-training interventions. These interventions improve language alignment but reduce accuracy, and the trade-off persists despite targeted adaptation.

  • Problem

    Current multilingual benchmarks and evaluations provide limited evidence about whether LRMs generate reasoning traces in users’ languages alongside accurate answers.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper introduces XReasoning and evaluates six LRMs across multilingual math and science tasks, including prompt hacking and targeted post-training.

    Source: §1 · p. 2, §3 · p. 2, §4.1 · p. 3

  • Results

    Prompt hacking raises language matching from roughly 45–50% to above 90%, while reducing accuracy; post-training also improves alignment but leaves a marked accuracy loss.

    Source: §5 · p. 5

  • Takeaways & Limitations

    Current LRMs have limited multilingual reasoning capabilities, and improving user-language alignment remains coupled to lower answer accuracy.

    Source: §1 · p. 2, §5 · p. 5

    The benchmark uses translated math and science tasks with short structured answers, so generalization to open-ended or domain-specific reasoning remains unclear.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Recent Large Reasoning Models (LRMs) with thinking traces have shown strong performance on English reasoning tasks. However, their ability to think in other languages is less studied. This capability is as important as answer accuracy for real world applications because users may find the reasoning trace useful for oversight only when it is expressed in their own language. We comprehensively evaluate two leading families of LRMs on our XReasoning benchmark and find that even the most advanced models often revert to English or produce fragmented reasoning in other languages, revealing a substantial gap in multilingual reasoning. Prompt based interventions that force models to reason in the users language improve readability and oversight but reduce answer accuracy, exposing an important trade off. We further show that targeted post training on just 100 examples mitigates this mismatch, though some accuracy loss remains. Our results highlight the limited multilingual reasoning capabilities of current LRMs and outline directions for future work. Code and data are available at https://github.com/Betswish/mCoT-XReasoning.

1 Introduction

Large Reasoning Models show strong English reasoning, but their ability to think in users’ languages remains insufficiently studied. This work evaluates multilingual thinking traces and identifies a trade-off between language matching and answer accuracy.

Motivation

  • Language-matched thinking traces matter for oversight because users may not understand reasoning expressed in another language, even when answers are correct.The paper links understandable traces to user verification, trust, and satisfaction on complex tasks.Source: §1 · p. 1
  • XReasoning evaluates whether LRMs can generate thinking traces in users’ specified languages, addressing a gap in predominantly English evaluations.The benchmark targets multilingual reasoning because language-matched traces are easier for users to read and verify.Source: §1 · pp. 1–2

Contributions

  • The study evaluates six open-source LRMs from the Distilled-R1 and Skywork-OR1 families on multilingual reasoning tasks.The evaluated models span multiple parameter scales and two model families.Source: §1 · pp. 1–2
  • Prompt hacking substantially improves thinking-language matching but reduces answer accuracy, exposing a trade-off between explainability and performance.The intervention improves readability and oversight while limiting practical value when accuracy falls.Source: §1 · p. 2
  • Post-training with few examples improves users’ language alignment but does not completely eliminate the accuracy trade-off.The paper presents post-training as a partial mitigation rather than a complete solution.Source: §1 · p. 2

2 Related Work

Existing multilingual reasoning benchmarks do not fully distinguish performance among modern LRMs or examine whether traces use the user’s language. XReasoning is introduced to address these evaluation gaps.

Multilingual Reasoning Benchmarks

  • Recent LRMs can achieve at least 90% accuracy in several high-resource languages on MGSM, limiting that benchmark’s ability to reveal nuanced differences.Earlier multilingual benchmarks also show uneven performance across languages and weaker results in low-resource languages.Source: §2 · p. 2
  • XReasoning extends evaluation to more challenging math and science questions because existing benchmarks are insufficient for recent LRMs.The benchmark is designed to expose distinctions that easier multilingual datasets may miss.Source: §2 · p. 2

Improving Answer Accuracy with Cross-lingual Reasoning

  • Prior methods improve multilingual answer accuracy through translation or language-independent prompting, but do not study thinking in the user-specified language.This work treats language matching in reasoning traces as an evaluation target beyond answer correctness.Source: §2 · p. 2

3 Experimental Setup

The experiments use XReasoning to test six LRMs across 11 languages and measure both answer behavior and thinking-language alignment. Language matching is determined from the generated reasoning traces.

Benchmarks

  • XReasoning contains 370 questions in 11 languages, combining MGSM with translated AIME2024, AIME2025, and GPQA questions.The non-English questions were machine-translated from challenging English reasoning datasets.Source: §3 · p. 2
  • The benchmark’s answers are Arabic numbers or option letters, so answer translation is unnecessary.This keeps evaluation focused on multilingual question and reasoning behavior rather than translated answer formats.Source: §3 · p. 2

Models

  • The evaluation covers six models: Distilled-R1 at 1.5B, 7B, 14B, and 32B, plus Skywork-OR1 at 7B and 32B.These models represent two advanced open-source LRM families.Source: §3 · p. 3

Evaluation

  • Language matching rate is the proportion of instances where a model follows the instruction to think in the specified language.The language of text between the thinking delimiters is classified with LANGDETECT.Source: §3 · p. 3

4 Experiments and Results

Prompt interventions and limited post-training substantially improve language matching in multilingual reasoning, but both expose a persistent accuracy trade-off. Larger models mitigate this trade-off without eliminating it, while actual traces often default to English or Chinese.

4.1 Language Matching vs. Answer Accuracy

  • Standard prompts frequently fail to produce thinking traces in the requested language, including French, where the model almost always reasons in English.The same issue appears even when prompting the 671B R1 to think in French.Source: §4.1 · p. 3
  • Prompt hacking raises language matching from roughly 45–50% to well above 90% across parameter scales, but answer accuracy drops, with the decrease shrinking as models grow.Larger models remain more accurate under language matching, yet measurable losses persist on harder tasks.Source: §4.1 · p. 3
  • On AIME, prompt hacking increases Distilled-R1-32B matching from 46% to 98% while reducing average accuracy from 26% to 17%.Figure 2 compares language-pair heatmaps before and after hacking.Source: §4.1 · p. 3, Figure 2 · p. 4
  • English reasoning consistently yields higher accuracy, including for non-English queries, revealing a persistent English preference among advanced reasoning models.The result is especially pronounced after prompt hacking and aligns with cross-lingual reasoning findings.Source: §4.1 · pp. 3–4
  • Actual mismatched thinking traces for French, Japanese, Thai, and Swahili concentrate in English or Chinese, linking mismatch patterns to the languages represented in reasoning data.Table 2 reports predicted thinking-language distributions averaged over questions in each language.Source: §4.1 · p. 4, Table 2 · p. 4

4.2 Post-Training with Few Instances

  • Post-training on 100 in-language instances raises matching to nearly 100% for Thai and Telugu and around 80% for Japanese, but accuracy declines across datasets.Increasing training from 100 to 250 instances does not reliably fix the trade-off and can reduce matching on several language–dataset combinations.Source: §4.2 · p. 5, Figure 3 · p. 5
  • Model merging improves matching over the base model and can outperform 250-instance post-training on some accuracy benchmarks, yet remains less accurate than the original model on every dataset.The merged model combines Distill-R1-7B with a model trained on 250 Japanese instances.Source: §4.2 · p. 5, Table 3 · p. 5

5 Conclusion

The study examines language matching in multilingual reasoning traces and its trade-off with answer accuracy. It finds that prompting and lightweight post-training improve alignment but reduce accuracy, especially on complex tasks.

  • Prompt hacking reduces thinking-language mismatch but substantially lowers accuracy, especially on complex tasks.Targeted post-training on 100–250 examples sharply improves alignment but still markedly reduces accuracy.Source: §5 · p. 5

Limitations

The analysis is limited by its benchmark scope, machine-translated non-English materials, surface-level evaluation, and lightweight mitigation study. These constraints leave generalization, faithful reasoning, and user-centered trust effects unresolved.

  • The benchmark covers only translated math and science questions, so generalization to open-ended or domain-specific reasoning remains unclear.The tasks derive from AIME 2024/2025, GPQA-Diamond, and MGSM and use short, well-structured answers.Source: Limitations · p. 6
  • Machine translation of all non-English materials may introduce semantic drift that adds noise to language-matching and accuracy scores.Source: Limitations · p. 6
  • Language identification may misclassify mathematical expressions or codeswitched traces, affecting reported matching rates.The study also measures only surface alignment and boxed-answer correctness, not whether traces faithfully derive answers.Source: Limitations · p. 6
  • Fine-tuning uses no more than 250 examples per language, which may not extrapolate to truly low-resource languages or different writing systems.Source: Limitations · p. 6
  • The study assumes user-language traces increase trust, but user-centered evaluations combining preferences and task performance remain necessary.Source: Limitations · p. 6

A Experimental Details

Experiments use reproducible greedy decoding, long generation limits, exact-match answer evaluation, and regex extraction of boxed answers.

  • Experiments use greedy decoding and a 16,392-token generation limit to support reproducibility and maximize reasoning capability.Most experiments run on A100 40GB GPUs with approximately 7,000 GPU hours, plus separate inference and training resources for 32B models.Source: A Experimental Details · p. 10
  • Answer accuracy is measured by exact matching after regex extraction of the pretrained boxed answer field.Source: A Experimental Details · p. 10

B XReasoning Benchmark Examples

XReasoning contains challenging AIME, GPQA, and MGSM math and science questions, while prompt variants test explicit language control and uncontrolled behavior. The uncontrolled results show both low language matching and an accuracy trade-off when models are encouraged to use the user language.

B XReasoning Benchmark Examples

C Prompts and Instructions

  • Language-specific instructions explicitly and implicitly guide models to generate thinking traces in English, Spanish, Chinese, or Japanese.Source: C Prompts and Instructions · p. 10

D Prompt Hacking Prefix

E Prompts Without Language-Specific Instruction

F Reliability of Automatic Language Detector

The study validates LANGDETECT against human judgments for Japanese thinking traces and extends analysis of the languages LRMs actually use.

F Reliability of Automatic Language Detector

  • Table 7 compares average language-matching rates and answer accuracy across 11 languages.Source: Table 7 · p. 11
  • Human checks confirmed that LANGDETECT correctly identified Chinese segments in traces produced after Japanese prompts.The evaluated traces were predominantly predicted as Chinese, including a Japanese query answered with a Chinese reasoning trace.Source: F Reliability of Automatic Language… · pp. 10–11

G Extension of Actual Thinking Languages

H Details of Post-Training Experiments

The post-training experiments translate LIMO data into Japanese, Telugu, and Thai, then release models trained with small language-specific datasets.

H.1 Translated LIMO Examples

  • LIMO training instances were translated into Japanese, Telugu, and Thai for language-specific post-training.The translations were produced with GPT-4O-MINI, and Japanese examples are illustrated in Table 9.Source: §H.1 · p. 12

H.2 Post-Trained LRMs and Accessibility

  • Seven models comprise the post-training study, including a base model and 100- or 250-instance variants for Japanese, Telugu, and Thai.A low learning rate was used to reduce overfitting and catastrophic forgetting, and the resulting models were made publicly accessible.Source: §H.2 · p. 12

I Full Results of Post-Training

The full post-training results evaluate language-specific models across AIME, GPQA, and MGSM while prompting them in their training language.

I Full Results of Post-Training

Related papers

Loading 2505.22888v2…