PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?

Zhi Rui Tam, Cheng-Kuang Wu, Yu Ying Chiu, Chieh-Yen Lin, Yun-Nung Chen, Hung-yi Lee

arXiv:2505.17407v1cs.CL

TL;DR

The paper investigates which languages LRMs use to reason and how that choice affects multilingual performance and behavior. It evaluates language patterns and steering approaches, finding that hub-language preferences and the effects of language choice vary across resource levels and task types.

  • Problem

    Despite multilingual training and use, how input language affects LRMs’ internal reasoning remains an open question with implications for serving users across languages.

    Source: §1 · p. 2

  • Method

    The study examines multilingual reasoning patterns, uses text prefilling to steer reasoning language, and applies segmentation-classification to analyze reasoning strategies.

    Source: §1 · p. 3, §4 · p. 4

  • Results

    LRMs tend to use hub languages for reasoning; forcing non-hub reasoning can hurt low-resource reasoning tasks, while native-language reasoning can benefit cultural tasks and safety behavior varies by language.

    Source: §1 · pp. 2–3, §7 · pp. 11–12

  • Takeaways & Limitations

    Considering internal reasoning language is important when aiming to make LRMs more inclusive for users across languages.

    Source: §7 · p. 12

    The study covers eight languages and medium-scale LRMs under 30B parameters, and its LLM-annotated, four-habit analysis may miss subtler strategies or establish causality.

    Source: §8 · p. 12

Abstract

from arXiv · show

Large reasoning models (LRMs) have demonstrated impressive performance across a range of reasoning tasks, yet little is known about their internal reasoning processes in multilingual settings. We begin with a critical question: {\it In which language do these models reason when solving problems presented in different languages?} Our findings reveal that, despite multilingual training, LRMs tend to default to reasoning in high-resource languages (e.g., English) at test time, regardless of the input language. When constrained to reason in the same language as the input, model performance declines, especially for low-resource languages. In contrast, reasoning in high-resource languages generally preserves performance. We conduct extensive evaluations across reasoning-intensive tasks (MMMLU, MATH-500) and non-reasoning benchmarks (CulturalBench, LMSYS-toxic), showing that the effect of language choice varies by task type: input-language reasoning degrades performance on reasoning tasks but benefits cultural tasks, while safety evaluations exhibit language-specific behavior. By exposing these linguistic biases in LRMs, our work highlights a critical step toward developing more equitable models that serve users across diverse linguistic backgrounds.

1. Introduction

The paper asks how input and reasoning languages shape LRM behavior, evaluating both task performance and behavioral outcomes. It finds that reasoning-language choice is consequential, with effects that differ across resource levels and task types.

1. Introduction

  • LRMs generate an intermediate thinking sequence before producing a final answer, making the internal reasoning process distinct from the response.Source: §1 · p. 2
  • Moderately resourced Japanese and Korean may reason in the input language or switch to related Chinese, whereas low-resource Swahili and Telugu default to English.Source: §1 · p. 2
  • Forcing reasoning away from a preferred hub can substantially degrade performance, especially for low-resource languages, while hub-language reasoning can maintain or improve results.Source: §1 · p. 2
  • The evaluation combines performance tests on reasoning-intensive tasks with behavior-oriented assessment of toxicity and cultural knowledge.Source: §1 · p. 2
  • Hub-language reasoning improves mathematical and knowledge-based task accuracy by up to 26.8%, while non-hub reasoning reduces toxicity and enhances cultural-task performance.Source: §1 · pp. 2–3
  • A segmentation-classification method identifies correlations between language-specific prefill tokens and reasoning strategies: Chinese correlates with subgoal setting (r = 0.51), and English with backward chaining (r = 0.30).Source: §1 · p. 3

2. Evaluation Setup

The evaluation framework examines both task accuracy and behavioral alignment, using reasoning benchmarks, non-reasoning benchmarks, and eight selected languages. Its language sample is intended to span geographic, linguistic-family, and resource variation.

2. Evaluation Setup

  • The framework measures reasoning-language effects on mathematics and knowledge-intensive accuracy, alongside safety and cultural appropriateness.Source: §2 · p. 3
  • MMMLU supplies human-verified translations of 14,042 questions across 14 languages, while MATH-500 contributes 500 translated problems spanning five mathematics areas.Source: §2 · p. 3
  • CulturalBench-Hard tests nuanced cultural knowledge with 1,200 questions, and LMSYS-Toxic uses 2,000 translated prompts selected for a high toxic rate.Source: §2 · p. 3

2.1. Languages

  • The study selects English, Chinese, Spanish, Russian, Japanese, Korean, Telugu, and Swahili to represent varied regions, language families, and resource levels.Source: §2.1 · p. 4

3. The Reasoning Hub Phenomenon in Multilingual LRMs

The paper identifies English and Chinese as dominant reasoning hubs: models often use these languages for internal reasoning regardless of input language. They can nevertheless respond in the input language, separating internal thinking language from external answer language.

3. The Reasoning Hub Phenomenon in Multilingual LRMs

  • The paper terms English and Chinese “reasoning hubs” because models predominantly default to this small set during long chain-of-thought reasoning, regardless of input language.Source: §3 · p. 4
  • Larger open-weight models reason in English across diverse input languages, producing reasoning-to-answer language mismatches in over 90% of analyzed cases for QwQ-32B and Qwen3-30B-A3B.Source: §3 · p. 4
  • The figure’s lower heatmap shows that models can still generate final answers in the input language despite their internal preference for hub languages.Source: §3 · p. 4
  • The observed hub-language pattern motivates testing what happens when reasoning is deliberately steered away from these dominant languages.Source: §3 · p. 4

4. Controlling Reasoning Languages with Text Prefilling

The paper uses language-specific text prefilling to steer LRMs’ reasoning language, then evaluates performance and behavior under English versus target-language reasoning. English generally benefits reasoning-task performance, while safety and cultural outcomes vary by language and region.

4. Controlling Reasoning Languages with Text Prefilling

  • Text prefilling seeds the reasoning prompt with a language-specific token or phrase, substantially improving alignment between the targeted reasoning language and the input language.The authors report no significant MATH-500 performance difference between prefilling and token masking, and select prefilling for cross-lingual versatility.Source: §4 · pp. 4–6, Figure 2 · p. 5

4.1. Performance-Oriented Results

  • 26.8% average improvement for DeepSeek-R1-Distill-Llama-8B with English rather than native-language reasoning; larger models also improve, by 16.1%, 5.4%, and 11.5%.The comparisons are for Qwen-14B, QwQ-32B, and Qwen3-30B-A3B, respectively.Source: §4.1 · p. 6, Table 1 · p. 6
  • English reasoning generally outperforms native-language reasoning on MMLU, with Swahili improving by an average of 13.6% across tested models.The comparison extends the English-reasoning advantage beyond mathematical problem-solving.Source: §4.1 · pp. 6–7, Table 2 · p. 7

4.2. Behavior-Oriented Results

  • For QwQ-32B and Qwen3, English reasoning raises toxicity rates by 1–3.5 percentage points for Japanese, Korean, Chinese, and Spanish inputs, while Russian shows the opposite pattern.Native-language cues keep reasoning in the target language at greater than 97% across languages, but toxicity outcomes still vary by language.Source: §4.2 · p. 7, Table 3 · p. 7
  • Native-language prompts improve CulturalBench-Hard scores in South Europe (+1.0% on average) and Oceania (+2.9% on average), but regional performance is not uniformly higher.Across models, the highest mean performance is in South Asia (57.3%); DeepSeek Distills and Qwen underperform other models in East Asia by 2.6 percentage points.Source: §4.2 · pp. 7–8, Figure 3 · p. 8

5. Reasoning Pattern Analysis

The authors develop a two-stage method for segmenting and classifying reasoning steps, then examine how reasoning habits relate to language prefilling and accuracy. Their analyses associate Chinese prefilling and Subgoal setting with higher MATH-500 accuracy.

5.1. Segmentation-Classification Method

  • The analysis segments reasoning chains into operational steps, then classifies each step using four reasoning habits and an “Others” category for steps that do not fit.The taxonomy includes Subgoal setting, Backtracking, Verification, and Backward chaining; “Others” is intended to prevent distortion from forced classification.Source: §5 · p. 8, §5.1 · pp. 9–10
  • A ModernBert-large token classifier trained on GPT-4o-annotated reasoning chains achieved a 95% F1 score for identifying step boundaries.The first stage adds <sep> tokens between distinct operations and trains a classifier to predict whether each sentence completes a reasoning step.Source: §5.1 · p. 9

5.2. Reasoning Behaviours and Performance across Models and Linguistic Contexts

  • Chinese prefilling correlates with Subgoal setting (r=0.50, p<0.001) and Verification (r=0.41, p<0.001), while Swahili correlates negatively with Subgoal setting (r=-0.35, p<0.01).The authors hypothesize that language-specific tokens activate culturally embedded problem-solving schemas, an interpretation they align with bilingual problem-solving studies [1].Source: §5.2 · p. 10
  • Models using Subgoal setting, predominantly triggered by Chinese prefilling, achieved 7.3% higher MATH-500 accuracy than models using other dominant strategies.The authors suggest selecting prefill languages strategically to encourage reasoning approaches suited to particular tasks.Source: §5.2 · p. 11
  • English correlates with Backward chaining (r=0.30, p<0.015), and the reported reasoning signatures persist across model architectures.The authors associate English-prefill reasoning with proof-based reasoning, but present this as an interpretation of the observed signature.Source: §5.2 · p. 11

6. Related Work

Prior work examines the faithfulness and reuse of verbalized reasoning chains, and proposes hub-language accounts of multilingual processing. The paper situates its analysis of LRMs within these discussions.

6.1. Chain-of-Thought Analysis

  • CoT resources support systematic evaluation [14], while studies report that reasoning chains may omit shortcuts [2] and that models reuse structural reasoning patterns through “concept vectors” [13].Source: §6.1 · p. 11

6.2. Hub Languages and Reasoning in Multilingual LLMs

  • Hub-language research traces the idea to cross-language retrieval, proposes shared semantic representations [20], and reports English-aligned processing [17] and performance benefits from English translation.The paper positions its work as an analysis of reasoning in LRMs and the effects of forcing particular reasoning languages.Source: §6.2 · p. 11

7. Conclusion

LRMs predominantly prefer hub languages such as English regardless of input language, and reasoning-language choice has an asymmetric effect across tasks. Non-hub reasoning can hurt low-resource reasoning performance, while native-language reasoning can benefit cultural tasks.

7. Conclusion

  • LRMs predominantly prefer hub languages such as English regardless of the input language.This preference persists despite the models' multilingual ability.Source: §7 · p. 11
  • Forcing non-hub reasoning degrades performance in low-resource languages, whereas hub-language reasoning improves or maintains reasoning-task performance; native-language reasoning can benefit cultural tasks.The conclusion emphasizes the importance of considering internal reasoning language for more inclusive future models.Source: §7 · p. 12

8. Limitations

The study covers eight languages and medium-scale LRMs below 30B parameters, limiting how broadly its findings represent global languages and larger models.

8. Limitations

  • The study covers only eight languages, which may not represent global linguistic diversity, especially extremely low-resource languages.Its reasoning analysis also relies on LLM annotators and a coarse four-habit taxonomy, and identifies correlations rather than causal relationships; the models studied are below 30B parameters.Source: §8 · p. 12

Appendices

The appendices detail the evaluation and analysis procedures and report additional benchmark findings. They also describe constraints in the language-control and reasoning-classification methods.

A.1. Inference Cost

  • Model inference used sglang, mostly on A100 GPUs; experiments took over two weeks, with reported costs of about $600 for QwQ-32B and $1,200 for the other listed models.The cost estimates use an A100 rate of $1.80 per hour per card.Source: A. Model Details · p. 15, §A.1 · p. 15

B.1. Segmentation Details

  • The segmentation pipeline uses GPT-4o-labeled reasoning steps to train a ModernBERT separator classifier, whose selected hyperparameters achieve an F1 score of 96.08.Reasoning examples were collected from several models and datasets; the segmentation task predicts whether each <sep> begins a new step or continues the current one.Source: §B.1 · p. 16

B.2. Reasoning Process Classification

  • The classification pipeline labels segmented reasoning steps with Gemini 2.0 Flash using a four-habit taxonomy because more complex taxonomies exceeded current LLM classification capabilities.The authors say this complexity limited potential downstream insights.Source: §B.2 · p. 19

B.3. Comparison of Segmentation-Classification and Prompt-base Counting Method

  • Segmentation-classification produces lower behavior numbers than the counting method on the reported comparison.The appendix compares seg-class with the prior-work counting method [5] on MATH-500.Source: §B.3 · p. 21, Table 7 · p. 21

C. Limiting Tokens to Control Output Language

  • Token masking limits output to identified target-language tokens; the Llama 3 tokenizer has 4,225 Chinese-related tokens and 1,410 Japanese-related tokens.The authors warn the smaller Japanese token pool may limit results and describe masking as a limited way to control reasoning language.Source: C. Limiting Tokens to Control Output… · p. 22

D.1. Distribution Found From MATH-500 Baseline

  • The MATH-500 baseline analysis records the first four generated tokens per model-language pair and tracks their sequence frequencies, with decoding used to inspect readable phrases.The appendix also lists preferred CulturalBench prefill phrases by country.Source: §D.1 · p. 23, §D.2 · pp. 24, 26, Table 10 · p. 26

F.1. MMMLU full models breakdown

  • Swahili MMLU reasoning in Swahili rather than English drops by over 36% for QwQ-32B.The comparison is among native-language and English reasoning settings.Source: §F.1 · p. 28, Table 12 · p. 28

F.2. Scores in subset versus full set

  • 0.9953 is the correlation between subset and full MMMLU scores across settings, with p-value below 0.0001; all settings score higher on the subset.The authors interpret the subset as representative of the full test set.Source: §F.2 · p. 28, Table 13 · p. 29

G. CulturalBench Results

The appendices describe CulturalBench language controls and further tests of language prompting in a base model and an LRM. They also note dataset and language coverage details.

G. CulturalBench Results

  • CulturalBench keeps the original English questions while varying reasoning language, and its figures compare English prefills with each country's predominant-language prefills.The stated reason for retaining English wording is to preserve cultural nuance that translation could compromise; the figure captions describe positive native-language improvements relative to English for three models.Source: G. CulturalBench Results · p. 30, Figure 12 · p. 31

H. Study of Impact of Prefill Tokens in Pretrained Model

  • English-led reasoning significantly outperforms other language-led reasoning in the Qwen3-30B-A3B base-model AIME-2024 zero-shot pass@8 experiment.The experiment used a target-language opening phrase to guide the reasoning language.Source: H. Study of Impact of Prefill Tokens… · p. 32, Table 14 · p. 32
  • The authors hypothesize that RL training favors the language that supports effective chain-of-thought generation because reasoning language is not usually an explicit reward target.They motivate the hypothesis with the base-model finding that English-led reasoning outperforms other languages.Source: H. Study of Impact of Prefill Tokens… · p. 32

I. Brittleness of language guidance in LRM compared to typical CoT found in LLMs

  • Changing the reasoning language incurs a much larger penalty than changing language in ordinary chain-of-thought prompting with reasoning mode off.The comparison uses Qwen3-30B-A3B with reasoning mode on versus response prefilling with reasoning mode off.Source: §I · p. 33, Table 15 · p. 33

J. Dataset Details

  • The appendix identifies the benchmark licenses and test-split sizes and lists languages used in MMMLU, CulturalBench-Hard, LMSYS-toxic, and MATH-500.The listed evaluation languages vary by benchmark.Source: J. Dataset Details · p. 34, Languages · p. 34

Languages:

K. Behavior Results Detail for MATH-500

On MATH-500, reasoning behavior varies with both prefill and input language: Chinese is associated with subgoal setting and verification, while English and Swahili show distinct prefill associations.

K. Behavior Results Detail for MATH-500

  • For the MATH-500 dataset, Chinese (zh-CN) prefill tokens show a strong positive correlation with subgoal setting (r=0.50, p<0.001).Source: K. Behavior Results Detail for MATH-500 · p. 35
  • Chinese input correlates with subgoal setting (r=0.42, p<0.001) and verification (r=0.33, p<0.01).These results concern input language rather than language-specific prefill tokens.Source: K. Behavior Results Detail for MATH-500 · p. 35
  • Across prefill and input conditions, the reported associations indicate that linguistic context can systematically influence mathematical reasoning patterns.The analysis considers backtracking, backward chaining, subgoal setting, and verification.Source: K. Behavior Results Detail for MATH-500 · p. 35
Loading 2505.17407v1…