PaperlayerView exact revision on arXiv

Source-linked AI summary

Long Chain-of-Thought Reasoning Across Languages

Josh Barua, Seun Eisape, Kayo Yin, Alane Suhr

arXiv:2508.14828v3cs.CLcs.AIcs.LG

TL;DR

Long-form reasoning remains poorly understood outside English, especially when models must reason in users’ languages rather than only understand non-English inputs. This paper evaluates that transfer across nine languages and four development stages, finding that multilingual pretraining and targeted translated traces improve cross-lingual performance, while target-language reasoning still lags substantially.

  • Problem

    The paper addresses the limited evidence on whether long chain-of-thought reasoning transfers beyond English and whether models can accurately reason in users’ native languages.

    Source: §1 · p. 1

  • Method

    The study compares En-CoT and Target-CoT across nine non-English languages while examining scaling, pretraining, post-training, and inference.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Target-CoT remains far behind English at large scales, while broad multilingual pretraining and translated reasoning traces improve multilingual performance; translated traces generally outperform distilled target-language traces.

    Source: §1 · p. 2, §6.2 · p. 7, §8 · p. 10

  • Takeaways & Limitations

    Seamless multilingual reasoning requires targeted interventions across model development rather than relying on cross-lingual transfer from English alone.

    Source: §8 · p. 10

Abstract

from arXiv · show

While large reasoning models have shown remarkable ability to generate long chains-of-thought (CoTs) in English, we still lack understanding of how these long-form reasoning abilities transfer to the vast majority of the world's languages. In this work, we systematically investigate four key stages of model development--scaling, pretraining, post-training, and inference--to understand how long CoT capabilities extend beyond English. We compare two reasoning settings across nine non-English target languages: En-CoT, where models process target-language inputs, but reason in English; and Target-CoT, where models both process inputs and generate long CoTs in the target language. We find that scaling reasoning model size improves multilingual task performance in En-CoT, but Target-CoT performance lags behind. This gap widens for tasks requiring long, multi-step CoTs such as mathematical reasoning. Shifting to pretraining, we find that adding a specialized reasoning stage enhances En-CoT performance but degrades Target-CoT, whereas broad multilingual pretraining improves both modes simultaneously. Given the scarcity of high-quality reasoning traces in languages other than English, we explore synthetic data curation approaches for post-training. We demonstrate that fine-tuning on reasoning traces automatically translated from gold English traces outperforms fine-tuning on target-language traces distilled from large reasoning models. Finally, we report disparities in inference efficiency between languages and uncover language-specific failure modes in CoTs. We release models, datasets, and code to foster further research.

1 INTRODUCTION

This work studies whether long chain-of-thought reasoning transfers beyond English, distinguishing understanding non-English inputs from reasoning in users’ languages across model-development stages.

  • The paper investigates long chain-of-thought reasoning across nine non-English languages and four development dimensions: scaling, pretraining, post-training, and inference.Source: §1 · pp. 1–2
  • Figure 1 illustrates the gap: Swahili En-CoT reaches a correct answer, whereas Swahili Target-CoT contains a reasoning error and answers incorrectly.Source: Figure 1 · p. 2
  • En-Only uses English inputs and reasoning, En-CoT uses target-language inputs with English reasoning, and Target-CoT uses target-language inputs and reasoning.Source: §1 · p. 2, Figure 1 · p. 2
  • Target-language reasoning remains far behind English even at large scales, while translated reasoning traces provide stronger multilingual gains than large-scale English post-training.Source: §1 · p. 2

2 RELATED WORK

Prior work established benefits from long CoT and English-pivot multilingual reasoning, but mostly studied short chains or English reasoning rather than direct non-English long-CoT transfer.

Long CoT Reasoning

  • Long-CoT methods use extended intermediate reasoning, enabling complex behaviors and expert-level performance on challenging programming and STEM benchmarks.Source: §2 · p. 2

Multilingual Reasoning

  • Earlier multilingual studies found English CoT stronger than target-language reasoning on grade-school mathematics, motivating English as an external or internal pivot.Source: §2 · pp. 2–3
  • Recent long-CoT transfer studies report benefits mainly for high-resource languages, while other work identifies language-compliance trade-offs and language mixing in non-English reasoning.Source: §2 · p. 3
  • This paper extends prior work by examining cross-lingual long-CoT transfer across all stages of the model-development pipeline.Source: §2 · p. 3

3 EXPERIMENTAL FRAMEWORK

The experimental framework separates target-language comprehension from target-language reasoning and evaluates both across resource levels, task types, and model-development interventions.

Preliminaries on Long CoT reasoning

  • Long CoT training typically combines supervised fine-tuning on reasoning traces with reinforcement learning, but non-English traces are scarce and reward shaping can harm language compliance.Source: §3 · p. 3

Reasoning setup

  • En-CoT measures target-language comprehension through English reasoning, while Target-CoT additionally tests whether reasoning can be sustained in the target language.Source: §3 · p. 3

Languages

  • The study covers nine languages grouped as three high-, three mid-, and three low-resource languages according to mC4 representation.Source: §3 · p. 3

Multilingual benchmarks

  • Evaluation uses mathematical benchmarks MATH-500 and AIME-Combined plus the multilingual general-reasoning benchmark MMLU-ProX.Source: §3 · p. 3

Experiments

  • The experiments examine scaling, pretrained-backbone differences, post-training with translated or distilled traces, and inference efficiency and failure modes.Source: §3 · p. 4

4 SCALING MODEL PARAMETERS

Scaling improves target-language comprehension more reliably than target-language reasoning: En-CoT approaches English performance in better-resourced languages, while Target-CoT remains substantially weaker.

Setup

  • The scaling study uses shared-backbone DeepSeek-R1-Distill models with the same post-training data and procedure, isolating parameter count as the primary changing factor.Source: §4 · p. 4

En-CoT performance

  • En-CoT nearly matches En-Only for high-resource languages and stays within roughly 10% for several mid-resource languages, but low-resource languages remain substantially below English at every scale.Source: §4 · p. 4

Target-CoT performance

  • These results indicate that scaling addresses comprehension more effectively than sustained structured reasoning in non-English languages, especially for lower-resource languages.Source: §4 · p. 4
  • At 32B parameters, Target-CoT accuracy is 28.8% lower on average than English reasoning, and all tested languages perform at or below the 7B En-Only baseline.Source: §4 · p. 4

5 MULTILINGUAL PRETRAINING

Pretraining reveals an asymmetry: specialized math reasoning data improves En-CoT but harms Target-CoT, whereas broad multilingual coverage strengthens both cross-lingual comprehension and reasoning.

5 MULTILINGUAL PRETRAINING

  • For high- and mid-resource languages, En-CoT performance increases and approaches En-Only performance with scale, while Target-CoT remains consistently lower, making target-language reasoning the bottleneck.Source: Figure 2 · p. 5
  • Specialized math pretraining raises En-CoT by 5.6% on average on AIME-Combined but lowers Target-CoT by 4.3%.On MATH-500, Target-CoT drops include 46.0% for French, 39.4% for Afrikaans, and 36.4% for Chinese.Source: §5 · p. 5
  • Broad multilingual pretraining recovers 85–97% of English performance for En-CoT while also improving Target-CoT transfer.Qwen3-8B-Base and Gemma3-12B-PT show higher English Performance Recovered than the Qwen 2.5 models.Source: Table 1 · p. 6, §5 · p. 6
  • Broad multilingual pretraining nearly closes target-language comprehension gaps in En-CoT, but Target-CoT English Performance Recovered remains only 35–65%.The results indicate that language coverage supports non-English input understanding more readily than long, structured target-language reasoning.Source: §5 · p. 6

6 SYNTHETIC DATA FOR POST-TRAINING

The post-training study compares translated and directly distilled target-language reasoning traces under severe data scarcity. Translation is generally stronger, while 1,000 target-language traces are especially valuable for mid- and low-resource languages.

6.1 EXPERIMENTAL SETUP

  • The study constructs Translated-s1k from 1,000 English traces and Distilled-s1k from 1,000 language-forced target-language traces, evaluating both on mathematical and general reasoning.Separate Qwen3-8B-Base models are fine-tuned for each dataset-language combination, with compute-equivalent validation reported separately.Source: §6.1 · p. 7, §6.2 · p. 7

6.2 RESULTS

  • Translated reasoning traces generally outperform distilled traces, with average gains of 24.2% in Chinese and 9.2% in French across three benchmarks.Most other languages improve by 1–4% on average, while Marathi is the sole language where Distilled-s1k wins.Source: §6.2 · p. 7
  • Mid- and low-resource languages achieve competitive performance with 1,000 target-language traces, 20× fewer than the 20k English baseline.The minimal target-language fine-tuning setup surpasses Qwen3-8B across Marathi, Telugu, and Swahili.Source: §6.2 · p. 7
  • English-only fine-tuning remains slightly better for high-resource French and Japanese, whereas target-language data becomes essential for mid- and low-resource languages.The preferred data strategy therefore depends on the target language’s resource level.Source: §6.2 · p. 7

7 ANALYSIS OF LONG COT INFERENCE

Inference analysis shows that multilingual long-CoT performance involves both efficiency and error-profile trade-offs. Target-language fine-tuning narrows cross-language efficiency gaps, but Target-CoT introduces more generation and conceptual errors.

Inference Efficiency

  • Translated target-language fine-tuning narrows inference-efficiency gaps across languages, especially benefiting low-resource languages.Accuracy and token cost are strongly negatively correlated in both settings, with r=−0.824 and r=−0.915.Source: §7 · pp. 7–8, Figure 3 · p. 8
  • Token-based efficiency gains do not fully measure information efficiency because language tokenization differs; byte-based correlations are weaker at r = −0.155 and r = −0.355.Thus, token cost is directly tied to computation but only imperfectly captures information content across languages.Source: §7 · p. 8

Error Analysis

  • En-CoT errors are dominated by reasoning mistakes at 47.6%, whereas Target-CoT has 34.4% reasoning errors, 11.3% output-generation errors, and 24.9% conceptual errors.Problem-comprehension and calculation errors remain relatively stable across reasoning languages.Source: §7 · p. 9, Figure 4 · p. 9
  • Telugu’s conceptual errors double from 14% in En-CoT to 28% in Target-CoT, while Latvian comprehension errors remain near 34% in both settings.These language-specific patterns show that Target-CoT can add generation instability and conceptual barriers before full reasoning begins.Source: §7 · p. 9

8 CONCLUSION

The conclusion consolidates evidence that multilingual long-CoT reasoning requires targeted interventions beyond English transfer. The study spans nine languages, multiple model families, controlled evaluation, language forcing, compliance checks, and a five-category error taxonomy.

8 CONCLUSION

  • Larger models do not necessarily produce stronger multilingual reasoning: a 7B model reasoning in English can outperform a 32B model reasoning in another language.The conclusion identifies target-language reasoning as a persistent bottleneck despite scale.Source: §8 · pp. 9–10
  • The paper concludes that cross-lingual transfer from English alone is insufficient and releases models, datasets, and code for further research.Its evidence instead supports targeted interventions across scaling, pretraining, post-training, and inference.Source: §8 · p. 10

A.1 LANGUAGES

  • The study evaluates high-, mid-, and low-resource languages across multiple model scales and architectures, including DeepSeek-R1-Distill models from 1.5B to 32B parameters.Additional validation uses DeepSeek-R1-Distill-Llama-70B and Llama-3.3-Nemotron-Super-49B.Source: §A.2 · p. 16

A.4 LANGUAGE FORCING.

  • Language forcing seeds generation with translated phrases and system prompts, while the error taxonomy separates five observed failure categories.The compliance algorithm classifies 128-character chunks using script heuristics or Lingua and requires at least 90% target-language chunks.Source: §A.4 · p. 16, §A.7 · p. 17, §A.8 · p. 17

A.5 EVALUATION SETUP

  • Evaluation uses MATH-500, AIME-Combined, and MMLU-ProX with fixed decoding, language forcing, answer matching, and a 90% target-language compliance threshold.AIME-Combined uses five independent samples per problem and reports mean correctness over 300 generations.Source: §A.5 · p. 17, §A.7 · p. 17

A.9 ERROR LABELING

The appendix evaluates error labeling, language compliance, chain-length effects, scaling generalization, and compute-controlled synthetic-data training. Across these analyses, target-language reasoning remains most fragile on long mathematical chains, while translated target-language data improves performance under equal compute.

A.9 ERROR LABELING

  • Error analysis samples 100 incorrect language-compliant responses per language and setting, classifies them with a five-category taxonomy, and validates classifications through native-speaker review.Manual review covers 20 samples each for Chinese, Telugu, French, and Japanese.Source: §A.9 · pp. 17–18, Table 7 · p. 18

B.1 LANGUAGE COMPLIANCE RATES ACROSS MODELS

  • Language compliance is generally high, but Qwen2.5-Math-7B switches to English in French, Afrikaans, and Swahili, while two DeepSeek distill models frequently switch to Chinese in Japanese.After switching languages, these models predominantly produce incorrect answers; Qwen2.5-Math-7B reaches only 15.6%, 13.6%, and 5% accuracy in French, Afrikaans, and Swahili.Source: §B.1 · p. 18, Table 8 · p. 19

B.2 THE EFFECT OF REASONING CHAIN LENGTH ON PERFORMANCE GAPS

  • Short-CoT MMLU-ProX narrows the Target-CoT–En-CoT gap, whereas longer AIME-Combined chains widen it because sustained target-language reasoning remains brittle.Target-language reasoning even outperforms English in Chinese and French at 32B parameters on MMLU-ProX.Source: §B.2 · p. 18, Table 9 · p. 19, Figure 5 · p. 20

B.3 VALIDATION OF SCALING TRENDS ON LARGER MODELS AND DIFFERENT ARCHITECTURES

  • Target-language reasoning trails English reasoning on AIME-Combined even for larger models and different base architectures, with average gaps of 28.4% and 40.8%.Source: §B.3 · p. 19, Table 10 · p. 20

B.4 COMPARING ENGLISH AND TARGET-LANGUAGE DATA UNDER FIXED COMPUTE

  • The fixed-compute comparison controls data volume and distribution, showing that the apparent English-only advantage was driven primarily by its 20× data advantage rather than effective cross-lingual transfer.Source: §B.4 · pp. 19–20
  • When compute is fixed at 1,000 traces, Translated-s1k and Distilled-s1k outperform English-s1k across non-English benchmarks, with Translated-s1k achieving gains of 44.9%, 12.8%, and 21.2%.These gains are reported on MATH-500, AIME-Combined, and MMLU-ProX, respectively.Source: §B.4 · p. 20

B.5 VALIDATION OF POST-TRAINING RESULTS ON OLMO MODELS

The Olmo3-7B-Instruct replication tests whether compute-equivalent post-training findings extend beyond Qwen. Target-language synthetic training again outperforms English-only training, although distilled data is less consistent across languages.

B.5 VALIDATION OF POST-TRAINING RESULTS ON OLMO MODELS

  • The Olmo replication shows the same directional pattern as Qwen: target-language training substantially outperforms English-only training under equivalent compute budgets.Source: §B.5 · pp. 20–21, Table 11 · p. 21
  • Translated-s1k outperforms English-s1k across most benchmarks and languages, with average gains of 16.5% on MATH-500, 5.2% on AIME-Combined, and 1.2% on MMLU-ProX.Source: §B.5 · p. 21, Table 12 · p. 21
  • Distilled-s1k is more variable, underperforming on high-resource Chinese and French for MATH-500 and MMLU-ProX, so its benefits depend on the base model’s multilingual strengths.Source: §B.5 · p. 21

B.6 EVALUATION OF TRANSLATION CAPABILITIES

The study evaluates translation quality for synthetic reasoning data and remeasures inference efficiency using UTF-8 bytes. Gemini-2.0-Flash leads the translation comparisons, while byte-based costs are only weakly related to accuracy.

B.6 EVALUATION OF TRANSLATION CAPABILITIES

  • Gemini-2.0-Flash achieves the highest English-to-target translation scores across all evaluated FLORES-200 languages, outperforming NLLB-200-3.3B and MADLAD-400-10B.Source: §B.6 · p. 22, Table 13 · p. 22
  • Reference-free evaluation of 100 translated reasoning traces per language again favors Gemini-2.0-Flash over NLLB-200-3.3B, while MADLAD is omitted because it could not reliably translate long paragraphs.Source: §B.6 · p. 22, Table 14 · p. 22
  • The translation benchmark uses general-domain FLORES-200 prose, so it may not fully represent mathematical notation and symbols found in the synthetic reasoning traces.Source: §B.6 · p. 22

B.7 INFERENCE EFFICIENCY IN TERMS OF BYTES

  • Using UTF-8 bytes instead of tokens makes cost–accuracy correlations weakly negative, suggesting script and tokenizer effects contribute to apparent inference-efficiency disparities.Source: §B.7 · p. 22, Figure 6 · p. 23

Related papers

Loading 2508.14828v3…