A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages

Research Gap

• The study addresses unanswered questions about crosslingual thinking-trace consistency and faithfulness, especially outside English and in low-resource languages.Prior multilingual work has concentrated mainly on final-answer accuracy.Source: §1 · p. 1

Contributions

• The paper presents the first comprehensive multilingual CoT evaluation across performance, consistency, and faithfulness.It examines language compliance, answer accuracy, answer consistency, crosslingual trace interchange, and perturbation-based faithfulness.Source: §1 · pp. 1–2
• Language preferences and performance vary substantially when models are instructed or prompt-hacked to reason in different languages.The evaluation compares explicit instruction with prompt hacking and measures language compliance, accuracy, and answer consistency.Source: §1 · p. 1
• Crosslingual thinking-trace interchange reveals that trace quality and effectiveness depend strongly on the prompt language.The method tests whether traces generated in one language remain effective when substituted into another language.Source: §1 · pp. 1–2
• Perturbation tests show that models rely on thinking traces to different degrees across languages, so faithfulness is not uniform.The interventions include truncation and error injection.Source: §1 · p. 2

Faithfulness in CoT Reasoning

• Faithfulness studies question whether stated CoT reflects internal decision-making, but they evaluate this issue almost exclusively in English.This leaves multilingual faithfulness comparatively underexamined.Source: §2 · p. 2

Research Gap

• Existing multilingual work therefore provides evidence about outcomes and controllable traces, but not a comprehensive account of trace consistency and faithfulness across languages.The paper positions its evaluation as addressing this remaining scope.Source: §2 · p. 2

Evaluation of Multilingual Reasoning

• Multilingual CoT prompting improves performance across varied tasks, while language-specific reasoning controls and larger test-time budgets can further affect performance.These studies motivate evaluating both reasoning language and the resulting behavior.Source: §2 · p. 2

Models

• The evaluation spans DeepSeek-R1 distilled Qwen and Llama models from 1.5B to 70B, plus Qwen3 models at 8B and 32B.This provides coverage across model families and scales.Source: §3.1 · p. 2

Datasets

• MMMLU measures multilingual general knowledge across 15 typologically different languages and multiple domains in a multiple-choice format.Its domains include Humanities and STEM.Source: §3.2 · p. 2
• MGSM contains 250 grade-school math problems translated from GSM8K into 10 additional languages.The benchmark supports multilingual mathematical reasoning evaluation.Source: §3.2 · p. 3

Language Control

• The study aims to align thinking language with prompt language because users may prefer inspecting reasoning in the query language.Thinking language refers to text generated between <think> and </think>.Source: §3.3 · p. 3
• Explicit instruction directly asks the model to reason in a target language, but models may still default to their preferred language, typically English.This approach is therefore less reliable for enforcing the desired thinking language.Source: §3.3 · p. 3
• Prompt hacking inserts a short target-language prefix after <think> so the model continues the trace in the desired language.It is described as a more targeted language-steering method than explicit instruction.Source: §3.3 · p. 3

Evaluation Metrics

• Language compliance measures the proportion of thinking-trace sentences generated in the intended prompt language, while accuracy measures correctness independently for each language.The trace is the text between <think> and </think>.Source: §4.1 · p. 3

Evaluation

• Table 1 reports per-language final-answer accuracy with sentence-level language-compliance rates for evaluated LRMs on MMMLU under explicit instruction and prompt hacking.Additional MGSM and token-level compliance results are reported elsewhere in the paper.Source: §4.2 · p. 3, Table 1

Language Compliance and Accuracy

• Prompt hacking substantially improves target-language compliance but often reduces final-answer accuracy, creating a trade-off between language control and task performance.Explicit instruction is frequently ignored, especially for Bengali and Yoruba, whose compliance stays below 0.2 across models; prompt-hacked Yoruba achieves high compliance but lower accuracy than explicit instruction.Source: §4.2 · p. 4

Answer Consistency

• Final-answer consistency is generally higher for typologically similar language pairs, including English, German, and French.Average consistency for Indo-European pairs is significantly higher than for mixed Indo-European and non-Indo-European pairs.Source: §4.2 · p. 4, Figure 1 · p. 4

Language Compliance and Accuracy

• Performance remains higher for high-resource languages such as English and Chinese than for low-resource languages such as Swahili and Yoruba under both control strategies.These gaps persist despite changing how the target thinking language is enforced.Source: §4.2 · pp. 4–5

Interpretation

• The authors attribute persistent cross-language performance gaps to differences in thinking-trace quality and semantics, which motivate later consistency analysis.The conclusion is presented as a hypothesis supported by the observed multilingual disparities.Source: §4.2 · pp. 4–5

5.2 Substitution Consistency

• Substitution consistency measures how stable correct predictions in target language l remain after replacing its trace with one from language l′.The metric is an asymmetric intersection-over-union between correct-prediction sets before and after substitution.Source: §5.2 · p. 5

5.3 Results and Discussion

• Low-resource languages generally benefit from high-resource traces, while high-resource languages degrade when low-resource traces are injected.Under HackSub, Chinese accuracy drops to 0.40 with Telugu traces, whereas Telugu rises to 0.87 with Chinese traces; this pattern holds across substitution strategies.Source: §5.3 · pp. 5–6
• Substitution consistency is higher for typologically similar or resource-rich language pairs and lower for distant pairs.English–German substitution is relatively consistent, whereas Bengali–French reaches only 0.61 in BaseSub.Source: §5.3 · p. 6, Figure 3 · p. 6
• Thinking traces alone do not determine accuracy: identical low-resource traces can yield better performance with high-resource prompts.Swahili traces injected into English prompts achieve 0.33 accuracy, compared with 0.26 for Swahili’s original HackSub accuracy.Source: §5.3 · p. 6
• Models often use English traces more effectively than semantically equivalent Swahili or Telugu traces after translation.This pattern is attributed to greater pretraining exposure and English-heavy instruction tuning.Source: §5.3 · p. 6

6.2 Results and Discussion

• Models exhibit varying faithfulness across languages, with perturbations often having little impact in low-resource languages whose baseline performance is already low.Many other languages show substantial accuracy drops, indicating differing reliance on thinking traces; English has a matching score of 0.12 for R1-Qwen-32B.Source: §6.2 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 8
• Final-part truncation is less disruptive than injecting an incorrect value, especially for R1 distilled models.The contrast may reflect latent-state reasoning, whereas plausible incorrect conclusions are often copied directly.Source: §6.2 · p. 8
• Larger models are less dependent on final trace segments and more sensitive to earlier reasoning steps than smaller models.Smaller models rely more on the final portion, consistent with higher matching ratios and stronger surface-level faithfulness.Source: §6.2 · p. 8
• Qwen3 models show the opposite scale trend, possibly because of data contamination or stronger latent-state reasoning.The contamination hypothesis is motivated by their rarely changing answers when the final sentence is corrupted.Source: §6.2 · p. 8

• Across performance, consistency, and faithfulness, multilingual reasoning varies substantially with the language used for prompting and thinking.The study reports strong language preferences, inconsistent traces across languages, language-associated trace quality, and uneven reliance on traces.Source: §7 · p. 8

Limitations

• More sophisticated or adversarial perturbations, including paraphrasing and distractor reasoning, remain unexplored.Source: Limitations · p. 9
• The study evaluates multilingual inconsistencies but does not provide a mechanistic explanation for why they arise.Mechanistic interpretability is proposed as a direction for understanding multilingual inconsistency and faithfulness.Source: Limitations · p. 9
• The experiments are limited in the number of models and downstream tasks because of resource constraints.Future work could extend the framework to more models, languages, and tasks.Source: Limitations · p. 9

Complete Results

• Prompt hacking improves target-language alignment but generally reduces accuracy, especially in low-resource languages such as Swahili, Telugu, and Bengali.High-resource languages show relatively stable performance across explicit instruction and prompt hacking.Source: §A.1.1 · pp. 11–12, Table 5 · p. 12
• Larger models exhibit higher consistency across MMMLU and MGSM, while prompt hacking can reduce internal language consistency relative to explicit instruction.Source: §A.1.2 · p. 12
• Prompt hacking improves sentence- and token-level target-language alignment, but token-level compliance remains lower than sentence-level compliance.The difference is attributed to difficulty identifying language from individual tokens and to borrowings or quoted words.Source: §A.1.3 · p. 12
• Under explicit instruction, models often default to English reasoning traces, while prompt hacking shifts trace distributions toward the target language.Chinese traces also appear under explicit instruction but less frequently than English traces.Source: §A.1.3 · p. 12
• Interchanging traces shows that low-resource traces can lower high-resource-prompt accuracy, whereas high-resource traces can improve low-resource prompts.Substitution consistency is generally higher between similar languages, with HackSub following the BaseSub trend but sometimes lowering consistency.Source: §A.2 · p. 12

Consistency

• Across MGSM and MMMLU, larger models show higher final-answer consistency, although prompt hacking can reduce internal language consistency relative to explicit instruction.Source: Figure 5 · p. 13, Figure 6 · p. 14

Language Compliance

• Prompt hacking improves target-language compliance, with stronger effects for low-resource languages, but token-level compliance is consistently below sentence-level compliance.Borrowings and quoted words complicate language identification at the token level.Source: Table 6 · p. 13, Table 7 · p. 14
• Explicitly instructed models often produce English reasoning traces in non-English settings, while prompt hacking shifts the distribution toward the target language.Chinese traces also occur under explicit instruction but less frequently than English traces.Source: Table 8 · p. 15, Table 9 · p. 15

Crosslingual Trace Substitution

• Injecting low-resource traces into high-resource prompts lowers accuracy, while high-resource traces can improve low-resource prompts.Trace quality therefore varies across languages; substitution consistency is generally higher between similar languages, with HackSub following the BaseSub trend but sometimes lowering consistency.Source: Figure 7 · p. 16, Figure 8 · p. 17
• Indo-European languages generally achieve higher trace-substitution consistency than non-Indo-European languages, with statistically significant differences after excluding Swahili and Telugu.Source: Table 10 · p. 18

Faithfulness

• Removing the final part of a reasoning trace causes larger performance drops than truncating its beginning or middle.This result highlights the importance of concluding reasoning steps.Source: §A.3 · pp. 12, 18

Datasets

• MMMLU is limited to 250 randomly sampled test instances, while MGSM uses its official 250 parallel examples per supported language.MMMLU sampling is applied consistently across parallel language versions for comparability.Source: §B.2 · p. 18, §B.1 · p. 18

Prompting

• Language-specific instructions are appended after the query and <think> tag to reduce mismatches between the prompt language and intermediate reasoning language.Explicit prompts frequently produced reasoning in a different language, motivating this enforcement strategy.Source: §B.3 · p. 18

Experimental Settings

• 8192 tokens is the maximum generation length for all models.Source: C Experimental Environment and… · p. 18
• DeepSeek R1 distilled models use temperature 0.6 and top-p 0.95, while Qwen3 additionally uses top-k 20.Experiments primarily use NVIDIA A100 GPUs, with H200 GPUs for larger models such as DeepSeek-R1-Distill-Llama-70B.Source: C Experimental Environment and… · p. 18

Evaluation

• Final-answer correctness is evaluated by exact matching the extracted content inside the model’s \boxed{} answer against the gold answer.Source: C Experimental Environment and… · p. 18

Evaluation Materials

• Figure 9 documents the prompts used for the MGSM task across different languages.Source: Figure 9 · p. 19
• Table 11 reports absolute and relative performance drops against baseline under first, middle, and last truncation.Source: Table 11 · p. 20
• Tables 11–13 document truncation results, covered languages in MMMLU and MGSM, and language-specific answer instructions.Source: Table 11 · p. 20, Table 12 · p. 20, Table 13 · p. 20