Source-linked AI summary
Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages
Anaelia Ovalle, Candace Ross, Sebastian Ruder, Adina Williams, Karen Ullrich, Mark Ibrahim, Levent Sagun
TL;DR
Multilingual evaluations often measure reasoning through answer accuracy, leaving unclear whether model-generated reasoning actually supports conclusions across languages. The paper introduces a human-validated alignment framework and applies it to multilingual traces, finding that non-Latin scripts show at least twice as much misalignment as Latin scripts. These results show that answer accuracy alone provides an incomplete picture of multilingual reasoning quality.
Problem
Answer accuracy does not establish whether model reasoning traces support their conclusions across languages, limiting assessment of multilingual reasoning capabilities.
Method
The paper isolates reasoning traces from final answers, evaluates their supported conclusions with human and automated judgments, and measures Trace Inconsistency Rate across multilingual GlobalMMLU data.
Results
Reasoning–answer alignment drops over two-fold across resource levels, and non-Latin-script languages show twice as many reasoning contradictions as Latin-script languages.
Takeaways & Limitations
Multilingual reasoning quality should be evaluated separately from answer correctness because models can reach correct conclusions for reasons that do not support them.
Takeaways & Limitations
The evaluation uses GlobalMMLU, a small sample of Boolean comparative questions, limited human annotations, and one judge for reasoning traces.
Abstract
from arXiv · showhide
Large language models demonstrate strong reasoning capabilities through chain-of-thought prompting, but whether this reasoning quality transfers across languages remains underexplored. We introduce a human-validated framework to evaluate whether model-generated reasoning traces logically support their conclusions across languages. Analyzing 65k reasoning traces from GlobalMMLU questions across 6 languages and 6 frontier models, we uncover a critical blind spot: while models achieve high task accuracy, their reasoning can fail to support their conclusions. Reasoning traces in non-Latin scripts show at least twice as much misalignment between their reasoning and conclusions than those in Latin scripts. We develop an error taxonomy through human annotation to characterize these failures, finding they stem primarily from evidential errors (unsupported claims, ambiguous facts) followed by illogical reasoning steps. Our findings demonstrate that current multilingual evaluation practices provide an incomplete picture of model reasoning capabilities and highlight the need for reasoning-aware evaluation frameworks.
1 Introduction
Multilingual answer accuracy does not fully reveal whether models reason coherently across languages. The paper introduces a human-validated framework and finds disproportionate reasoning–answer misalignment, especially in non-Latin scripts.
- Motivation: Reasoning traces can support correct answers poorly, making answer accuracy an incomplete measure of multilingual reasoning capability.The same distinction also matters when models use sound reasoning but reach incorrect conclusions.
- Approach: The framework evaluates whether model-generated reasoning traces logically support their conclusions across languages using isolated traces and human annotation.It also develops an error taxonomy for recurring reasoning failures.
- Findings: 80% -> 70% answer accuracy declines from high- to low-resource languages, while reasoning–answer alignment drops over two-fold.Misalignment occurs for both correct and incorrect answers and varies significantly between Latin and non-Latin scripts.
- Findings: Alignment drops twice as much for non-Latin scripts as for Latin scripts, and misalignment occurs independently of answer correctness.These results indicate systematic differences in reasoning processes across languages rather than only knowledge gaps.
2 Related Works
Prior multilingual reasoning evaluations largely emphasize whether models reach correct answers, while research on reasoning faithfulness documents divergence between chain-of-thought traces and genuine problem solving.
- Multilingual reasoning evaluation: Multilingual reasoning benchmarks have largely focused on mathematical tasks where intermediate steps can be formally validated.This literature commonly equates reasoning evaluation with performance on reasoning-intensive tasks.
- Multilingual reasoning evaluation: Existing evaluations often judge reasoning quality by final-answer correctness rather than whether reasoning processes support stated conclusions.This motivates evaluating reasoning–answer alignment directly.
- Reasoning faithfulness: Prior faithfulness studies show that chain-of-thought traces can diverge from genuine problem-solving processes and function as post-hoc rationalizations.These shortcomings appear in both controlled counterfactual settings and real-world scenarios.
3 Methodology
The study evaluates reasoning–answer alignment by removing final answers from multilingual traces and testing which conclusions the reasoning itself supports. It combines human annotation, automated evaluation, and cross-language benchmarking.
- Evaluation framework: The framework back-translates non-English traces, removes their final answers, and asks evaluators to infer supported conclusions or mark traces inconclusive.This isolates reasoning content from the model’s stated answer.
- Evaluation framework: Trace Inconsistency Rate (TIR) measures the proportion of cases where reasoning fails to support the model’s conclusion, regardless of answer correctness.The protocol also assesses logical coherence and information sufficiency.
- Data and models: The benchmark covers GlobalMMLU fact-retrieval domains across six languages spanning higher- and lower-resource levels and Latin versus non-Latin scripts.The evaluation uses six instruction-tuned models from the Llama and Qwen families.
- Human evaluation: Human annotation first induces an inconsistency taxonomy from approximately 100 binary-task traces, then scales evaluation through stratified sampling across language, model, and cultural-sensitivity dimensions.The scaled protocol collects 72 annotations with two annotators per shard.
- Automated evaluation: Automated evaluation applies the human protocol to 66,615 GlobalMMLU examples, retaining instances with valid A–D predictions.Non-English responses are back-translated with Command-R+, with translation quality manually checked on a small sample.
4 Evaluation Validation
Human annotation supports the reliability of the evaluation framework, while automated evaluation achieves usable agreement with human judgments for scaling analysis. The study identifies systematic reasoning inconsistencies across languages and models.
- Human evaluation results: Human annotators show substantial agreement on answer selection, with κ = 0.7 and 80% agreement, while coherence and sufficiency receive lower but meaningful agreement.Logical coherence reaches κ = 0.39 with 69.7% agreement, and information sufficiency reaches κ = 0.53 with 79% agreement.
- Human evaluation results: Thematic analysis of 72 annotated instances identifies four primary categories of reasoning inconsistency across languages and models.The taxonomy is derived inductively from recurring patterns in model-generated traces.
- Automated evaluation validation: Automated evaluation agrees moderately with human consensus on inferred answer selection, with κ = 0.53 and 62.5% agreement.Agreement is lower for logical coherence and information sufficiency, with κ = 0.44 and 0.43, respectively.
- Automated evaluation validation: The study scales automated analysis primarily through answer selection because human–automated alignment is strongest on that dimension.This supports systematic analysis while preserving caution around more complex reasoning-quality judgments.
5 Results
Trace inconsistency exposes reasoning failures that answer accuracy alone obscures, especially across scripts and when answers are wrong. Evidential errors dominate the automated taxonomy, while extended reasoning improves consistency overall but may increase unsupported claims in low-resource languages.
- 5 Results: 13.3% Korean TIR despite 77% accuracy shows that correct-looking performance can conceal reasoning that does not support the conclusion.Qwen2.5-32B-Instruct exhibits 13.3% Korean TIR, while Qwen3-32B reaches 9.81% Korean TIR versus 2.16% in English.
- 5 Results: Latin-script languages outperform non-Latin scripts by 4–6 percentage points in both accuracy and TIR, twice the high- versus low-resource gap.The pattern holds across models, indicating stronger script-based differences than resource-availability effects in this study.
- 5.1 Trace Inconsistency for Error Diagnosis: When final answers are wrong, TIR rises substantially across languages and models, exceeding 25% for some Korean, Arabic, and Hindi cases.Even English and Spanish average 11% TIR when models are incorrect, and these inconsistencies may reflect model uncertainty useful for diagnostics.
- 5.2 Analysis of Automated Human-Derived Error Taxonomy: Unsupported claims and ambiguous facts account for 23% and 24% of detected errors, respectively, while illogical leaps account for 18%.The prevalence of evidential errors suggests greater difficulty grounding reasoning in verifiable information than maintaining internal logical consistency.
- 5.2 Analysis of Automated Human-Derived Error Taxonomy: Thinking models show better overall consistency, but unsupported claims substantially increase for low-resource languages, suggesting a possible extended-reasoning trade-off.The paper frames this as a hypothesis for further evaluation rather than a definitive causal result.
6 Conclusion
The paper introduces an automatic multilingual evaluation framework for reasoning-trace fidelity and finds that accuracy alone understates reasoning quality. Non-Latin-script languages exhibit twice as many reasoning contradictions as Latin-script languages, motivating reasoning-aware diagnosis and further study of thinking models.
- 6 Conclusion: Accuracy alone underestimates reasoning-trace quality because models with similar accuracy can differ in trace fidelity.The framework evaluates misalignment between reasoning traces and answers in multilingual settings.
- 6 Conclusion: Figure 2 compares taxonomy distributions across models for higher- and lower-resourced languages.The figure organizes the distributions with higher-resourced languages on top and lower-resourced languages below.
- 6 Conclusion: Non-Latin-script languages reflect twice as many reasoning contradictions as Latin-script languages.The finding motivates closer examination of whether thinking models maintain factual grounding in resource-constrained languages.
- 6 Conclusion: The evaluation framework is presented as a tool for diagnosing reasoning-trace errors and ultimately improving multilingual reasoning capabilities.This consequence is stated as a future-facing use of the framework.
Limitations
The evaluation is limited in dataset, question-type, annotation, judge, language, and translation scope. Its implementation also uses fixed generation and translation settings, while a small Spanish backtranslation check found only 2 of 30 examples marked for translation errors.
- Limitations: The study tests GlobalMMLU and only a small sample of boolean comparative questions, limiting the evaluated task scope.The framework may apply to other reasoning datasets, but those datasets were not tested here.
- Limitations: Human annotations cover a limited number of samples, and reasoning traces are assessed by one judge rather than majority voting across three judges.The paper identifies scaling annotation and using three-judge majority voting as future improvements.
- Limitations: The language set does not include languages even lower-resourced than those selected, although it was chosen to provide script diversity.The authors also suggest broader language coverage in future work.
- Limitations: The pipeline uses model-specific sampling where available, otherwise nucleus sampling with top-p=0.95, plus separate 8192-token translator and judge budgets.Backtranslation uses temperature 0 and top-p 1.
- Limitations: A small Spanish check found translation errors in 2 of 30 backtranslated examples.The check used one native Spanish speaker, and the paper proposes stronger translator models and additional native-speaker verification.
Reasoning Trace Error Example 1: Unsupported Claims
The trace selects Africa but supports the answer with claims about poverty and growth rather than explicit evidence for population growth. This illustrates how a correct conclusion can rest on unsupported reasoning.
- The reasoning rejects A, B, and C using claims about regional poverty and economic growth, but does not establish the population-growth comparison.
- The trace selects D, identifying Africa as having steady economic growth and the highest population growth rate.
- The evaluator identifies an unsupported claim because the trace provides no explicit data or evidence for Africa’s highest population growth rate.
AI Judge JSON Evaluation
The evaluation output records which options were mentioned and supported, while separately identifying unsupported claims and an inferred likely answer. The trace’s lack of detailed data leads it to rely on general inferences and uncertain estimates.
- The evaluator marks option D as supported while identifying Africa’s highest population-growth claim as unsupported.
- The trace relies on general inferences because it lacks detailed population and per-capita-income data.
- The analysis estimates a population share above 40% but states that exact determination requires more detailed data and analysis.
- The evaluator records A as the likely concluded answer and explains it through Africa’s poverty, economic growth, and population-growth claims.
Reasoning Trace Error Example 2: Ambiguous Facts and Multiple Answers
A second trace produces two plausible percentage answers without choosing between them. Its vague facts and absent exact data make the reasoning inconclusive.
- The trace also contains ambiguous facts, using terms such as “considerable” and “significant portion” without exact data.
- The evaluator identifies multiple answers because the trace ends with 40% and 55% without a clear preference.
D More results
The evaluation prompt separates answer inference from error identification and defines a taxonomy for unsupported, contradictory, ambiguous, linguistic, irrelevant, and other reasoning failures. It requires structured judgments about which options the trace supports and whether the reasoning is conclusive.
- Evaluation protocol: The protocol first infers the answer supported by the reasoning trace, then identifies errors separately.
- Error taxonomy: Illogical leaps are conclusions that do not follow from stated evidence or rely on implicit assumptions.
- Error taxonomy: The taxonomy includes contradictions, multiple answers, unsupported claims, ambiguous facts, linguistic errors, irrelevant content, and other uncategorized errors.
- Evaluation protocol: Evaluators mark which options are mentioned and supported, select an inferred answer, and use E when the conclusion is ambiguous or unknowable.
- Output format: The prompt instructs evaluators to return only the required JSON structure, including option-level support, identified errors, explanations, and the final answer.