PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Language Models Reason Across Languages?

Yan Meng, Wafaa Mohammed, Christof Monz

arXiv:2601.06644v2cs.CLcs.AI

TL;DR

This paper studies whether language models can reason across multilingual documents in controlled two-hop question answering. It finds language sensitivity and unfaithful decomposition, while structured SUBQ prompting improves averaged token-level F1 from 7.16% to 39.77%.

  • Problem

    The paper asks whether language models can synthesize information across multilingual documents when answering two-hop questions.

    Source: §1 · p. 1

  • Method

    The paper introduces controlled multilingual two-hop evaluations and a three-stage SUBQ prompting method using explicit decomposed sub-questions.

    Source: §1 · p. 2, §2.1 · p. 3, §5 · p. 9

  • Results

    Models are more sensitive to language changes in answer-span than bridging documents, and up to 33% of multilingual cases show unfaithful first-step reasoning despite correct final answers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Explicit structured sub-questions enhance robustness and reduce compositional failures in multilingual multi-hop reasoning.

    Source: §1 · p. 2, §5 · p. 9

    SUBQ is primarily diagnostic because it assumes oracle decomposed sub-questions, and the translated dataset covers only four high-resource languages.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The real-world information sources are inherently multilingual, which naturally raises a question about whether language models can synthesize information across languages. In this paper, we introduce a simple two-hop question answering setting, where answering a question requires making inferences over two multilingual documents. We find that language models are more sensitive to language variation in answer-span documents than in those providing bridging information, despite the equal importance of both documents for answering a question. Under a step-by-step sub-question evaluation, we further show that in up to 33% of multilingual cases, models fail to infer the bridging information in the first step yet still answer the overall question correctly. This indicates that reasoning in language models, especially in multilingual settings, does not follow a faithful step-by-step decomposition. Subsequently, we show that the absence of reasoning decomposition leads to around 18% composition failure, where both sub-questions are answered correctly but fail for the final two-hop questions. To mitigate this, we propose a simple three-stage SUBQ prompting method to guide the multi-step reasoning with sub-questions, which boosts accuracy from 10.1% to 66.5%.

1 Introduction

The paper introduces a controlled multilingual two-hop QA setting to test whether language models integrate evidence across languages and explicitly decompose reasoning steps. It finds uneven language sensitivity, substantial unfaithful and compositional reasoning, and improved results from structured sub-question prompting.

1 Introduction

  • The benchmark extends decomposed HOTPOTQA and MUSIQUE to French, Russian, Arabic, and Chinese, enabling step-level multilingual evaluation across distributed contexts.The task uses two-hop questions, Hop-1 and Hop-2 documents, and decomposed sub-questions for fine-grained analysis.Source: §1 · p. 1
  • Language variation degrades two-hop QA more when applied to answer-span documents than to bridging documents, despite both being needed for the answer.This controlled comparison motivates examining whether models rely on mechanisms that bypass faithful step-by-step reasoning.Source: §1 · p. 2
  • Up to 33% of multilingual cases are unfaithful: models answer the two-hop question correctly while failing the first bridging sub-question.Thus, correct two-hop QA does not require explicit solution of every intermediate step.Source: §1 · p. 2
  • Context attribution shows that bridging documents remain important in unfaithful cases, receiving at least one-third of attribution scores, while related distractors sharply reduce their performance.This suggests that models can use bridging information without explicitly identifying it in the first sub-question.Source: §1 · p. 2
  • Up to 18% of cases exhibit compositional failure, where models answer both sub-questions correctly but fail to produce the final two-hop answer.Faithful sub-question answering therefore does not guarantee successful composition.Source: §1 · p. 2
  • Three-stage SUBQ prompting substantially improves multilingual multi-step reasoning, increasing averaged token-level F1 from 7.16% to 39.77% by explicitly guiding final answer generation with decomposed sub-questions.The method substantially reduces compositional failures.Source: §1 · p. 2

2 Multilingual Two-Hop QA

The paper defines multilingual two-hop QA over two documents, then evaluates how language variation and reasoning depth affect performance. Cross-lingual settings degrade increasingly with more hops, with Hop-2 language producing the larger performance penalty and reasoning-related errors dominating failures.

2 Multilingual Two-Hop QA

  • The task requires identifying a bridge entity in Hop-1 before using Hop-2 to infer the target answer across potentially different languages.Inputs contain a two-hop question and two gold documents: Hop-1 provides bridging information and Hop-2 contains the answer span.Source: §2 · pp. 2–3

2.1 Experimental Setup

  • The evaluation extends decomposed HOTPOTQA and MUSIQUE, filters instances requiring every gold document, and translates them into French, Russian, Arabic, and Chinese.After filtering, the corpus contains 1,312 examples spanning 803 two-hop, 327 three-hop, and 182 four-hop questions.Source: §2.1 · p. 3

2.2 Results

  • Cross-lingual two-hop QA performance generally decreases when one or both document languages differ from the query language, and the gap persists as reasoning hops increase.Table 2 compares monolingual English documents with cross-lingual settings averaged over Chinese, French, Arabic, and Russian.Source: Table 2 · p. 3, §2.2 · p. 4
  • Reasoning-related errors, especially factual failures and imprecise answers, dominate cross-lingual degradation rather than final-answer translation errors.Wrong-language responses increase for Arabic, Chinese, and Russian queries, while factual failures increase mainly for English and French queries.Source: §2.2 · p. 4
  • −8.49 F1 is the average drop when changing Hop-2 language, versus −0.83 F1 for changing Hop-1 language across models.The pattern holds across all evaluated models, with drops ranging from −2.79 F1 for Gemma-3-27B to −13.43 F1 for Aya-Expanse-8B.Source: §2.2 · p. 4

3 Multilingual Reasoning Decomposition

The paper evaluates multilingual two-hop reasoning by decomposing questions into bridge-entity and final-answer sub-questions, then analyzes unfaithfulness and compositional failure. Models are more sensitive to Hop-2 language, can answer correctly without faithful SubQ1 reasoning, and struggle with distractors and document order.

3.1 Setup

  • The evaluation defines SubQ1 as extracting bridge entities from Hop-1 and SubQ2 as retrieving final answers from Hop-2.This decomposition enables separate tests of unfaithfulness and compositional failure.Source: §3.1 · pp. 4–5

3.2 Unfaithfulness

  • Up to 33% of multilingual two-hop answers are unfaithful because models answer correctly while failing the first-step bridge-entity sub-question.Arabic and Chinese show first-step unfaithfulness rates of 25% and 33%, respectively.Source: §3.2 · p. 5, §3.1 · p. 5

3.2.1 Context Attribution Analysis

  • In English two-hop questions, unfaithful cases show relatively higher Hop-1 attribution than faithful cases, despite failed explicit bridge-entity inference, indicating implicit contribution to correct two-hop answer generation.These relatively high scores suggest a more nuanced intrinsic reasoning mechanism beyond explicit step-level evaluation.Source: §3.2.1 · p. 5

3.2.2 Context Distractor Analysis

  • Topic-relevant distractors reduce multilingual reasoning more than irrelevant distractors, while reversing document order further worsens performance.These effects indicate sensitivity to semantically similar noise and the position of intermediate bridge information.Source: §3.2.2 · pp. 6–7

3.3 Compositional Failure

  • Up to 18% of incorrectly answered two-hop questions are compositional failures, where both sub-questions are correct but the final answer is wrong.This evaluates whether correct intermediate answers are successfully integrated into the overall response.Source: §3.3 · p. 7

Step 1 SUBQ

SUBQ prompting explicitly structures multilingual two-hop reasoning through sequential sub-question answers and a final synthesis step. It substantially outperforms zero-shot CoT, although generated sub-question errors remain on the full evaluation set.

Step 1 SUBQ

  • SUBQ first answers the bridge-entity sub-question, then uses that output for the target sub-question, and finally combines both steps for the two-hop answer.Ground-truth sub-question answers are also evaluated as an upper-bound comparison.Source: Step 1 SUBQ · p. 7, Figure 5 · p. 7
  • SUBQ is a diagnostic upper bound rather than a deployable method because it relies on gold decomposed sub-questions.Automatic decomposition is left for future work.Source: Step 1 SUBQ · p. 8

3.3.1 Results.

  • SUBQ prompting substantially improves multilingual two-hop QA over zero-shot CoT on compositional-failure cases and the full evaluation set.The method uses explicit decomposition to help models integrate information step-wise.Source: §3.3.1 · p. 8, Figure 6 · p. 8
  • On the full evaluation set, model-generated sub-question answers trail ground-truth guidance more than on compositional-failure cases.Nevertheless, SUBQ produces substantial gains across all query languages despite prediction errors.Source: §3.3.1 · p. 8, Figure 6 · p. 8

4 Related Work

Prior work studies multilingual capability and reasoning across tasks, but existing evaluations often lack controlled intermediate steps. This paper instead examines how models integrate linguistic information across languages in decomposed multi-hop questions.

Multilingual Capabilities

  • Prior multilingual reasoning work covers mathematics, coding, translation, question answering, and linguistic benchmarks, but often evaluates only final performance or a single inference step.The paper positions controlled multi-hop QA as a way to inspect intermediate reasoning.Source: §4 · pp. 8–9

Intrinsic Reasoning

  • Intrinsic-reasoning studies reveal latent two-hop paths or analyze attention, whereas this work evaluates reasoning through decomposed sub-questions.The distinction is whether multilingual reasoning aligns with explicit step-by-step question answering.Source: §4 · pp. 8–9

Failure Modes of Reasoning

  • Related failure-mode research covers unfaithfulness, premise order, distractibility, and long-context limits; this study extends those concerns to multilingual multi-hop reasoning.It combines long-context difficulty with multilinguality and examines unfaithful decomposed outputs.Source: §4 · p. 9

Effectiveness of Reasoning Decomposition

  • Reasoning decomposition is used in chain-of-thought, translation refinement, and multi-hop QA, and this paper extends decomposed sub-question evaluation to cross-lingual settings.The related work connects decomposition methods with step-wise reasoning strategies.Source: §4 · p. 9

5 Conclusion

The paper introduces controlled multilingual two-hop reasoning and finds that models are more sensitive to answer-span than bridging-document language variation. Explicit structured sub-questions improve robustness despite revealing non-faithful reasoning patterns.

5 Conclusion

  • The paper introduces a controlled setting for evaluating multilingual two-hop reasoning.The setting broadens multilingual evaluation while distributing information across languages.Source: §5 · p. 9
  • Models are more sensitive to cross-lingual variation in answer-span documents than bridging documents, although both are essential for the final answer.The paper also finds that explicit sub-question prompting helps mitigate compositional failures.Source: §5 · p. 9

Limitations

The paper frames SUBQ as a diagnostic tool for structured reasoning rather than a directly deployable method, and limits its analysis to four high-resource languages with oracle sub-questions.

Limitations

  • SUBQ is intended to diagnose whether language models benefit from structured, step-by-step reasoning, not to serve as a directly deployable method.Its applicability is limited because the experiments provide oracle-decomposed sub-questions, whereas real-world use requires question decomposition.Source: Limitations · p. 9
  • The translated dataset covers four high-resource languages and excludes low-resource languages to reduce the effect of training-data scarcity on cross-lingual reasoning.The selected languages still have less pre-training data than English but provide a controlled setting for analysis.Source: Limitations · p. 9

A Use of Large Language Models

  • The paper uses LLMs for dataset translation, manuscript grammar polishing, and error annotation, but not for research ideation, experimental design, or substantive scientific content.GPT-4o-mini, ChatGPT, and Claude-OPUS-4.6 are assigned these respective uses.Source: A Use of Large Language Models · p. 12

B Translation Quality Evaluation

The paper evaluates translation quality automatically and with native-speaker ratings, measures linguistic similarity through subword-vocabulary overlap, and matches instruction language to query language.

B Translation Quality Evaluation

  • Translation quality is assessed using reference-free COMETKiwi scores and human ratings of a random 20% sample for each target language.Native speakers of French, Chinese, Arabic, and Russian who also speak English rate the translations using a four-point scale.Source: B Translation Quality Evaluation · p. 12
  • Human ratings range from 3 for fully meaning-preserving, error-free translations to 0 for translations preserving little or none of the source meaning.Scores 2 and 1 represent mostly preserved meaning with minor errors and partially preserved meaning with fluency or grammatical issues, respectively.Source: B Translation Quality Evaluation · p. 12

C Linguistic Similarity Calculation

  • Linguistic similarity between two languages is computed as subword-vocabulary overlap on the 128-language NTREX multi-parallel corpus.The overlap score uses the intersection and union of the subword vocabularies induced for the two languages.Source: C Linguistic Similarity Calculation · p. 12

D Prompts

  • Across experiments, the instruction language is matched to the query language, with native speakers translating instructions for each target language.The paper presents these prompt templates in Figures 7 and 8.Source: D Prompts · p. 12

E Error Analysis

The error taxonomy distinguishes exact, partial, translation-related, factual, and language-selection outcomes for two-hop question-answering predictions.

E Error Analysis

  • An answer is correct when it exactly matches the gold answer, defined as token-level F1 equal to 1.The taxonomy also distinguishes partial answers, imprecise translations, factual errors, and wrong-language responses.Source: E Error Analysis · p. 12
  • Partial answers identify the correct answer but are incomplete or overly specific relative to the gold answer.Examples include naming only one member of a multi-person answer or giving only part of a date range.Source: E Error Analysis · p. 12
  • Imprecise translations preserve the underlying answer despite translation errors, whereas factual errors provide incorrect information in the query language.These categories separate language-form problems from incorrect content.Source: E Error Analysis · p. 12
  • Wrong-language responses are predictions expressed in a language different from the query language, regardless of factual correctness.This category isolates language-selection errors from answer-content accuracy.Source: E Error Analysis · p. 12

F Additional Model Results

Additional results are reported for Qwen-3-8B, Aya-Expanse-8B, and Llama-3.1-8B-Instruct.

F Additional Model Results

G Full Multilingual Two-hop QA Results

The evaluation examines multilingual two-hop question answering across query and document languages, using controlled settings and multiple models. The supplied materials define the prompting templates, dataset translation checks, and performance-table scopes, but do not report cell-level results.

G Full Multilingual Two-hop QA Results

  • The evaluation varies query language and the languages of both evidence documents in controlled multilingual two-hop QA settings.Figure 9 fixes the query language in each panel while varying the Hop-1 and Hop-2 document languages.Source: G Full Multilingual Two-hop QA Results · p. 12, Figure 9 · p. 16
  • The multilingual dataset evaluation reports translation quality using COMETKiwi and human ratings on a 0–3 scale, with both indicating preserved source meaning.Table 5 evaluates translated questions and documents against their English sources.Source: Table 5 · p. 13
  • The default prompt asks for a concise answer from the article without explanation, while zero-shot CoT adds a step-by-step reasoning instruction.The supplied prompt materials distinguish the default multilingual two-hop template from the zero-shot CoT template.Source: G Full Multilingual Two-hop QA Results · p. 13, Figure 7 · p. 13, Figure 8 · p. 14
  • Performance tables for Qwen-3-8B, Aya-Expanse-8B, and Llama-3.1-8B-Instruct report token-level F1 for matching and non-matching document languages.For non-matching documents, the tables report averages across the four non-query languages.Source: Table 6 · p. 14, Table 7 · p. 14, Table 8 · p. 15

Related papers

Loading 2601.06644v2…