PaperlayerView exact revision on arXiv

Source-linked AI summary

XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answering

Iman Barati, Arash Ghafouri, Behrouz Minaei-Bidgoli

arXiv:2608.27481v1cs.CLcs.AI

TL;DR

Multilingual benchmarks often translate whole examples, leaving language-boundary failures inside multi-hop reasoning underexamined. XHotpotQA introduces a controlled, audited benchmark with role-specific language assignments and evidence graphs; across reader artifacts, mixed-language conditions show larger associated deficits than in the adapted selector.

  • Problem

    Existing multilingual QA resources often translate an entire example into one language, so they do not isolate language boundaries inside a reasoning chain.

  • Method

    XHotpotQA assigns languages independently across question, evidence roles, and distractors while preserving supplied candidates, sentence-level support labels, and an evidence-dependency graph.

  • Results

    Full question–evidence mismatch is associated with 10.25–15.79 lower reader F1, while the corresponding adapted-selector alignment gaps are small.

  • Takeaways & Limitations

    XHotpotQA provides role-aware diagnostics and modular evaluation for systems integrating evidence across languages under fixed candidate recall.

  • Takeaways & Limitations

    The benchmark is a translation-derived, English-centric stress test under fixed candidate recall, not a naturally authored corpus or full-Wikipedia retrieval evaluation.

Abstract

from arXiv · show

Knowledge-intensive multi-hop question answering requires systems to select evidence and compose dependent facts, yet multilingual benchmarks usually translate an entire example into one language. This hides failures at language boundaries inside the reasoning chain. We introduce XHotpotQA, a controlled benchmark for cross-lingual knowledge composition over mixed-language evidence. Each instance is modeled as an evidence-dependency graph whose question, bridge evidence, answer-bearing evidence, and distractors have explicit language assignments. The audited resource contains 15,661 training and 7,405 validation instances, with sentence-level support supervision and supplied distractors. In validation, 99.81% of items cross the question-to-gold-evidence language interface and 95.60% use gold paragraphs in different languages. Across three reader artifacts, full question-evidence mismatch is associated with 10.25 to 15.79 lower Unicode-aware answer F1 than partial alignment, and different-script evidence with deficits of 11.98 to 23.70 points; the corresponding adapted-selector contrasts are 1.71 and 1.78 points. Under this supplied-candidate design, the evaluated readers therefore show substantially larger condition-associated deficits than the selector. XHotpotQA provides role-aware diagnostics, modular evaluation, and an audited test bed for knowledge-based systems that must integrate evidence across languages.

1. Introduction

XHotpotQA addresses a gap in multilingual multi-hop QA by testing reasoning across mixed-language evidence roles rather than translating whole examples. It contributes a controlled, audited benchmark and diagnostics for measuring cross-lingual evidence selection and composition.

  • Motivation: Whole-example translation can hide failures when a reasoning chain crosses language boundaries between question, bridge evidence, and answer-bearing evidence.A system may succeed when question and evidence share a translated language yet fail to transfer the intermediate entity between differently translated hops.
  • Benchmark: XHotpotQA assigns languages independently to the question–answer pair, candidate paragraphs, and distractors while preserving evidence structure and supervision.Paragraph and sentence order, question type, supporting-fact indices, and answer supervision remain explicit for evaluating selection and composition.
  • Benchmark: The benchmark combines supplied distractors, sentence-level evidence supervision, 15,661 training instances, and 7,405 validation instances.This controlled resource transforms the HotpotQA supplied-candidate distractor task into a cross-lingual knowledge-composition setting.
  • Diagnostics: Its role-aware language geometry represents the question, bridge evidence, answer-hop evidence, and distractors with measurable cross-lingual conditions.The framework makes interfaces, scripts, diversity, and entropy descriptors available for diagnosis.
  • Findings: Modular selector–reader experiments find greater language sensitivity in cross-lingual composition than in supplied-candidate selection, while translate-train adaptation recovers much of the loss.The benchmark separates evidence selection from answer composition so these modules can be evaluated independently.
  • Scope and reproducibility: The release is audited, versioned, and provenance-aware, with explicit status boundaries because the resource is translation-derived and does not evaluate full-Wikipedia retrieval.The authors frame it as a controlled stress test under fixed candidate recall, not a naturally authored multilingual information-seeking corpus.

2. Related work

Prior multilingual QA benchmarks test transfer and retrieval, but often do not isolate language boundaries within multi-hop reasoning. XHotpotQA positions itself as a controlled, role-aware benchmark for mixed-language evidence composition with supplied candidates.

  • Cross-lingual QA benchmarks: Multilingual QA resources differ in training language, question–evidence interface, and language composition within the evidence.This terminology distinguishes cross-language transfer from crossing an evidence interface and from within-chain language mixing.
  • XHotpotQA positioning: XHotpotQA combines 24-language coverage, supplied distractor candidates, all 7,405 HotpotQA distractor validation questions, and sentence-level supporting-fact labels.It is broader than the controlled probe of Meng et al. but less ecologically realistic than naturally authored questions or open-corpus retrieval.
  • Formalization: The benchmark represents multi-hop instances with questions, candidates, supporting facts, answers, and an evidence-dependency graph linked by premise dependence.Intermediate conclusions can be entities, relations, numerical results, or propositions, while atomic fact identifiers preserve paragraph and sentence identity.
  • Evaluation design: Its modular evaluation separates reader answer composition from selector evidence selection and end-to-end prediction under fixed candidate recall.The reader receives annotated support sentences, whereas the selector receives all supplied candidates; therefore selection is not full-corpus retrieval.

4. Resource construction

XHotpotQA is constructed by translating HotpotQA questions, answers, and independently assigned candidate paragraphs while preserving source alignment and support annotations. Audits show the intended high-mixing validation regime, alongside explicit release and translation-quality boundaries.

  • Dataset composition: 23,066 base instances comprise 15,661 hard training examples and 7,405 validation examples with approximately ten supplied paragraphs per item.The resource records 55,830 supporting-fact annotation occurrences and 229,688 candidate-paragraph occurrences.
  • Scope and validity: The benchmark is a translation-derived, English-centric stress test under fixed candidate recall rather than a naturally authored corpus or full-Wikipedia retrieval evaluation.Translation can alter lexical relations, transliteration, cultural content, and answer aliases, making translation quality part of benchmark validity.
  • Mixing audit: Question-language counts range from 274 to 339 per validation language, but marginal uniformity does not ensure balance across ordered language pairs, scripts, or hop roles.The paper therefore uses role-aware descriptors to characterize validation geometry.
  • Transformation pipeline: Each instance assigns one language to the question and answer, independently assigns a language to every candidate paragraph, and preserves paragraph and sentence order.The pipeline also retains support indices, question type, difficulty, language codes, and transformation provenance.
  • Mixing audit: 99.811% of validation items cross the question-to-gold-evidence interface, while 95.598% have gold paragraphs differing in language.The realized rates closely track the intended IID expectations of 99.826% and 95.833%, respectively.
  • Release status: V1.1 freezes a reproducible one-view projection, whereas V2 contains 22,836 of 23,066 intended instances and is released as an audited incomplete snapshot.The missing V2 rows include source-triggered defects and clean-source training-generation omissions; a completed revision requires source-ID coverage and bilingual review.

5. Experimental protocol

The experimental protocol evaluates reader, selector, and joint behavior separately on the 7,405-instance validation split, using multilingual one-shot screening and adapted Llama modules. Metrics and audits are designed to distinguish composition, evidence selection, transfer, and translation-quality uncertainty.

  • Evaluation tasks: The reader receives annotated supporting sentences, the selector receives all supplied candidates, and the end-to-end system passes selected evidence to the reader.Reader results isolate composition under oracle evidence, while selector and joint results expose the evidence bottleneck.
  • Reader screening: The one-shot reader screen covers proprietary and open multilingual instruction models ranging from 7B to 405B parameters.Evaluated open families include Llama 3.1 Instruct, Qwen2 Instruct, Aya 23, and Gemma 2 Instruct.
  • Adaptation conditions: Four adaptation conditions compare direct prompting, English HotpotQA transfer, mixed-language translate-train modules, and rationale continuation.The xLlama reader and selector are initialized from Llama 3.1 8B Instruct and adapted on 15,661 mixed-language training instances.
  • Metrics: The evaluator applies Unicode NFKC, defined case folding, punctuation and whitespace normalization, and English-only article removal; Chinese, Japanese, and Thai use character tokens.It also emits macro-language, mismatch, entropy, candidate-diversity, and script aggregates.
  • Translation audit: The translation audit samples 120 units per language across 23 non-English languages and scores them with a source-referenced integer rubric from 0 to 100.V1 and V2 samples are independent, and the audit supports triage and comparison rather than bilingual certification.
  • Uncertainty: Bootstrap intervals quantify sampling uncertainty for frozen runs but exclude judge-model, prompt, endpoint, and training-seed uncertainty.Repeated XHotpotQA+ views require resampling clustered by source ID.

6. Results

The results show substantial reader sensitivity to cross-lingual evidence conditions, while selector adaptation yields larger historical within-task gains. End-to-end performance remains constrained by evidence selection and imperfect alignment across languages and scripts.

  • Reader performance: 50.82 one-shot reader F1 is the highest reported score, achieved by GPT-4o despite oracle supporting sentences.Llama 3.1 405B reaches 48.32 and GPT-4o mini 45.13.
  • Transfer and adaptation: 21.18 points is xLlama-Reader’s F1 gain over direct prompting, while English-task transfer is 3.75 points lower than direct prompting.The reported xLlama-Reader gain is 65.7% relative; English Bactrainus reaches 28.48 versus direct Llama 3.1 8B at 32.23.
  • Module comparison: 40.54 points is xLlama-Selector’s support-F1 gain, exceeding the reader’s 24.93-point gain within the matched comparisons.Support F1 rises from 43.69 to 84.23, while support EM rises from 20.27 to 57.42.
  • End-to-end composition: 57.42 support EM still leaves 42.58% of items without an exact evidence set, contributing to joint performance remaining below oracle-reader F1.The evidence-set gap limits pipeline reliability even when reader evidence is supplied or adapted.
  • End-to-end composition: 40.49 joint F1 is reached by the rationale-continued translate-train pipeline, while answer F1 reaches 51.62.Replacing both modules raises answer F1 to 50.97 and joint F1 to 40.12 before the continuation adds 0.65 and 0.37 points.
  • Language conditions: 15.79 points is the largest reported S4-versus-S2 reader-F1 deficit, compared with 1.71 points for the adapted selector.The reader deficits range from 10.25 to 15.79 points across artifacts; the selector contrast is much smaller.
  • Language conditions: 19.69 reader-F1 points are associated with matching the answer-hop paragraph language to the question language for bridge items.The corresponding associations are 14.43 points for GPT-4o mini and 21.51 for Qwen.
  • Selector diagnostics: 0.51 selector-F1 points is the association for having a distractor in the question language, with confidence intervals that include zero.Support-language distractors and candidate counts likewise show intervals including zero.

7. Discussion

The discussion argues that attaching language labels to evidence units enables role-aware diagnosis of cross-lingual reasoning. Under supplied candidates, readers are more language-sensitive than selectors, but the comparisons remain descriptive and bounded by the benchmark design.

  • 7.1. Implications for knowledge-based reasoning systems: Evidence-unit language vectors distinguish failures at question, bridge, answer-hop, and distractor interfaces that aggregate example-level labels cannot expose.The design supports actionable diagnosis for retrieval, prompting, and training.
  • 7.2. Modular evaluation: 51.62 answer F1 versus 40.49 joint F1 shows why answer-only evaluation can miss remaining evidence-support gaps.Sentence support reveals selector failure and supports error attribution, although exact labels may not be logically exhaustive.
  • 7.3. Cross-lingual composition is the principal observed sensitivity: The selector’s larger historical adaptation gain and the reader’s larger conditional language gaps compare different quantities, so they are compatible rather than contradictory.The first comparison concerns training regimes; the second compares inputs within frozen systems.
  • 7.3. Cross-lingual composition is the principal observed sensitivity: 10.25–15.79 reader-F1 points is the complete-mismatch deficit, versus 1.71 points for the adapted selector under supplied candidates.Different-script evidence produces 11.98–23.70 reader deficits but only 1.78 selector points.
  • 7.4. Model capability and diagnostic decomposition: No monotonic model-size rule appears across families, motivating separate tests of single-hop access, cross-language correspondence, and connected reasoning.Capability, lexical coverage, multilingual transfer, and answer-language control vary across systems.
  • 7.5. Scope and interpretation: Different-script evidence is associated with 11.98–23.70 fewer reader-F1 points than same-script evidence, supporting separate modeling of script and language family.The inventory spans ten writing-system groups and includes languages with non-whitespace metrics.

8. Limitations and threats to validity

The paper’s validity is bounded by translated data, incomplete controls and artifacts, uncertain metric behavior across scripts, and a supplied-candidate evaluation setting. Several analyses also rely on development data, single runs, or non-paired audit designs.

  • Data and translation: Translation-derived content inherits English Wikipedia’s entity and cultural distribution rather than native information needs.Translation quality can vary by language, script, entity type, and sentence length.
  • Assignment and provenance: Independent language assignment does not guarantee equal counts for every ordered question–bridge–answer triple.Role-balanced paired subsets are needed for fine-grained comparisons.
  • Assignment and provenance: No realized monolingual control exists: S0 has no observations and S1 has only 14 validation items.The contrasts therefore measure degrees and locations of mixing rather than a matched cross-linguality penalty.
  • Metrics across scripts: Exact match and token F1 can penalize valid aliases, morphology, segmentation, and transliteration across scripts.Character F1 and semantic review help but introduce their own biases, so per-language human analysis remains necessary.
  • Evaluation design: No hidden multilingual test set exists; the 7,405 validation instances support comparison but allow reported maxima to include development-set selection.A frozen held-out test subset or public evaluation server is proposed for future releases.
  • Statistical validity: Single-run training results are point estimates, while language-pair analyses require paired uncertainty, multiple seeds, and multiplicity correction.Holm’s procedure is given as an example correction.
  • Artifact coverage: Complete item-level files survive for three one-shot readers and one selector, leaving transfer and end-to-end tables as historical aggregate evidence.Absent responses are scored as empty in the surviving artifacts.
  • Evidence roles: Role-aware analyses cannot force bridge/answer-hop interpretations for every comparison question; an unordered category is required.Bridge and answer-hop order is determinable for many, but not all, bridge questions.

9. Conclusion

XHotpotQA formalizes cross-lingual multi-hop evidence composition with explicit linguistic structure and audited version boundaries. Its results show substantial reader associations with language-interface and script conditions, while the resource remains a controlled, translation-derived benchmark rather than a natural multilingual corpus.

  • XHotpotQA assigns languages independently to question–answer pairs and candidate paragraphs while preserving reasoning roles and source support keys.The resource separates question/evidence interface crossing from multilingual evidence composition and includes measurable mismatch, entropy, richness, and script descriptors.
  • 10.25–15.79 lower reader F1 is associated with full question–evidence mismatch, while different-script evidence is associated with 11.98–23.70 lower F1.The corresponding adapted-selector alignment gaps are small, and direct distractor-language contrasts have intervals spanning zero.
  • Higher paragraph and question judgments for the Gemma-generated V2 candidate yield no supported answer improvement.Lower Thai, Chinese, and Greek answer scores co-occur with more script-sensitive exact copies.
  • XHotpotQA is an English-centric, translation-derived stress test under fixed candidate recall, not a naturally authored corpus or full-Wikipedia retrieval evaluation.Its controlled scope makes language boundaries measurable, while translation and cultural audits remain central to responsible use.

CRediT authorship contribution statement

The statement records individual contributions, funding status, release materials, research-artifact review, and a worked multilingual example. The listed resources include public code and datasets, while the study reports no specific grant funding or applicable human-subject approval.

  • CRediT contributions: Iman Barati contributed conceptualization, methodology, data curation, software, investigation, formal analysis, and original drafting.
  • CRediT contributions: Arash Ghafouri provided supervision, and Behrouz Minaei-Bidgoli provided supervision and validation; all authors approved the manuscript.
  • Funding: The research received no specific grant funding from public, commercial, or not-for-profit agencies.
  • Data and code availability: The code repository, audited V1.1 dataset, Gemma-generated V2 candidate, and cross-linked judge samples are publicly available.Dataset files inherit CC BY-SA 4.0 from HotpotQA, while repository software uses MIT.
  • Ethics: Three specialists judged machine-translated text as expert assessment of research artifacts, with no participant characteristics collected.Formal research ethics approval and informed consent for publication of personal data were therefore not applicable.
  • AI assistance: OpenAI Codex supported drafting, language editing, and LaTeX restructuring; the authors reviewed the content and retained responsibility.
  • Worked example: The worked example demonstrates a Portuguese question paired with Swedish and Dutch gold paragraphs.The displayed source sentences illustrate the multilingual evidence setting used by the benchmark.

B.1. Normalization protocol

The normalization and audit protocol standardizes multilingual scoring while checking translation structure, language compliance, and source alignment. Results emphasize substantial reader metric sensitivity across scripts and numerical stability after excluding structurally flagged items.

  • Normalization: Unicode normalization and character tokenization increase aggregate F1 by 2.78–3.21 points while leaving EM nearly unchanged.The largest per-language F1 changes occur for Chinese, Japanese, and Thai, so comparisons are meaningful only within one protocol.
  • Metric sensitivity: Reader F1 ranges from 29.99–65.56 for Llama, 34.18–57.81 for GPT-4o mini, and 13.19–62.32 for Qwen, versus selector F1 of 81.21–87.53.
  • Validity dimensions: The protocol defines semantic, answer, entity/number, structural, and language-compliance preservation as five translation-validity dimensions.
  • Validation gates: Release checks verify Unicode validity, sentence cardinality, identifier stability, support bounds, content integrity, and question/answer language equality.Language identification, duplicate-content review, entity and number consistency, and answer occurrence belong to the versioned audit protocol.
  • Structural audit: 424 of 73,700 paragraph occurrences changed sentence cardinality, affecting 407 instances; 33 instances contained a blank translated sentence.
  • Sensitivity analysis: Excluding 439 flagged instances changes release-normalized F1 by only +0.10, −0.12, +0.11, and +0.24 across the three readers and selector.The largest tested EM/F1 change is 0.51 points, and the largest primary-contrast shift is 0.22 points with every paired shift interval including zero.

C.3. V2 RC1 completeness and source preservation

The V2 RC1 audit finds an incomplete but source-preserving snapshot whose omissions and structural anomalies require explicit attribution and correction. Future canonical-release claims require complete paired evidence, repaired defects, and bilingual adequacy checks.

  • Completeness: V2 RC1 contains 22,836 of 23,066 intended instances, or 99.00%, with exact English source-sentence arrays for every retained instance.The 230 absent instances prevent a complete row-level audit and a canonical-release claim.
  • Source preservation: The official HotpotQA source contains 137 blank-string sentence records and five out-of-bounds supporting indices, so not every downstream anomaly is a translation failure.
  • Anomaly attribution: Of 230 missing rows, 50 correspond to source records with blank sentences or out-of-bounds support annotations.Both missing validation rows are source-triggered, while clean-source omissions occur only in training.
  • Audit method: Source-ID reconciliation finds 230 absent IDs, superseding the incomplete retry ledger as the authoritative completeness count.
  • Release decision: V2 is suitable for public analysis as an audited incomplete snapshot but is not yet a complete replacement for V1.Completion requires restoring absent source IDs and preserving inherited defects without silent semantic substitution.
  • Acceptance target: A canonical V2 quality claim requires key equality, zero release-blocking structural defects, positive bilingual adequacy evidence, and no material preservation regression.The current independent GLM audit does not satisfy these paired-identification requirements.
  • Human audit design: The primary audit unit is the complete instance because independently fluent sentences can still translate a bridge entity inconsistently.Corrected text retains the original generation record and a versioned adjudication trace.
  • Scope boundary: The resource measures controlled linguistic transformations rather than naturally authored questions, community knowledge preferences, or practical code-switching.

D.2. Aggregate distributions

The audit reports descriptive aggregate and per-language score patterns, emphasizing ceiling effects, fixed weighting, and targeted anomalies rather than causal model comparisons.

  • Aggregate distributions: V2 answer medians and interquartile ranges remain at 100, while a small low-scoring tail increases answer SD.Short factual answers often receive ceiling scores, so the −1.29 mean difference does not represent uniform degradation.
  • Aggregate distributions: The per-language overall mean uses fixed 80:20:20 paragraph-question-answer weighting, making rows comparable within this audit but not population-weighted.Figure 8 indicates that the pooled gain is not homogeneous across languages.
  • Per-language results: Greek shows a −4.96 overall difference, with all three unit means decreasing amid mixed English intrusions and corrupted lexical material.The authors treat this as a high-priority bilingual audit stratum rather than a causal comparison.
  • Per-language results: Thai paragraph and question means increase by 3.54 and 7.65 points, while answers decrease by 30.70 points.Twelve of 20 RC1 Thai answers exactly copy English and receive a mean of 48.0.
  • Per-language results: Chinese paragraph and question means increase by 4.40 and 2.55 points, while answers decrease by 13.75 points.Three RC1 Chinese answers exactly copy English and average 43.33.
  • Answer-copy patterns: Exact source copies rise from 116/460 in V1 to 179/460 in V2 RC1, while their means fall from 99.05 to 89.44.Non-copy answer means rise from 94.27 to 97.21, making the split diagnostic rather than a validity rule.

E.2. Regression and uncertainty

The evaluation specifies regression, resampling, uncertainty, and complete-chain diagnostics while documenting prompt and endpoint boundaries that limit historical reproducibility.

  • Regression: A prespecified logistic model tests conditional associations for binary EM using mismatch and heterogeneity covariates.No-distractor items are omitted from terms involving 𝜌_D or modeled with an explicit missingness indicator.
  • Uncertainty: For bounded F1, the protocol prefers item bootstrap estimates clustered by source for repeated views.Pairwise system comparisons resample source IDs, while the source question—not the random seed—is the independent sampling unit.
  • Uncertainty: Confirmatory language and condition contrasts should use Holm correction, and trained or stochastic systems should use at least three seeds.The source question remains the independent resampling unit.
  • Chain diagnostics: The complete-chain gain compares both gold paragraphs with bridge-only, answer-hop-only, and empty-context inputs.A positive G_chain indicates benefit from both hops but does not prove faithful internal reasoning.
  • Reproducibility: The recovered V1 contract is exact at the message level but not at the endpoint level because the provider-resolved gpt-4o-mini revision and random assignment state were not retained.This constrains bitwise reconstruction of the historical generation process.
  • Translation protocol: V2 RC1 uses a stricter translation instruction with Gemma 4 31B, requiring preservation of entities, numbers, dates, polarity, and sentence boundaries.The parser rejects prose, malformed JSON, empty translations, and changed sentence cardinality; exhausted requests are recorded as failures.

F.3. Source-referenced GLM judge prompts

This appendix records source-referenced translation-judge prompts, answer-specific scoring criteria, and recovered reader, selector, and rationale-teacher protocols with explicit historical boundaries.

  • F.3. Source-referenced GLM judge prompts: The paragraph and question translation judge assigns one integer score from 0 to 100 using source-candidate adequacy, terminology, fluency, and style criteria.The rubric weights adequacy and faithfulness at 60%, terminology/entities/numbers at 15%, fluency at 20%, and style at 5%.
  • F.3. Source-referenced GLM judge prompts: Short-answer judging uses a separate instruction because names, dates, numbers, places, and yes/no responses require different preservation criteria.The judge receives the English question and answer alongside the translated answer.
  • F.3. Source-referenced GLM judge prompts: The audit retains the historical label “Critical error floors” but analyzes these rules as score ceilings because each specifies a maximum.The requested glm-5.2 alias used temperature zero, and the parser read the integer from the final SCORE: line.
  • F.4. Reader and selector prompts: Table 23 records repository identifiers for the historical reader screen, while the exact checkpoint behind the Llama 3.1 70B endpoint label could not be recovered.That endpoint is therefore treated as an artifact label in the article.
  • F.4. Reader and selector prompts: The cross-lingual reader is instructed to answer from multilingual context in the question’s language and to trace cross-language connections when needed.Complete one-shot variants preserve distinct demonstrations and message formatting across model artifacts.
  • F.4. Reader and selector prompts: The evidence selector uses the question and multilingual candidate set to predict supporting paragraph and sentence identifiers.The released evaluator uses stable identifiers rather than translated-title string matching.
  • F.5. Rationale-teacher prompt: The rationale teacher generates a step-by-step explanation linking question, context, and gold answer, but its provider revision and complete tokenizer/trainer state were not retained.The recovered serialization is reported as a historical protocol rather than a bitwise reconstruction.
Loading 2608.27481v1…