PaperlayerView exact revision on arXiv

Source-linked AI summary

XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answering

Iman Barati, Arash Ghafouri, Behrouz Minaei-Bidgoli

arXiv:2608.27481v1cs.CLcs.AI

TL;DR

Multilingual benchmarks often translate whole examples, offering limited evidence about language boundaries inside multi-hop reasoning. XHotpotQA provides a controlled benchmark with explicit linguistic roles for mixed-language evidence. Across three readers, full question–evidence mismatch is associated with 10.25–15.79 lower reader F1 than partial alignment.

  • Problem

    Multilingual QA benchmarks often translate whole examples, leaving limited evidence about language boundaries between dependent facts in multi-hop reasoning.

    Source: §1 · pp. 1–2

  • Method

    XHotpotQA transforms HotpotQA’s supplied-candidate task into a controlled benchmark assigning languages to question–answer pairs and candidate paragraphs while retaining reasoning roles and support keys.

    Source: §1 · p. 2, §9 · p. 18

  • Results

    Across three readers, full question–evidence mismatch is associated with 10.25–15.79 lower reader F1 than partial alignment, and different-script evidence with 11.98–23.70 lower F1.

    Source: §9 · p. 18

  • Takeaways & Limitations

    XHotpotQA provides an audited test bed for evaluating knowledge-based systems that must integrate evidence across languages.

    Source: §9 · p. 18

    Validation lacks a realized monolingual control, so condition contrasts compare degrees and locations of cross-lingual mixing rather than cross-linguality against a matched monolingual baseline.

    Source: §8 · p. 18

Abstract

from arXiv · show

Knowledge-intensive multi-hop question answering requires systems to select evidence and compose dependent facts, yet multilingual benchmarks usually translate an entire example into one language. This hides failures at language boundaries inside the reasoning chain. We introduce XHotpotQA, a controlled benchmark for cross-lingual knowledge composition over mixed-language evidence. Each instance is modeled as an evidence-dependency graph whose question, bridge evidence, answer-bearing evidence, and distractors have explicit language assignments. The audited resource contains 15,661 training and 7,405 validation instances, with sentence-level support supervision and supplied distractors. In validation, 99.81% of items cross the question-to-gold-evidence language interface and 95.60% use gold paragraphs in different languages. Across three reader artifacts, full question-evidence mismatch is associated with 10.25 to 15.79 lower Unicode-aware answer F1 than partial alignment, and different-script evidence with deficits of 11.98 to 23.70 points; the corresponding adapted-selector contrasts are 1.71 and 1.78 points. Under this supplied-candidate design, the evaluated readers therefore show substantially larger condition-associated deficits than the selector. XHotpotQA provides role-aware diagnostics, modular evaluation, and an audited test bed for knowledge-based systems that must integrate evidence across languages.

1. Introduction

XHotpotQA targets a gap in multilingual multi-hop QA: whole-example translation can conceal failures at language boundaries within a reasoning chain. It assigns languages to evidence roles and distractors to make cross-lingual composition measurable under a controlled supplied-candidate design.

Introduction

  • Earlier multilingual resources test transfer and retrieval, but whole-example translation does not isolate language boundaries inside multi-hop reasoning chains.Prior work includes translated benchmarks and controlled two-document questions; the paper therefore frames its resource contribution specifically around role-specific language assignments, distractors, and sentence-level evidence labels.Source: §1 · p. 2
  • The benchmark preserves supplied distractors and sentence-level evidence supervision across 15,661 training and 7,405 validation instances.It retains paragraph and sentence ordering, answer supervision, question type, and supporting-fact indices while assigning languages independently to candidate titles and paragraphs.Source: §1 · p. 2
  • XHotpotQA models language across the question, bridge evidence, answer-bearing evidence, and distractors, making reasoning interfaces and role-aware language geometry measurable.The framework includes interface, script, diversity, and entropy descriptors.Source: §1 · p. 2
  • The authors report that evaluated selector–reader experiments find cross-lingual composition more language-sensitive than supplied-candidate selection, with translate-train adaptation recovering much of the loss.The benchmark is designed to measure evidence selection and answer composition jointly rather than relying on answer scores alone.Source: §1 · pp. 1–2
  • The release is audited, versioned, and provenance-aware, preserving source alignment and prompt contracts while marking the status of future revisions.The audit verifies a recovered positional join and identifies structural conditions that a corrected canonical release must repair.Source: §1 · p. 2
  • XHotpotQA is a translation-derived stress test for cross-lingual evidence selection and composition, not a naturally authored multilingual corpus or a full-Wikipedia retrieval benchmark.Its controlled scope fixes candidate recall and makes language boundaries measurable, while making translation and cultural audits important for responsible use.Source: §1 · p. 3

2. Related work · 3. Formal problem definition

XHotpotQA builds on connected-evidence approaches to multi-hop QA while making language assignments across questions, evidence roles, and candidate contexts explicit. Its formalization distinguishes cross-language interfaces from language mixing within evidence and defines separate tasks and measures for evaluating them.

2.1. Multi-hop question answering resources

  • Multi-hop QA work distinguishes connected evidence chains from merely presenting multiple documents, with HotpotQA, 2WikiMultiHopQA, and MuSiQue informing evidence and dependency supervision [26].Related methods extract sentence chains [6], retrieve iteratively through entities, or alternate retrieval and reasoning; XHotpotQA instead treats context language as an experimental variable.Source: §2.1 · p. 3

2.2. Multilingual and cross-lingual QA

  • XQuAD, MLQA, MKQA [18], XOR QA, Mintaka, and JEMHopQA represent varied multilingual and cross-lingual QA settings.The paper distinguishes model-training transfer, question–evidence interface crossing, and within-evidence crossing, which require different capabilities.Source: §2.2 · p. 3

2.3. Cross-lingual multi-hop and RAG benchmarks

  • XHotpotQA combines 24-language coverage, supplied distractors, all 7,405 HotpotQA distractor validation questions, and sentence-level support labels.It is larger and linguistically broader than Meng et al.’s 182-item controlled probe [20], but less ecologically realistic than naturally authored questions or open-corpus retrieval.Source: §2.3 · p. 3

2.4. Translation-derived resources and evaluation validity

  • Translation can distort lexical relations, entity forms, and answer aliases, while source-language cultural content persists; translation quality and label preservation therefore affect benchmark validity.The paper distinguishes its mixed-language composition and role/interface analyses from a companion English HotpotQA selector–reader study [4].Source: §2.4 · p. 4

3.1. Instances, facts, and evidence chains

  • Instances encode a question, answer, ordered candidates, supporting facts, dependency graph, and language map; graph edges represent premise dependence among questions, facts, intermediate conclusions, and answers [26].Stable fact identifiers remain separate from translated sentence forms; annotated chains are distinguished from necessary chains, and partial-evidence tests estimate necessity rather than prove it.Source: §3.1 · pp. 4–5

3.2. Reader, selector, and end-to-end tasks

  • The historical reader predicts answers from gold supporting sentences, the selector predicts support from supplied candidates, and the end-to-end task predicts both [10].Because candidate recall is fixed, the paper calls this task selection rather than retrieval.Source: §3.2 · pp. 5–6

3.3. Language map and cross-lingual conditions

  • Language labels apply to whole candidate paragraphs; interface crossing means gold evidence differs from the question language, while composition crossing means gold evidence is multilingual.Composition crossing alone does not establish that both paragraphs are necessary for a particular model.Source: §3.3 · p. 6

3.4. Evaluation measures

  • Evaluation defines language-aware answer exact match and token-overlap answer and support scores; joint metrics combine component precision and recall, with macro-by-question-language aggregation recommended [30].The present schema supplies one canonical answer, and joint F1 is not the product of answer F1 and support F1.Source: §3.4 · p. 7

4. Resource construction

XHotpotQA transforms HotpotQA distractor examples into mixed-language question-and-evidence instances with assigned languages, preserved support annotations, and supplied candidate paragraphs. Audits document the realized language mixing and distinguish the complete V1.1 resource from incomplete or prospective releases.

4.1. Source configuration and splits

  • The 23,066 base instances comprise 15,661 hard training examples and all 7,405 official distractor-development validation instances, with no hidden test set.Each item has approximately ten supplied paragraphs, normally two gold paragraphs plus distractors; the resource includes 55,830 supporting-fact annotation occurrences and 229,688 candidate-paragraph occurrences.Source: §4.1 · p. 8

4.2. Language inventory

  • The 24-language inventory spans ten scripts and several language families, and question-language counts are nearly uniform in validation but do not ensure balance across language pairs, scripts, or hop roles.Validation has 274–339 questions per language (mean 308.54, population SD 16.26, coefficient of variation 5.27%); the inventory includes languages that are not all low-resource, and resource-level labels are not inferred from language identity.Source: §4.2 · p. 8

4.3. Translation and assignment pipeline

  • The intended pipeline independently assigns a language to each candidate paragraph and translates the question, answer, paragraph titles, and ordered sentences while preserving sentence structure and support indices.The audited legacy generator used 24 question–answer translations per training source before an unseeded draw formed the historical base split; V1.1 freezes a reproducible one-view projection without claiming to reconstruct that lost draw.Source: §4.3 · pp. 8–9

4.4. Translator selection and model freeze

  • At the January 2025 experimental cutoff, GPT-4o and GPT-4o mini were the latest GPT variants considered for reproduction; historical V1 used GPT-4o mini, while the V2 release candidate used Gemma 4 31B Instruct [10].Source: §4.4 · p. 9

4.5. Expected language mixing under the design

  • 99.811% of validation items had gold evidence differing in language from the question, and 95.598% had gold paragraphs in different languages.These realized rates were close to the IID expectations of 99.826% and 95.833%, respectively; the design’s intended high-mixing regime was realized without anomalous assignment imbalance.Source: §4.5 · p. 9

4.6. Parallel views and the status of XHotpotQA+

  • The audited XHotpotQA+ archive contains all 375,864 training views, but lacks the 177,720 validation views needed for the proposed 553,584-view resource.Its fixed mixed-language candidate set is paired with all 24 question–answer translations; the paired design supports question-language effects, with uncertainty clustered by source_id.Source: §4.6 · p. 10

4.7. Release status

  • V1.1 retains all 23,066 instances, while the Gemma-generated V2 candidate has 22,836 and is released as an audited incomplete snapshot.V2 omits 230 rows: 50 source-triggered and 180 clean-source training-generation omissions. A completed revision requires source-ID coverage, explicit treatment of inherited annotations, and bilingual review; parallel validation for XHotpotQA+ remains future work.Source: §4.7 · p. 10

5. Experimental protocol

The experimental protocol separates evidence selection from answer composition, evaluates transfer and adaptation under specified reader and selector conditions, and applies distinct procedures for normalization, uncertainty, and translation-quality auditing. Its historical comparisons use legacy Hotpot normalization, while the audit intervals capture sampling variation rather than model or endpoint uncertainty.

5.1. Evaluation tasks

  • On 7,405 validation instances, readers receive gold supporting sentences, selectors score all supplied candidates, and end-to-end systems pass selected evidence to readers.Reader scores measure composition with oracle evidence; selector and joint scores expose the evidence bottleneck. Missing predictions in the three complete reader artifacts and adapted-selector artifact are scored as empty; other values remain historical aggregates.Source: §5.1 · p. 10

5.2. One-shot model screen

  • One-shot readers spanning proprietary and open multilingual models from 7B to 405B receive one demonstration and must answer in the question language using annotated supporting sentences.Open families include Llama 3.1 Instruct [12], Qwen2 Instruct [29], Aya 23 [2], and Gemma 2 Instruct [9]; the screen is a comparative baseline, not a bitwise reproduction, because immutable serving records are unavailable for every historical endpoint.Source: §5.2 · p. 10

5.3. Transfer and translate-train systems

  • Four adaptation conditions span direct one-shot Llama 3.1 8B, English HotpotQA-tuned Bactrainus transfer, and xLlama reader and selector adapted on 15,661 mixed-language instances.The xLlama-Reader also receives a continuation stage with English rationales from a Llama 3.1 70B teacher, so its associated gain does not isolate rationale quality from training duration or supervision.Source: §5.3 · pp. 10–11

5.4. Normalization and aggregation

  • Answer scoring uses Unicode-aware normalization and character-token F1 for Chinese, Japanese, and Thai, while historical baseline tables retain legacy Hotpot normalization.Output-language correctness requires separately versioned language identification or human audit; historical transfer values are single-run percentages, and bootstrap intervals for frozen RQ4 artifacts measure sampling uncertainty, not training-seed variation.Source: §5.4 · p. 11

5.5. Source-referenced LLM-as-judge audit

  • The source-referenced judge audit samples 2,760 units per version across 23 non-English languages, using 80 paragraph translations, 20 questions, and 20 answers per language.Independent V1 and V2 samples are not paired; GLM-5.2 [11] assigns source-referenced integer scores from 0 to 100, supporting triage and comparison rather than bilingual certification.Source: §5.5 · p. 11
  • Audit intervals use 20,000-replicate independent, within-stratum bootstrap resampling and fixed 80:20:20 paragraph/question/answer weighting, excluding judge, prompt, and endpoint uncertainty.The pooled difference interval quantifies sampling variation under this audit design only.Source: §5.5 · p. 12

6. Results

Across the supplied-candidate evaluations, translate-train adaptation improved reader and selector performance, while realized language mismatches were associated with larger deficits for readers than for the selector. The source-referenced audit found higher V2 RC1 paragraph and question scores, but not answer scores.

6.1. One-shot multilingual reader capability

  • GPT-4o achieved the highest one-shot reader F1 at 50.82, yet the best one-shot F1 remained below 51 despite oracle supporting sentences.This leaves evidence alignment across scripts and fact composition as reader challenges even when annotated evidence is supplied; the historical 8B screening and matched-transfer values differ slightly and are retained separately.Source: §6.1 · p. 12

6.2. RQ1: direct transfer versus translate-train adaptation

  • xLlama-Reader reached 53.41 F1, gaining 21.18 points over direct Llama 3.1 8B prompting and 24.93 over English Bactrainus transfer.Direct prompting scored 32.23 F1, while English-task tuning scored 28.48; EM rose from 21.94 to 39.85 with xLlama-Reader. These are historical comparisons, not artifact-level recomputations.Source: §6.2 · p. 12, Table 7 · p. 13

6.5. RQ3: limited incremental effect of teacher rationales

  • The rationale continuation reached 51.62 end-to-end answer F1 and 40.49 joint F1; its incremental gains were 0.65 and 0.37 points, respectively.Oracle-reader F1 rose from 53.41 to 55.62 (2.21 points), but training examples and duration changed simultaneously, so the causal effect of rationale supervision is unidentified.Source: Table 8 · p. 13, §6.4 · p. 13, §6.5 · p. 14

6.4. End-to-end composition

  • The best end-to-end answer F1 was 51.62, 4.00 points below the corresponding oracle-evidence reader F1 of 55.62.Replacing both English modules with translate-train variants raised answer F1 to 50.97 and joint F1 to 40.12; the rationale continuation reached 51.62 and 40.49. The answer–joint gap may reflect answer credit without exact sentence support, but aligned per-example outputs are needed to quantify such cases.Source: Table 8 · p. 13, §6.4 · pp. 13–14

6.3. RQ2: selection shows the larger observed adaptation gain

  • xLlama-Selector improved support F1 by 40.54 points, from 43.69 to 84.23, while support EM rose from 20.27 to 57.42.The larger selector change than the reader change is compatible with the selector searching an average of more than eight candidate languages, but the different metrics do not establish that selection is causally harder. At 57.42 support EM, 42.58% of items still lacked an exact evidence set.Source: §6.3 · p. 13

6.6. RQ4: realized question–paragraph language effects

  • S4 versus S2 answer F1 was lower by 10.25–15.79 points across readers, while different-script evidence gaps were 11.98–23.70 points; selector contrasts were only −1.71 and −1.78.None of the reader intervals for S4 versus S2 included zero, whereas the selector interval included zero. These are associations in frozen outputs under one realized language assignment.Source: Table 9 · p. 14, §6.6 · p. 14
  • For 5,010 bridge items, matching the answer-hop language to the question language was associated with +14.43 to +21.51 reader-F1 points, but language-specific associations were model-dependent.Only Llama retained FDR-significant adjusted gold-language associations; no gold-language coefficient survived correction for GPT-4o mini, Qwen, or the selector. Distractor-language and candidate-count selector contrasts had intervals including zero. A paired analysis changing only question–answer language was not reported because parallel validation views were missing.Source: §6.6 · pp. 14–15

6.7. RQ5: V2 RC1 improves paragraph and question judgments, not answers

  • V2 RC1 scored 94.68 versus 92.10 for V1 overall, with higher paragraph and question scores, but answer scores did not show the same pattern.Answer-score difference was −1.29 points; lower V2 scores were concentrated in script-sensitive cases. The candidate covered 99.00% of intended sources, while both validation omissions involved malformed HotpotQA records.Source: Table 10 · p. 15, §6.7 · pp. 15–16

7. Discussion

XHotpotQA’s discussion argues that language boundaries should be analyzed at the evidence-unit level, because readers’ cross-lingual composition deficits are larger than the adapted selector’s under the supplied-candidate design. It also emphasizes that the benchmark diagnoses evidence composition rather than open-corpus retrieval, and that answer-only or corpus-level translation scores miss important evidence and language-specific differences.

7.1. Implications for knowledge-based reasoning systems

  • Attaching language to evidence units rather than labeling each example as a whole distinguishes systems’ cross-hop and question-language distractor failure modes.The supplied candidate set controls selector input but excludes multilingual indexing, corpus coverage, and first-stage retrieval recall; complete multilingual RAG evaluation should pair evidence composition with open-corpus resources such as XOR QA or XRAG.Source: §7.1 · p. 16

7.2. Evidence fidelity matters more than answer-only gains

  • 51.62 answer F1 but 40.49 joint F1 for the best pipeline shows that answer-only gains leave an evidence gap.Sentence support exposes selector failures, enables error attribution, and prevents unsupported answers counting as complete successes; labels may not cover alternative correct chains, so future evaluation could adjudicate alternatives or test entailment while retaining original labels. Oracle-evidence reader improvements do not fix upstream candidate-selection errors, and the historically reported selector adaptation gains and within-system language-arrangement gaps answer different questions.Source: §7.2 · p. 16

7.3. Cross-lingual composition is the principal observed sensitivity

  • 10.25–15.79-point reader-F1 deficits accompany complete question–gold mismatch, versus 1.71 points for the adapted selector; different-script evidence corresponds to 11.98–23.70-point reader deficits versus 1.78 points.These are descriptive contrasts within the supplied-candidate design, not causal comparisons between modules; S0 has no observed items and S1 only 14. The pattern recurs across two language geometries and after excluding structurally flagged validation items, motivating bridge-entity normalization and hop-aware composition.Source: §7.3 · p. 16

7.4. Scale, multilinguality, and reasoning are not interchangeable

  • The one-shot screen finds a broad association between model capability and reader F1 but no monotonic rule across model families.Evaluation should separately test single-hop access in each language, cross-language bridge transfer, and composition across both hops; aggregate model size is not a substitute.Source: §7.4 · p. 17

7.5. Translation quality is role- and language-dependent

  • The V2 audit finds that answer behavior depends on names, titles, acronyms, and script conventions, so per-language scores are a triage map rather than a language ranking or conclusive version preference.Thai, Chinese, and Greek answer regressions co-occur with more exact English copies, but conventional Latin-script names can be appropriate and the judge applies that exception unevenly; bilingual review is needed.Source: §7.5 · p. 17

7.6. Cross-script and typological analysis

  • 11.98–23.70 fewer reader-F1 points for different-script versus same-script evidence, and a 1.78-point adapted-selector gap, make script a useful diagnostic distinct from language family.Persian has the lowest reader F1 in all three verified systems while its selector F1 remains near the macro average; these associations do not isolate orthography from exposure, translation quality, morphology, or transliteration.Source: §7.6 · p. 17

7.7. Versioned regeneration

  • Versioned regeneration preserves earlier translations by assigning each translator a new version over stable source IDs and recorded language assignments.V2 is one Gemma 4 31B instantiation, not a retroactive correction of V1; version comparisons require stratified bilingual audits and completeness checks, since model recency alone does not establish better translation.Source: §7.7 · p. 17

8. Limitations and threats to validity

The benchmark’s design and available artifacts constrain what its comparisons establish: translation-derived content, imperfect assignments and resources, and evaluation limitations narrow claims about cross-lingual reasoning, retrieval, and generalization.

Limitations and threats to validity

  • Translation-derived examples inherit English Wikipedia’s entity and cultural distribution, while random language assignment does not guarantee balanced ordered question–bridge–answer triples.Fluent translations may not reflect native information needs, and quality may vary; role-balanced paired subsets are needed for fine-grained comparisons.Source: §8 · p. 17
  • Historical V1 lacks its assignment seed, V2 cannot support paired translator comparisons with V1, and neither artifact is certified semantically correct in every language.V1.1 freezes a deterministic one-view projection as a packaging choice, not a reconstruction; V2 has inherited validation defects, clean-source training omissions, and incomplete parallel validation in XHotpotQA+.Source: §8 · pp. 17–18
  • Validation has no realized monolingual control, so condition contrasts measure degrees and locations of cross-lingual mixing, not the effect of introducing cross-linguality.The intended assignment yields no S0 observations and only 14 S1 validation items.Source: §8 · p. 18
  • The translation audit relies on one requested model alias, one rubric, and one score per unit, without bilingual human calibration or repeated judge models.The stored configuration does not authenticate the resolved checkpoint; independent samples, omitted judge and endpoint uncertainty, and inconsistent script conventions further limit validity.Source: §8 · p. 18
  • Support annotations do not establish two-hop necessity, and fixed-candidate selector results do not measure corpus-scale retrieval, latency, or robustness to missing gold documents.Parametric knowledge, shortcuts, or one sufficient paragraph can yield correct answers; Eq. 9 and partial-evidence evaluation should accompany reasoning claims.Source: §8 · p. 18
  • Exact match and token F1 can penalize valid cross-script aliases, morphology, segmentation, and transliteration, while character F1 and semantic review have their own biases.Per-language human analysis remains necessary.Source: §8 · p. 18
  • The 7,405-instance validation set is a development set without a hidden multilingual test, and single-run training results are point estimates.Reported maxima may reflect development-set selection; language-pair analyses need paired uncertainty, several seeds, and multiple-comparison correction such as Holm’s procedure.Source: §8 · p. 18
  • Complete item-level prediction files cover only three one-shot readers and one selector, while bridge and answer-hop roles are not well defined for every comparison question.Absent responses are scored as empty, transfer and end-to-end results remain historical aggregates, and role-aware analyses should preserve an unordered-comparison category.Source: §8 · p. 18

9. Conclusion

XHotpotQA makes cross-lingual multi-hop QA auditable through explicit language assignments, role-aware evidence, and support keys. Its reader artifacts show sizable condition-associated F1 gaps, while selector and language effects do not support a universal hierarchy.

9. Conclusion

  • XHotpotQA assigns languages independently to question–answer pairs and candidate paragraphs, retaining HotpotQA reasoning roles and support keys while measuring interface crossing and multilingual evidence composition.Its descriptors include mismatch, entropy, richness, and script, and the audit emphasizes measuring transformation fidelity rather than assuming it.Source: §9 · p. 18
  • Full question–evidence mismatch is associated with 10.25–15.79 lower reader F1 than partial alignment, and different-script evidence with 11.98–23.70 lower F1 than same-script evidence.The corresponding adapted-selector alignment gaps are small; direct distractor-language contrasts have intervals spanning zero.Source: §9 · p. 18
  • Only one reader retains significant gold-language coefficients after joint covariate adjustment and within-outcome FDR correction, supporting no universal gold-language hierarchy across readers or the selector.Marginal scores vary with assigned gold-paragraph language, but the other two readers and selector do not support a universal hierarchy.Source: §9 · p. 18
  • Historical all-split experiments report translate-train reader/support F1 of 53.41/84.23 and strongest end-to-end answer/joint F1 of 51.62/40.49.These remain historical aggregates pending deposit of their full predictions.Source: §9 · p. 18
  • The Gemma-generated V2 candidate receives higher paragraph and question judgments but no supported answer improvement, and incomplete clean-source coverage prevents it from replacing V1.The benchmark complements same-language translations, small probes, and open-retrieval multilingual RAG datasets; complete V2 and XHotpotQA+ releases require full coverage and bilingual review.Source: §9 · pp. 18–19

CRediT authorship contribution statement … B. Artifact-verified metric and language sensitivity

The appendix records authorship, funding, resource availability, ethics, and AI-use disclosures, then notes limits on interpreting language-distribution counts and the provenance of a historical reader-artifact label.

CRediT authorship contribution statement

  • Iman Barati is credited with conceptualization, methodology, data curation, software, investigation, formal analysis, and the original draft; all authors approved the manuscript and are accountable.Arash Ghafouri is credited with supervision, and Behrouz Minaei-Bidgoli with supervision and validation.Source: CRediT authorship contribution statement · p. 19

Funding

  • The study reports no specific grant from public, commercial, or not-for-profit funding agencies.Source: Funding · p. 19

Data availability

  • The XHotpotQA code, audited V1.1 dataset, and Gemma-generated V2 candidate are publicly available, with judge samples cross-linked as one release family.Dataset files inherit CC BY-SA 4.0 from HotpotQA; repository software uses MIT.Source: Data availability · p. 19

Ethics statement

  • The study used a public benchmark without intervention, private records, or sensitive-data collection; ethics approval and informed consent for personal-data publication were deemed inapplicable.Three specialists assessed machine-translated text for translator selection; no participant characteristics were collected.Source: Ethics statement · p. 19

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

  • The authors used OpenAI Codex for drafting support, language editing, and LaTeX restructuring, then reviewed and edited its contributions and retained responsibility for the publication.Experimental models are reported separately as research methods.Source: Declaration of generative AI and… · p. 19

A. Full language distribution

  • Table 11 gives marginal language-assignment counts for the historical base-train projection and audited legacy validation, but question counts alone do not characterize question–evidence pairs.“Support” counts supporting-fact annotation occurrences, not unique paragraphs; “Docs” counts candidate-paragraph occurrences.Source: A. Full language distribution · p. 23, Table 11 · p. 23

B. Artifact-verified metric and language sensitivity

  • The labels “Llama 70B” and “Llama 3.1 70B” are historical reader-artifact labels; the surviving client cannot independently establish the private endpoint’s checkpoint, whose identity requires the server log.Source: B. Artifact-verified metric and… · p. 23

B.1. Normalization protocol … C.2. V1 structural audit and release decision

The section documents reader-score sensitivity to answer normalization, language-conditioned variation and exploratory role/distractor analyses, then audits release validity and structural defects. Excluding flagged validation items barely changes aggregate scores or primary language-condition contrasts, but semantic adequacy remains unverified and a corrected V2 requires documented repairs.

B.1. Normalization protocol

  • Unicode normalization and character tokenization raise aggregate F1 by 2.78–3.21 points while leaving EM nearly unchanged; cross-paper language rankings require naming tokenization.The largest per-language F1 changes are Chinese (22.26–26.05 points), Japanese (24.11–28.73), and Thai (13.70–17.20).Source: §B.1 · p. 23, Table 12 · p. 24

B.2. Question-language results

  • Reader F1 ranged 29.99–65.56 (Llama), 34.18–57.81 (GPT-4o mini), and 13.19–62.32 (Qwen), while selector support F1 ranged 81.21–87.53.Persian was the minimum reader language for all three systems (29.99/34.18/13.19), while its selector F1 was 84.25; these descriptive patterns do not isolate translation adequacy, transliteration, training exposure, script, or answer morphology.Source: §B.2 · pp. 23–24, Table 13 · p. 24

B.3. Gold-paragraph-language results

  • Gold-paragraph language pairs comprise 326 (4.40%) same-language pairs, 2,006 (27.09%) same-script pairs, and 5,073 (68.51%) cross-script pairs.The reported language strata overlap, counts are not additive, and their differences are not pairwise interventions; bootstrap intervals resample source items, not paragraph rows.Source: Table 14 · p. 25, §B.3 · p. 25

B.4. Role and distractor contrasts

  • The exploratory role comparison covers bridge questions whose annotated support under one gold title uniquely contains the normalized answer, and its positive and negative groups are neither balanced nor paired.Table 15 reports role- and distractor-language contrasts in percentage-point F1 using 2,000 stratified item-bootstrap replicates; no contrast values are supplied in the passages.Source: §B.4 · p. 25, Table 15 · p. 26

C.1. Validity dimensions and automatic gates

  • The five validity dimensions require preserved meaning, question–answer relation and polarity, entities/numbers and comparison direction, paragraph/sentence cardinality and order, and assigned language with acceptable script or variant.Source: §C.1 · p. 26
  • Release gates check non-empty output, Unicode validity, sentence cardinality, identifier stability, support bounds, content integrity, and question/answer language equality.Other versioned audit checks include language identification, duplicate-content review, named-entity/number consistency, and answer occurrence; back-translation can triage but cannot establish semantic equivalence, and not all gates were enforced historically.Source: §C.1 · p. 26

C.2. V1 structural audit and release decision

  • Structural audit found 424 of 73,700 paragraph occurrences with sentence-cardinality changes, affecting 407 instances; 33 instances had blank translated sentences and nine gold support indices exceeded shortened arrays.The historical reader notebook silently omitted those nine support facts; 439 instances (5.93%) had canonical-release-blocking flags and remain quarantined in audited V1.1, while four duplicate-title items remain review-required. The audit does not establish semantic adequacy for unflagged items.Source: §C.2 · p. 26, Table 16 · p. 27

C. Resource-quality and release audit

  • The largest absolute shift after excluding 439 flagged items was 0.51 points across tested EM/F1 metrics; primary language-condition contrasts shifted by at most 0.22 points, with all paired intervals including zero.The audit therefore finds the reported aggregate and condition-level results numerically insensitive to exclusion, but a corrected V2 still requires repaired text and a versioned correction record.Source: §C.2 · p. 26, Table 17 · p. 27

C.3. V2 RC1 completeness and source preservation … D.1. Audit-set construction

V2 is an auditable but incomplete snapshot, and the paper specifies paired acceptance criteria and human-audit safeguards for any future canonical release. The audit is intended for controlled benchmark research, not claims about natural language use or community-level abilities.

C.3. V2 RC1 completeness and source preservation

  • V2 contains 22,836 of 23,066 intended instances (99.00%), so its 230 missing IDs prevent a complete row-level audit or canonical-release claim.The retry ledger recorded only three unresolved records and is not authoritative; source-ID reconciliation establishes the missing count. V2 is suitable as an audited incomplete snapshot, but completion requires restoring missing IDs, handling inherited defects, adjudicating support-label anomalies, and repeating alignment and bilingual quality review.Source: §C.3 · pp. 27–28
  • Of 230 missing rows, 50 have source defects and 180 are clean-source training omissions; 92 retained blank sentences use em-dash placeholders.The 50 source-defect rows comprise 48 training rows and both validation omissions; the other 180 are training omissions introduced by the V2 workflow. The 92 placeholders preserve recoverability in stored source arrays. A separate screen found 1,612 training and 837 validation non-English outputs identical to English, which are review signals rather than automatic defects. The official source also has 137 blank-string sentence elements and five out-of-bounds support indices; Table 18 classifies anomaly attribution.Source: §C.3 · p. 27, Table 18 · p. 28

C.4. Future paired acceptance target

  • Future canonical V2 acceptance requires complete key equality, zero release-blocking defects, positive bilingual adequacy evidence, and no material preservation regression.The supplied RC1 does not use the frozen V1 target-language assignment, so the paired equations are acceptance targets, not analyses of current judge samples. A negative structural-defect change alone is insufficient because structurally valid mistranslations can still alter meaning; the current independent GLM audit does not meet paired-identification requirements.Source: §C.4 · p. 28

C.5. Human audit design

  • The release-grade bilingual audit scores complete instances, uses two annotators plus third-reviewer adjudication, and reports source-clustered uncertainty.Sampling is stratified across 24 languages and other dimensions; each unit receives 1–5 adequacy and fluency scores and binary preservation labels. Complete-instance review captures inconsistencies such as a bridge entity translated differently across individually fluent sentences. Only cosmetic errors may remain uncorrected in canonical V2; audited V1.1 retains all error categories with flags and adjudication traces.Source: §C.5 · p. 29

C.6. Data statement and intended use

  • The benchmark measures behavior under controlled machine translation, not natural speaker questions, community knowledge preferences, or real-world code-switching.Its language inventory is not balanced across speaker population, region, gender, dialect, or socioeconomic context. Intended uses include benchmark research, controlled ablation, multilingual selection, transfer, and evaluation tools; out-of-scope uses include ranking communities, estimating human reasoning, or high-stakes QA without independent validation.Source: §C.6 · p. 29

D.1. Audit-set construction

  • The final V2 audit population concatenates 1,840 paragraph, 460 question, and 460 corrected-answer units, with 80/20/20 units per target language in both versions.The initial V2 answer rows were sourced from English questions and did not judge answer translation, so all 240 were discarded and replaced with the separate corrected run; no statistic uses the invalid rows.Source: §D.1 · p. 29, Table 19 · p. 29
  • The evaluator reports the “requested GLM-5.2 alias,” not an immutable checkpoint, and serializes one source-referenced score per unit.Public documentation identifies an intended model family but cannot establish which provider checkpoint served the private alias; historical module comments about two scores do not describe the serialized records.Source: §D.1 · p. 29

D.2. Aggregate distributions … E.1. Reporting matrix

The audit finds that source-copy answers are more frequent and score lower in V2 RC1, with notable per-language differences that warrant targeted review rather than causal generalization. It also recommends reporting results across language, evidence, answer, context, and audit strata instead of relying on aggregate scores alone.

D.2. Aggregate distributions

  • V2 paragraph and question score distributions shift upward, while answer medians and interquartile ranges stay at 100 and the low-scoring tail increases answer SD.The ceiling-heavy answer distribution means the −1.29 mean difference should not be interpreted as uniform answer degradation.Source: §D.2 · p. 30

D.3. Per-language results and targeted interpretation

  • Each Table 21 language row uses the same 80:20:20 allocation, making overall means comparable within this audit but not population-weighted estimates of real-world language use.Figure 8 emphasizes that the pooled gain is not homogeneous; its cells are descriptive, not paired translator effects or language rankings.Source: §D.3 · p. 30, Table 21 · p. 30, Figure 8 · p. 33
  • −4.96 is the only large negative overall language difference, occurring in Greek, where all three unit means decrease.Low-scoring RC1 Greek paragraphs include mixed English intrusions and corrupted lexical material. Greek RC1 has ten exact-copy answers averaging 76.5, versus two in V1, both scored 100; the small, differently sampled sets preclude causal comparison.Source: Table 21 · p. 30, §D.3 · p. 34
  • Thai answer means decrease by 30.70 points, while Chinese answer means decrease by 13.75, despite paragraph and question increases in both languages.Twelve of 20 RC1 Thai answers exactly copy English and average 48.0, versus none in V1; three RC1 Chinese answers copy English and average 43.33, versus one V1 copy scored 100.Source: Table 21 · p. 30, §D.3 · p. 34
  • 179/460 V2 RC1 answers exactly copy the source versus 116/460 in V1, while non-copy answer means rise from 94.27 to 97.21.Exact-copy means fall from 99.05 to 89.44. The passage describes this split as diagnostic, not a validity rule.Source: §D.3 · p. 34
  • Exact-copy detection should be combined with script detection, entity typing, and bilingual adjudication, not used to automatically reject every invariant answer string.The rubric accepts conventional Latin-script names and minor transliteration differences, while unchanged multiword common-noun answers can genuinely violate target-language requirements.Source: §D.3 · p. 34

E.1. Reporting matrix

  • The reporting matrix calls for answer, support, and joint EM/F1 disaggregated by question language, ordered question–gold language pairs, mismatch conditions, and bridge or answer-hop roles.It specifies macro-language and balanced pair macro-averages, populated S0–S4 conditions, ρG, ρD (marking no-distractor items NA), HG, HC, and KC.Source: §E.1 · p. 34
  • The matrix also stratifies results by evidence script and language family, answer type, context and support complexity, candidate-language count, and audit or translation-error status.It allows the language index to denote the answer-hop language or an unordered evidence-language set, preventing high-count pairs from masking rare directions.Source: §E.1 · p. 34

E.2. Regression and uncertainty … F.2. Translation prompts

These sections specify methods for modeling validation accuracy and testing dependence on both evidence hops, then document the scope and recovered translation-prompt contracts. They distinguish what can be reproduced from prompt text alone from dependencies that were not retained.

E.2. Regression and uncertainty

  • For base-split binary EM, a prespecified logistic model tests conditional associations with gold-evidence and distractor language conditions.The model also accounts for question and answer types; the 21 no-distractor items are omitted from distractor terms or modeled with a missingness indicator, while audit quality and length serve as sensitivity covariates.Source: §E.2 · p. 35
  • For bounded F1, use item bootstrap estimates, clustering by source for repeated views; resample source IDs for pairwise system comparisons.Confirmatory language-condition contrasts should use Holm correction, and trained or stochastic systems should use at least three seeds; source questions, not seeds, are the independent sampling unit.Source: §E.2 · p. 35

E.3. Partial-evidence necessity tests

  • For two-support instances, compare both gold paragraphs, each hop alone, and neither using a complete-chain gain over the best partial-evidence condition.A positive gain suggests benefit from both hops but does not prove faithful internal reasoning; report it by question, bridge, and answer-hop languages to test whether language crossing changes evidence use.Source: §E.3 · p. 35

F.1. Scope and provenance

  • Recovered prompt text does not by itself ensure bitwise reproducibility, and credential-bearing legacy scripts are excluded from every release.Table 22 distinguishes text recovered literally from dependencies that were not frozen.Source: §F.1 · p. 35

F.2. Translation prompts

  • V1 used four role-specific translation messages; paragraph requests numbered source sentences, while title, question, and answer requests supplied only the English source string.The mutable gpt-4o-mini alias's resolved revision and random assignment state were not retained, so the recovered contract is exact at the message level but not the endpoint level.Source: §F.2 · p. 36
  • V2 RC1 used Gemma 4 31B with a stricter JSON translation contract that preserves entities, numbers, dates, polarity, and sentence boundaries.The parser requires one non-empty value for titles, questions, or answers and a same-cardinality translations array for paragraphs; malformed, empty, or exhausted requests are rejected or recorded as failures rather than copied from English.Source: §F.2 · p. 36

F.3. Source-referenced GLM judge prompts · F.4. Reader and selector prompts · F.5. Rationale-teacher prompt

The appendix documents recovered prompts and evaluation protocols for translation judging, cross-lingual reading and evidence selection, and rationale generation. It distinguishes historical artifacts from fully reproducible reconstructions where records were incomplete.

F.3. Source-referenced GLM judge prompts

  • The paragraph-and-question translation judge scores source-to-candidate fidelity from 0–100, applying weighted criteria and maximum-score rules for critical errors.Its rubric weights adequacy/faithfulness at 60%, terminology and entities at 15%, fluency at 20%, and style/register at 5%; critical-error ceilings include 10 for wrong-language output and 40 for contradiction or hallucination.Source: §F.3 · p. 37
  • The separate short-answer judge prioritizes preserving the factual response, including yes/no polarity, while allowing conventional Latin-script names and minor transliteration differences.The rubric weights faithfulness at 70%, terminology/entities/numbers at 20%, and fluency at 10%; specified error ceilings include 20 for opposite polarity or a completely different fact.Source: §F.3 · p. 37
  • Each judge request uses the English source, target language, and candidate; answer requests also include the English question, and the audit yields one source-referenced score.The requested alias was glm-5.2 at temperature zero, with the integer parsed from the final SCORE line; the historical “error floors” are maximum-score ceilings.Source: §F.3 · pp. 37–38

F.4. Reader and selector prompts

  • The recovered reader prefix directs the model to answer from potentially multilingual context, use cross-language connections when needed, and respond only in the question’s language.The exact checkpoint behind the historical Llama 3.1 70B endpoint label could not be recovered.Source: §F.4 · p. 38
  • Complete one-shot reader variants embed a preserved demonstration in the system message; Llama/GPT use row 2, while Qwen2 72B uses row 0.The evaluated question and context arrive as separate user messages, and retained punctuation or whitespace differences count as distinct historical variants.Source: §F.4 · p. 38
  • The historical selector prompt asks for relevant paragraphs and supporting-fact sentence indices across languages, and the selector predicts their identifiers from the question and candidate set.The released evaluator uses stable identifiers rather than translated-title string matching.Source: §F.4 · p. 38

F.5. Rationale-teacher prompt

  • The rationale teacher is prompted to generate a clear, step-by-step explanation connecting the question, context, and answer; the recovered protocol is not a bitwise reconstruction.Its serialization uses Question:, Context:, and answer: ***<answer>***, with teacher explanations appended as rationale targets; the provider-resolved revision and complete tokenizer/trainer state were not retained.Source: §F.5 · p. 39

Related papers

Loading 2608.27481v1…