PaperlayerView exact revision on arXiv

Source-linked AI summary

Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities

Dayeon Ki, Ruochen Zhang, Silviu Cucerzan, Ryen W. White, Ning Gao

arXiv:2610.00606v1cs.CLcs.AI

TL;DR

Query-language preference has mostly been studied where equivalent knowledge is available across languages, despite real-world gaps and conflicting accounts. WALDO evaluates this preference using multilingual Wikipedia QA pairs and finds little preference under gaps but strong preference under conflicts; QLORA reduces the gap by up to 61.5%.

  • Problem

    Prior work has largely examined query-language preference where equivalent knowledge is available across languages, leaving its behavior under incomplete or conflicting accounts less characterized.

    Source: §1 · p. 2

  • Method

    WALDO uses 12K multilingual Wikipedia QA pairs targeting knowledge gaps and controlled conflicts, and evaluates eight models across five languages.

    Source: §1 · pp. 2–3

  • Results

    Models largely overcome query-language preference under knowledge gaps but exhibit it strongly under conflicts, and QLORA reduces the preference gap by up to 61.5%.

    Source: §1 · p. 3, CONCLUSION & DISCUSSION · p. 9

  • Takeaways & Limitations

    When responses depend on query language, users may be confined to language-bounded evidence, reinforcing filter bubbles and information cocoons.

    Source: CONCLUSION & DISCUSSION · p. 9

    The mitigation methods need evaluation across languages and domains because reducing language preference may unevenly affect answer quality, source faithfulness, or linguistic and cultural coverage.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Large Language Models increasingly serve as interfaces for knowledge-intensive information seeking tasks across languages by synthesizing multilingual evidence. Prior work has shown that they often exhibit query-language preference -- the tendency to favor sources written in the language of the query -- but has largely examined this behavior in settings where equivalent knowledge is available across languages. However, this bias becomes consequential when sources in different languages provide incomplete or inconsistent accounts of the same fact, since the information users receive then depends on the sources a model selects to use. To characterize query-language preference under such cross-lingual knowledge disparities, we introduce Waldo, a multilingual Question-Answering (QA) benchmark constructed from Wikipedia. Waldo contains 12K QA pairs targeting knowledge gaps, where a fact is available in one language but absent in another, and knowledge conflicts, where language editions provide conflicting versions of the same fact. Evaluating eight models across five languages, we find that when one language edition merely lacks the relevant fact, models generally use evidence from the other language regardless of the query language. Under conflicting accounts, however, model responses strongly align with the document in the query language, causing semantically equivalent queries to elicit different accounts depending on the user's language. Finally, we explore two different approaches that could mitigate this preference under knowledge conflicts: a mechanistic intervention that ablates attention heads associated with query-language preference, and LoRA-based training, which reduces the preference gap by up to 61.5%.

1 INTRODUCTION

Query-language preference can shape which multilingual sources inform an answer, but its effects are less understood when language editions contain gaps or conflicting facts. WALDO evaluates this behavior and finds that conflicts, unlike gaps, produce strong query-language preference.

1 INTRODUCTION

  • Query-language preference matters because a user’s query language can affect which sources inform the response and what information the user receives.Prior work largely assumed equivalent information was available across languages, although multilingual sources can be incomplete or inconsistent.Source: §1 · p. 1
  • Cross-lingual disparities include knowledge gaps, where a fact is absent in one language, and conflicts, where language editions give incompatible accounts.The paper links language-dependent answers to potential asymmetries in answer quality and downstream performance, as well as polarization, marginalized narratives, and reduced trust.Source: §1 · p. 2
  • Under knowledge gaps, models largely overcome query-language preference, but under knowledge conflicts their response distributions differ substantially by query language.The study evaluates eight models across five languages; QLORA reduces the preference gap by up to 61.5%.Source: §1 · pp. 2–3
  • WALDO is a 12K-query multilingual Wikipedia benchmark covering facts that are unevenly available across languages and facts with incompatible accounts.It holds multilingual context fixed while varying query language, and uses facts postdating evaluated models’ reported knowledge cutoffs to reduce reliance on parametric knowledge.Source: §1 · pp. 2–3
  • The paper explores QPHA and QLORA as approaches for reducing query-language preference under conflicting evidence.The introduction describes QPHA as a mechanistic intervention and QLORA as training with two customized objectives.Source: §1 · p. 3

2 BACKGROUND: DEFINITION, EVALUATION, AND MITIGATION

The paper defines query-language preference as favoring sources in the query’s language and situates its study among work on retrieval and generation. WALDO addresses the lack of a reusable benchmark connecting naturally occurring cross-lingual disparities to preference at generation time.

DEFINITION

  • Query-language preference is a specific form of language preference in which a model favors sources written in the same language as the query.The paper focuses on generation while holding retrieved multilingual context fixed to isolate the effect of query language on source use.Source: §2 · p. 3

DATASETS AND EVALUATION

  • Prior evaluation settings either cannot isolate preference when language editions differ in fact availability or content, or use fictional conflict documents.WALDO instead targets naturally occurring heterogeneous facts from Wikipedia, including both gaps and controlled conflicts.Source: §2 · p. 3

MITIGATION

  • Prior mitigation work targets retrieval or unifies evidence through translation, while this paper studies generation-time preference and explores interventions and fine-tuning.The authors state that, to their knowledge, prior work had not specifically studied these generation-time mitigation approaches.Source: §2 · p. 3

3 CREATING WALDO

WALDO is built from Wikipedia revisions and linked documents in five source languages. Its pipeline selects cross-lingual knowledge gaps, creates controlled conflicts, and filters examples to reduce the chance that evaluated models answer from prior knowledge alone.

3 CREATING WALDO

  • WALDO draws on Wikipedia documents and revision histories, using five source languages and a balanced sample of five dump files per language for experiments.The languages are French, Simplified Chinese, Japanese, Korean, and Bengali; revisions from January 1, 2020 onward are retained.Source: §3 · p. 4

3.1 KNOWLEDGE GAP [

  • For knowledge gaps, revisions must introduce a factual statement that remains in the source-language document but is absent from its linked English document.The pipeline excludes bot or maintenance edits and uses aligned source-language and English QA pairs targeting the introduced fact; each aligned pair must score at least 80.0 on XCOMET.Source: §3.1 · p. 4

3.2 KNOWLEDGE CONFLICT [

  • For knowledge conflicts, the pipeline injects a revised core attribute into the English document and retains pairs only if a separate model can answer the original question from that document.The final set contains approximately 12K QA pairs; model-specific evaluation further retains examples whose timestamps postdate reported knowledge cutoffs and whose questions models answer incorrectly without context in both languages.Source: §3.2 · p. 4

4 EXPERIMENT SETUP

WALDO presents aligned questions with paired source-language and English documents, holding context fixed while varying query language. It measures whether models recover the original target fact through answer agreement and source-supported answer probability.

4.1 STUDY FORMULATION

  • Each WALDO instance pairs aligned questions with source-language and English documents; models receive the fixed multilingual context in randomized order while the query language varies.Target facts postdate each evaluated model’s knowledge cutoff, making responses less likely to be determined by parametric knowledge.Source: §4.1 · p. 5
  • Query-language preference means following the source-language account for a source-language query but not giving the equivalent English answer for an English query.In gap settings this can appear as abstention or an unsupported answer; in conflict settings, as selecting the conflict-injected English answer.Source: §4.1 · p. 5

4.2 EVALUATION METRICS

  • Answer agreement checks whether a response matches the reference in the query language, while answer probability measures the length-normalized likelihood of the source-supported answer.Agreement gaps and likelihood shifts compare these measures across source-language and English queries.Source: §4.2 · pp. 5–6
  • A positive source-answer likelihood shift means the model assigns greater likelihood to the source-supported answer for a source-language query than for an English query.Likelihood scores are averaged across evaluated QA pairs; length normalization makes scores comparable across answers of different lengths.Source: §4.2 · p. 6

5 DIAGNOSIS RESULTS

Query-language preference is substantially stronger under conflicting accounts than under knowledge gaps. It persists when document roles are reversed, and the examined confounders do not account for the measured effect.

5 DIAGNOSIS RESULTS

  • Under knowledge gaps, closed-source models showed little or no significant answer-agreement difference, while open-weight models had significant gaps of 15.9% for LLAMA-3.1, 12.1% for QWEN-3, and 18.2% for AYA-EXPANSE.All three open-weight models also assigned significantly different likelihoods to the source-supported answer across query languages; results generalize when external retrieval is added.Source: §5 · p. 6
  • 55.7% average answer-agreement gap across models under knowledge conflicts, where all models significantly favored the query-language-matched document.The answer-probability gap also increased relative to the knowledge-gap setting; disaggregated gaps were largest for lower-resource languages such as Bengali and lowest for French.Source: §5 · p. 7
  • All models retained strong query-language preference after document roles were swapped, with the direction of the effect reversed.Table 2 reports answer-agreement and probability gaps under the swapped-role conflict condition.Source: §5 · p. 7, Table 2 · p. 7
  • Tests of pre-training effect, embedding similarity, surface cues, and query-topic locality found no significant associations with the query-language preference metrics.Together, the results suggest the measured effect is not accounted for by these examined confounders.Source: §5 · p. 7

6 MITIGATION FOR KNOWLEDGE CONFLICTS

Under knowledge conflicts, models favor the document written in the query language; the paper tests head ablation and QLORA training as mitigations. QLORA yields larger reported reductions, while QPHA reduces preference through targeted head ablation.

6.1 MITIGATION APPROACHES

  • QPHA identifies heads whose attribution shifts toward source-supported answers under source-language queries and conflicting English-supported answers under English queries, then ablates the highest-scoring significant heads.The study compares zero ablation, mean-activation replacement, and random-head controls.Source: §6.1 · pp. 7–8
  • QLORA uses promote and contrastive objectives: the latter forgets the conflicting English-supported answer while retaining the source-supported answer under source-language queries.The objectives use length-normalized answer log-likelihoods.Source: §6.1 · p. 8
  • Mitigation experiments cover three open-weight models and 5.1K preference-bearing conflict pairs, with 20% of revisions reserved for training, development, or head identification.The remaining 80% evaluates both methods.Source: §6.1 · p. 8

6.2 RESULTS & ANALYSES

  • QPHA’s zero and mean ablations each cut agreement gaps by 4.8 pp; probability-gap reductions were 5.3 pp for zero and 9.2 pp for mean.Both modes exceeded their respective random-ablation controls.Source: §6.2 · p. 9, Table 3 · p. 9
  • 59.0% agreement-gap and 20.7% probability-gap reductions came from QLORA’s contrastive objective, which outperformed its promote objective on both measures.The promote objective closed 58.0% of the agreement gap and 19.5% of the probability gap; QLORA used 10% of the data for training and also generalized to the swapped-roles setting.Source: §6.2 · p. 9
  • With more QPHA-selected heads ablated, both gaps shrink; ablating just two heads significantly reduces the gap versus vanilla (p < .05).Figure 3 shows the trend for LLAMA-3.1; the text reports the small-head result.Source: Figure 3 · p. 8, §6.2 · p. 9
  • QPHA is training-free but broader ablation may affect other behavior; QLORA reduces preference more, but needs training data and knowledge of desired and conflicting answers.The authors frame these as trade-offs for future work combining their advantages.Source: §6.2 · p. 9

CONCLUSION & DISCUSSION

The paper concludes that query-language preference is largely overcome under knowledge gaps but remains strong under conflicts, and reports mitigation through head ablation or LORA training. It identifies deployment impacts and broader-source evaluation as important next steps.

CONCLUSION & DISCUSSION

  • The authors call for evaluating real-world effects on comprehension, trust, and information access, and extending benchmarks beyond Wikipedia to sources such as academic papers.They also identify source-selection biases beyond query language, including formality, relevance, recency, and source origin, as future research directions.Source: CONCLUSION & DISCUSSION · p. 9
  • The conclusion reports that models largely overcome query-language preference under knowledge gaps but exhibit it strongly under knowledge conflicts.The paper states that both mechanistic intervention and LORA training can effectively mitigate preference under conflicts.Source: CONCLUSION & DISCUSSION · p. 9
  • The authors report using generative AI for synthetic-data generation, method implementation, hypothesis refinement, qualitative analyses, and writing assistance, while stating that authors reviewed or verified outputs.They state they did not use AI to develop theoretical models, formulate or prove mathematical claims, design experiments, or interpret results.Source: AI USE STATEMENT · p. 10
  • The authors caution that mitigation methods need evaluation across languages and domains because reducing language preference may unevenly affect answer quality, source faithfulness, or linguistic and cultural coverage.Source: ETHICS STATEMENT · p. 10

A.1 SOURCE LANGUAGES

  • Table 4 characterizes source languages by family, script, linguistic typology, and resource level measured by speaker and Wikipedia-article counts.Source: Table 4 · p. 22, §A.1 · p. 22

A.2 HEURISTICS & LLM ANNOTATION

The appendix describes how revisions are filtered and annotated, then reports QA translation quality, conflict examples, and the final answerability-checked evaluation set.

A.2 HEURISTICS & LLM ANNOTATION

  • The filtering pipeline combines heuristics and LLM annotation to retain factual additions absent from English Wikipedia but present in the latest source-language document.It retains approximately 4.8K revisions across five source languages.Source: §A.2 · pp. 22–23
  • LLM annotation was selected over INFOGAP because it better handles cases where English contains only part of the added information.The methods agreed on 87.9% of labels overall.Source: §A.2 · pp. 23–24
  • The retained revisions are a highly curated, non-representative subset, and their low retention reflects the rarity of revisions suitable for controlled evaluation.Source: §A.2 · p. 24

A.3 QA GENERATION

  • 0.960 for questions and 0.983 for answers were the average XCOMET-XL translation-quality scores against English counterparts.Source: §A.3 · p. 24, Table 7 · p. 24

A.4 QUALITATIVE EXAMPLES OF KNOWLEDGE CONFLICTS

  • Conflict examples contrast original target facts with plausible alternatives injected into English documents.One example changes “TV Tokyo” to “Nippon TV.”Source: §A.4 · p. 24, Table 8 · p. 25

A.5 ANSWERABILITY CHECK

  • The answerability check retained 11,888 knowledge-conflict QA pairs across five source languages for the final evaluation set.Source: §A.5 · p. 26, Table 9 · p. 26

B.1 RETRIEVAL-AUGMENTED ROBUSTNESS RESULTS

In a retrieval-augmented test, additional web documents reduced absolute answer agreement and probability, but the gap pattern persisted: small under knowledge gaps and larger under conflicts.

B.1 RETRIEVAL-AUGMENTED ROBUSTNESS RESULTS

  • The test sampled 100 QA pairs and supplied each model with six documents: two retrieved in each language plus the original source-language and English Wikipedia documents.Source: §B.1 · p. 26
  • With retrieved evidence, answer agreement and probability gaps remained small under knowledge gaps but substantially larger under knowledge conflicts.The pattern persisted despite the overall reduction in absolute agreement and probability.Source: §B.1 · p. 26
  • Manual inspection suggests that added documents introduce candidate sources and make the target fact harder to identify.Source: §B.1 · p. 26

B.2 PER-LANGUAGE RESULTS

Across five source languages, query-language preference was minimal under knowledge gaps but pronounced under conflicts; an additional probability measure found preference in both directions for open-weight models.

B.2 PER-LANGUAGE RESULTS

  • Across all five source languages, query-language preference was minimal under knowledge gaps but pronounced under conflicts.Answer-probability gaps were consistently larger for lower-resource languages, while answer-agreement gaps showed no clear relationship with language-resource level.Source: §B.2 · p. 26

B.3 ANSWER PROBABILITY FOR KNOWLEDGE CONFLICTS

  • The additional conflict measure evaluates the likelihood of the conflicting English-supported answer under each query language, complementing the source-supported-answer probability metric.Source: §B.3 · pp. 26–27
  • All three open-weight models assigned greater likelihood to the conflicting English answer for English queries than for source-language queries, indicating preference in both directions.Source: §B.3 · p. 27

B.4 ADDITIONAL EVALUATION METRICS

Additional metrics corroborate query-language preference under conflicts, while a knowledge-parity control finds generally near-zero preference gaps when both documents support the same fact.

B.4 ADDITIONAL EVALUATION METRICS

  • The additional metrics include token-level F1 overlap, document citation attribution, and whether the response language matches the query language.Source: §B.4 · p. 27
  • F1-score and citation-attribution gaps mirror answer-agreement results: differences are small under knowledge gaps but substantially larger under conflicts.Models more often cite the source-language document for source-language queries and the English document for English queries.Source: §B.4 · p. 28
  • Response language matched the query language in 93.4% of source-language and 92.9% of English gap cases, and 92.1% of source-language and 95.1% of English conflict cases.Under knowledge gaps, models usually responded in the query language even when retrieving the fact from the source-language document.Source: §B.4 · p. 28

B.5 KNOWLEDGE PARITY

  • When source-language and English documents support the same target fact, answer-agreement and answer-probability preference gaps are generally close to zero.Source: Table 12 · p. 28, §B.5 · p. 29

C HYPOTHESES

  • The next section formulates four hypotheses to investigate potential causes of query-language preference and test plausible confounders.Source: C HYPOTHESES · p. 29

C.1 PRE-TRAINING EFFECT

Across four analyses, the authors find little evidence that pre-training alignment, embedding similarity, surface cues, or topic locality reliably explain query-language preference. The pre-training analysis specifically provides limited evidence that query–document language alignment is a substantial driver.

C.1 PRE-TRAINING EFFECT

  • Most perplexity gaps were not statistically significant, providing limited evidence that query–document language alignment substantially drives preference.Gaps were generally positive for conflict-supported English answers, while source-supported answer results were mixed.Source: §C.1 · p. 30

C.2 EMBEDDING SIMILARITY

  • Embedding-similarity advantages do not reliably predict query-language preference from citation attribution across models and languages.Only a small number of model–language combinations showed statistically significant positive correlations.Source: §C.2 · p. 31

C.3 SURFACE CUES

  • Most correlations between relative document length and answer-agreement differences were not significant, while document-order correlations were near zero and nonsignificant.The length correlations were generally negative as the source-language document grew longer relative to the English document.Source: §C.3 · pp. 31–32

C.4 TOPIC LOCALITY

  • Answer-agreement differences did not significantly differ between culturally specific and agnostic queries, providing little evidence for topic-locality confounding.Source: §C.4 · p. 32

D.1 QLORA IMPLEMENTATION DETAILS

QLoRA-based training reduces agreement and probability gaps the most, while QPHA also reduces them; both approaches remain effective when the conflicting and original facts swap document roles. The appendices describe the ablation trends and selected heads, implementation settings, and WALDO construction prompts.

D.1 QLORA IMPLEMENTATION DETAILS

  • QLoRA uses 4-bit quantization, rank 16, alpha 32, dropout 0.05, and three training epochs on a single L40s GPU per model and objective.The implementation also uses bf16 compute and AdamW with a 2e-4 learning rate.Source: §D.1 · p. 33

D.2 PER-QUERY-LANGUAGE RESULTS

D.3 SWAPPED ROLES MITIGATION

  • Both QPHA and QLoRA reduce agreement and probability gaps when document roles are reversed, with QLoRA again yielding the largest reduction.Source: §D.3 · p. 33, Table 19 · p. 34

D.4 INCREASING THE NUMBER OF ABLATED HEADS

  • As more heads are ablated, agreement and probability gaps shrink; QPHA reduces them more than random ablation relative to vanilla.Source: §D.4 · p. 33, Figure 5 · p. 35

D.5 ABLATED ATTENTION HEADS

  • The highest-scoring QPHA heads cluster in layers 12–15 for LLAMA-3.1, 17–22 for QWEN-3, and 8–12 for AYA-EXPANSE.Figure 6 maps head scores, with darker cells indicating higher QPHA scores.Source: §D.5 · pp. 33, 35, Figure 6 · p. 38

E PROMPTS

  • The prompts document WALDO’s annotation and QA generation for knowledge gaps, conflict injection and answerability checks for conflicts, and prompts for querying and evaluating models.The gap-setting examples vary by source language.Source: E PROMPTS · p. 35
Loading 2610.00606v1…