Source-linked AI summary
Output Language Confusion under Multilingual Prompt Contamination
Riju Marwah, Ritvik Garimella, Khusham Bansal, Atishay Jain, Amit Sheth
TL;DR
Factual QA benchmarks assume monolingual prompts and exact-match scoring, assumptions that do not capture mixed-language prompt contexts or the distinction between semantic error and a correct answer in the wrong script. The paper introduces MDI to test this interaction across existing QA benchmarks; it finds that exact-match hallucination scoring can overstate semantic error for Hindi script-switching responses. The authors argue that mixed-language evaluations should report script-switching and adjusted semantic error alongside raw hallucination proxies.
Problem
Factual QA benchmarks assume monolingual prompts and exact-match scoring, although mixed-language deployment contexts can make a semantically correct answer in the wrong script score like a hallucination.
Source: §1 · p. 1Method
Results
For Hindi distraction, 120 of 148 clean-correct script-switched Llama-3.1-8B responses were semantically correct, reducing the raw hallucination proxy from 0.710 to 0.470.
Source: §7 · p. 9Takeaways & Limitations
Mixed-language evaluations should report exact-match hallucination proxies alongside script-switching and adjusted semantic error rates.
Source: §6 · p. 9The hallucination proxy conflates script-switched but semantically correct responses with genuine semantic errors, so it must be decomposed in multilingual settings.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multilingual deployment, from retrieval-augmented generation pipelines returning mixed-language passages to users pasting multilingual web content. We introduce Multilingual Distractor Interference (MDI), a lightweight and fully replicable evaluation protocol requiring no new data or annotation, in which factual questions are preceded by a semantically irrelevant foreign-language sentence, and evaluate five instruction-tuned LLMs across TruthfulQA and TriviaQA under eight distractor conditions (40,000 evaluations). Our central finding is a metric confound: for Llama-3.1-8B under a Hindi distractor, 58% of responses switch to Devanagari script, yielding a raw hallucination proxy of 0.710, but manual review reveals that 120 of 148 script-switched responses that were correct under clean conditions remain semantically correct despite being written in the wrong script, reducing the adjusted semantic hallucination rate to 0.470. All other models respond through abstention escalation with no hallucination increase. A paragraph-length English distractor triggers near-universal abstention (0.806-0.998) across all models, consistent with reading-comprehension confusion, a failure mode with direct consequences for multilingual RAG pipelines. TruthfulQA multiple-choice accuracy is unaffected under all single-sentence conditions. These results show that exact-match hallucination rates in mixed-language settings should be decomposed into script-switching and semantic error components before drawing conclusions about model reliability.
1 Introduction
Factual QA benchmarks assume monolingual prompts and exact-match scoring, but multilingual deployment challenges both assumptions. The paper introduces MDI and identifies script-switching, abstention escalation, and stability as distinct responses to multilingual distractors.
1 Introduction
- Benchmarks assuming clean monolingual prompts can mischaracterize failures when exact-match scoring treats a correct answer in the wrong script as a hallucination.Mixed-language retrieved passages and pasted multilingual content are examples of deployment contexts that challenge monolingual-prompt assumptions.Source: §1 · p. 1
- The MDI pipeline evaluates five instruction-tuned models under eight distractor conditions and measures accuracy, abstention, hallucination-proxy, and script-switching outcomes.Conditions include clean prompts, foreign-language sentences and their repeated variants, and a paragraph-length English excerpt.Source: Figure 1 · p. 2
- MDI is introduced as a lightweight, reproducible protocol for probing factual robustness under mixed-language prompt contamination.It uses existing factual QA benchmarks and requires no new data or annotation, as described in the paper context.Source: §1 · p. 2
- 120 of 148 clean-correct Hindi script-switched responses from Llama-3.1-8B remained semantically correct, lowering the raw hallucination proxy from 0.710 to 0.470.The metric confound shows that exact-match scoring can count wrong-script correct answers as hallucinations.Source: §1 · p. 2
- The paper distinguishes script-switching hallucination, abstention escalation, and stability as three response modes under multilingual distractors.The stated model-specific distinction is tied to spontaneous generative capacity in the distractor language.Source: §1 · p. 2
2 Related Work
Prior multilingual benchmarks generally evaluate tasks in a target language, while distracting-context studies focus on relevant or adversarial material, often within one language. This paper instead examines how semantically irrelevant foreign-language context affects otherwise English factual QA.
Multilingual evaluation
- Unlike multilingual QA benchmarks that align task and prompt language, this setup keeps the task English and introduces foreign language only as prompt contamination.This isolates instruction-following robustness from cross-lingual comprehension ability, a dimension the cited per-language benchmarks do not probe.Source: §2 · p. 3
Robustness to distracting context
- Prior distracting-context studies examined semantically relevant or adversarial distractors, whereas this work uses semantically irrelevant foreign-language distractors with an explicit ignore instruction.The contrast frames the study as probing instruction-following breakdown rather than comprehension difficulty.Source: §2 · p. 3
Multilingual contamination in RAG
- Systematic study of how mixed-language retrieved passages affect generator behavior, rather than retrieval ranking, has been limited.The paper describes its English paragraph condition as a controlled analog for contextual format confusion in RAG.Source: §2 · p. 3
Hallucination and factuality metrics
- In multilingual settings, exact-match scoring can conflate semantically correct script-switched responses with genuine hallucinations, inflating apparent error rates according to a model’s generative profile.The passage contrasts this directional bias with exact-match scoring’s known imperfections in monolingual settings.Source: §2 · p. 3
3 Multilingual Distractor Interference (MDI)
MDI probes whether semantically irrelevant foreign-language context disrupts English factual QA using existing benchmarks and standard inference. Its design varies distractor language, script, length, and format, while its metrics separate answer outcomes and flag a limitation of the hallucination proxy.
3 Multilingual Distractor Interference (MDI)
- MDI probes whether semantically irrelevant foreign-language context disrupts English factual question answering without requiring new data collection, annotation, or model access beyond standard inference.The distractor explicitly says it is irrelevant and instructs the model to ignore it, making the probe interpretable as language-grounded instruction-following failure rather than comprehension failure.Source: §3 · p. 3
3.1 Task Definition
- Eight conditions include clean prompts, Spanish, Hindi, and Chinese distractors repeated once or three times, plus a paragraph-length English excerpt.The paragraph condition isolates context-length and format-confusion effects from script-switching; Spanish shares English’s Latin script, while Hindi and Chinese introduce non-Latin scripts with differing pretraining resource levels.Source: §3.1 · pp. 3–4
3.2 Prompt Format
- TruthfulQA uses four-choice log-probability scoring without generation, while TriviaQA uses open-ended short answers and prompts for abstention when the answer is unknown.The multiple-choice format structurally prevents script-switching; TriviaQA generation uses greedy decoding with a 32-token maximum.Source: §3.2 · p. 4
3.3 Metrics
- TriviaQA reports exact-match accuracy, abstention, hallucination-proxy, and script-switching rates; TruthfulQA reports accuracy, and paired distractor comparisons use McNemar’s test.The hallucination proxy is the share of responses that are neither correct nor abstaining, but it can conflate script-switched correct answers with semantic errors and should be decomposed accordingly.Source: §3.3 · p. 4
4 Experimental Setup
The experiments compare five instruction-tuned models across complementary factual-QA formats and use paired questions across distractor conditions. The scale comprises 40,000 evaluations under identical comparisons across models.
4.1 Models
- Five publicly available instruction-tuned models span four training lineages and two parameter scales, with a controlled Llama-family comparison sharing a tokenizer.The Llama-3.1-8B and Llama-3.2-3B comparison is intended to isolate capacity and pretraining exposure from tokenizer fragmentation.Source: §4.1 · pp. 4–5
4.2 Datasets
- TruthfulQA provides multiple-choice answer selection without generation, while TriviaQA’s open short-answer format exposes generation, abstention, and hallucination behavior.The two formats separate failure modes arising from generation from those arising during answer selection.Source: §4.2 · p. 5
4.3 Evaluation Scale
- 40,000 evaluations cover five models, two datasets, eight conditions, and 500 questions per dataset; fixed question-distractor pairings enable paired comparisons.The pairing supports McNemar’s significance test throughout the evaluation.Source: §4.3 · p. 5
5 Results
Across factual QA formats, multilingual distractors produce model-specific effects: TruthfulQA multiple-choice accuracy is stable under single-sentence conditions, while TriviaQA exposes script-switching, abstention, and metric-confound patterns. Paragraph-length English context causes near-universal abstention, and repeating distractors modestly increases degradation without changing the qualitative pattern.
5.1 TruthfulQA: Multiple-Choice Is Robust
- TruthfulQA accuracy is unaffected by single-sentence distractors overall, although Llama-3.2-3B drops ∆= −0.100 under zh_long (p < 0.001).The authors describe the drop as a distinct sensitivity, small in absolute terms and absent from open-answer results.Source: §5.1 · pp. 5–6, Table 2 · p. 6
5.2 TriviaQA: A Clean Failure Mode Dissociation
- 58.2% of Llama-3.1-8B responses switch to Devanagari under Hindi, while other models switch on fewer than 1.2% and show no hallucination increase.Phi-3-mini, Mistral-7B, and Qwen-2.5-3B instead show falling hallucination rates as abstention increases.Source: Table 3 · p. 6, §5.2 · p. 6
5.3 The Metric Confound: Unpacking the Hallucination Proxy
- 120 of 148 clean-correct, script-switched responses remain semantically correct, reducing Llama-3.1-8B’s Hindi hallucination proxy from 0.710 to an adjusted semantic rate of 0.470.The adjusted rate remains above the clean baseline of 0.416; exact-match scoring conflates wrong-script correct answers with genuine errors.Source: §5.3 · pp. 6–7, Figure 2 · p. 7
5.4 Script-Switching as the Proximal Mechanism
- Llama-3.1-8B’s Hindi script-switching is not explained by language resource level or tokenizer fragmentation: Chinese produces 0% CJK outputs, and Llama-3.2-3B switches only 1.2%.The authors identify spontaneous script-switching propensity, associated with scale and Hindi pretraining exposure, as the relevant variable.Source: §5.4 · pp. 7–8
5.5 Paragraph-Length Distractor
- 0.806–0.998 abstention under the paragraph-length English distractor indicates context-format confusion, as models appear to treat the paragraph as reading-comprehension context.The paper connects this pattern to mixed-language or contextually mismatched retrieved passages in RAG deployments.Source: §5.5 · p. 8
5.6 Distractor Length Effects
- Repeating a single-sentence distractor three times modestly increases degradation but leaves the failure-mode pattern unchanged.For Llama-3.1-8B, hi_long hallucination proxy is 0.704 versus 0.710 for hi; Qwen-2.5-3B abstention rises from 0.730 to 0.842.Source: §5.6 · p. 8
6 Discussion
The discussion explains why open-answer models diverge under foreign-script distractors, distinguishes three response modes, and argues that evaluation should measure output-language control alongside factuality. Its conclusions are bounded by the evaluated models and distractor design.
6 Discussion
- Llama-3.1-8B’s Devanagari switching is consistent with strong spontaneous Hindi generation and insufficient suppression, but unavailable training-data composition prevents a definitive explanation.The 8B/3B Llama contrast suggests the failure mode scales with multilingual capacity.Source: §6 · p. 8
- Open-answer responses fall into script-switching hallucination, abstention escalation, or stability, with spontaneous generation in the distractor language determining the mode.TruthfulQA multiple-choice format removes the generation step and suppresses this failure mode regardless of multilingual profile.Source: §6 · p. 9
- Evaluation should report exact-match hallucination alongside script-switching and adjusted semantic error, and test output-language control under mixed-script prompts.The discussion also calls for mixed-language retrieved context in RAG benchmarks because paragraph-length English noise produces widespread abstention.Source: §6 · p. 9
- The study evaluates five open-source models; its pattern may not generalize to closed-source models or different pretraining mixes, and naturalistic distractors without ignore instructions remain future work.The evaluated models were released in 2023–2024, excluding newer generations.Source: §6 · p. 9
7 Conclusion
The conclusion finds that multilingual distractors can expose both a scoring confound and distinct response failures, depending on models’ spontaneous generation profiles. It concludes that mixed-script output-language control warrants explicit evaluation alongside cross-lingual transfer and per-language accuracy.
7 Conclusion
- 120 of 148 clean-condition-correct script-switched responses were semantically correct, reducing Llama-3.1-8B’s Hindi-distractor hallucination proxy from 0.710 to an adjusted semantic hallucination rate of 0.470.Exact-match error should be treated as an upper bound for models with strong non-English generation and decomposed by script switching.Source: §7 · p. 9
- Models without spontaneous Hindi generation show abstention escalation, while paragraph-length English distractors produce near-universal abstention consistent with reading-comprehension confusion.The conclusion identifies distinct practical consequences across these response modes.Source: §7 · p. 9
- Output-language control under mixed-script prompts is a distinct, underexamined axis of multilingual model quality that should be evaluated explicitly.The conclusion places it alongside cross-lingual transfer and per-language accuracy.Source: §7 · p. 9