PaperlayerView exact revision on arXiv

Source-linked AI summary

SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection

Sanghyeok Park, Minji Kang, Hosung Kwak, Jinhyuk Yun

arXiv:2609.09349v1cs.CL

TL;DR

Standard multilingual benchmarks largely measure selecting correct answers, leaving consistent rejection of false statements insufficiently assessed. SWORD addresses this gap by generating controlled Wikidata distortions in eight languages and finds that distortion-based evaluation exposes counterintuitive strategy effects and cross-lingual performance gaps.

  • Problem

    Existing multilingual benchmarks emphasize selecting true answers and provide limited evidence about whether models consistently reject false statements across languages.

  • Method

    SWORD perturbs Wikidata triples to generate syntactically well-formed false statements ranging from random substitutions to semantically plausible property-based distortions across eight languages.

  • Results

    Models generally achieve higher accuracy on semantically plausible than random distortions, while some show Western–East Asian rejection gaps up to 28 percentage points (49% relative reduction).

  • Takeaways & Limitations

    False-statement rejection and cross-lingual consistency provide diagnostic information that aggregate true-statement accuracy can miss.

  • Takeaways & Limitations

    The main results cover eight languages, and generalization to lower-resource languages remains an open question; limited official support for Korean or Japanese may partly affect some degradations.

Abstract

from arXiv · show

Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating genuine factual understanding. We introduce Systematic Wikidata-based Object-Relation Distortion (SWORD), a benchmark that evaluates whether models consistently reject factual errors across languages. SWORD generates syntactically well-formed but factually incorrect statements in eight widely spoken languages through controlled perturbations of Wikidata triples, ranging from random entity substitutions to semantically plausible property-based selections. Our distortion-based evaluation surfaces two critical insights that remain entirely obscured by conventional benchmarks. First, models counterintuitively achieve higher accuracy on semantically plausible distortions than on nonsensical random substitutions, suggesting reliance on distributional familiarity rather than genuine factual verification. Second, models exhibiting comparable baseline accuracy across languages show substantial performance degradation specifically on (East) Asian languages when presented with distorted statements, with cross-lingual performance gaps reaching up to 28 percentage points (49\% relative reduction) in some models. These findings demonstrate that multilingual factual reasoning involves asymmetric capabilities that aggregate accuracy metrics systematically obscure.

1 Introduction

Standard multilingual benchmarks measure selecting correct answers but provide limited evidence about rejecting false statements consistently across languages. SWORD addresses this gap with controlled Wikidata distortions and reveals cross-lingual disparities and counterintuitive distortion effects.

  • Existing multilingual benchmarks emphasize multiple-choice selection or answer extraction, leaving false-statement rejection insufficiently evaluated.
  • SWORD perturbs Wikidata subjects and objects to create syntactically well-formed false statements with controlled semantic proximity across eight languages.
  • Systematic false-statement evaluation reveals language-specific patterns that true-statement evaluation alone leaves invisible.
  • 28 percentage points (49% relative reduction) is the maximum reported gap between Western and East Asian languages for rejecting false statements despite comparable baseline accuracy.
  • Models achieve higher accuracy on semantically plausible property-based distortions than on nonsensical random substitutions, suggesting distributional familiarity rather than genuine factual verification.
  • The automated pipeline controls distortion difficulty through semantic proximity and supports scalable benchmark generation without expert validation.

2 Related Works

Prior multilingual benchmarks and factuality datasets evaluate consistency or factual knowledge using varied tasks and resources. SWORD targets an underaddressed gap by controlling semantic proximity before evaluation and testing cross-lingual rejection of false statements.

  • Multilingual evaluation has expanded from standardized frameworks to benchmarks such as XNLI, XTREME, MMLU variants, and Global MMLU.
  • Cross-lingual consistency research includes ranking consistency metrics, translate-then-evaluate strategies, and studies of factual prediction consistency across multilingual inputs.
  • Knowledge-graph benchmarks use expert-validated factual statements, automated Wikidata sampling, and time-sensitive factual knowledge evaluation.
  • Existing approaches assess difficulty post hoc and do not evaluate cross-lingual consistency in false-statement rejection.
  • SWORD addresses both gaps through property-based substitution with graph-structure-controlled semantic proximity across eight languages.

3 SWORD Benchmark Construction

SWORD constructs multilingual true and distorted false statements from Wikidata subject-property-object triples, controlling distortion plausibility through random or property-based substitutions. The pipeline translates these statements into eight languages and evaluates model True/False judgments.

  • SWORD evaluates whether models accept true statements and reject systematically distorted false ones.
  • Only triples with natural-language labels for all components in all eight target languages are retained for cross-lingual comparison.
  • The benchmark creates original relations and four distorted relations by substituting subjects or objects while keeping the property fixed.
  • 500 original triples are sampled, and accidental distorted triples that already exist in Wikidata are resampled to guarantee false labels.
  • Property-based strategies produce semantically closer substitutions than random strategies, creating more plausible false statements while random shuffling produces obviously nonsensical ones.
  • Each triple is converted into one sentence per language using Gemini-2.5-Flash, yielding 2,500 statements per language and 20,000 statements overall.
  • Models classify each statement as True or False; original statements are True labels, distortions are False labels, and Unsure responses count as incorrect.

4 Cross-Lingual Evaluation Reveals Hidden Performance Disparities

Distortion-based evaluation reveals cross-lingual disparities and model-specific response patterns that are not apparent on original statements. Property-based distortions are often easier to reject than random ones, while LLaMA 3.1 and Mistral Nemo show especially severe Asian-language degradation.

  • 4.1 Plausible Distortions Are Easier to Reject than Nonsensical Ones: 73.75% of 80 model-language comparisons show equal or higher accuracy on property-based than random shuffles.This reverses the expectation that semantically plausible false statements would be harder to reject.
  • 4.1 Plausible Distortions Are Easier to Reject than Nonsensical Ones: LLaMA 3.1 and Gemini favor property-based distortions in 14/16 and 16/16 cases, respectively, while Mistral Nemo shows no directional preference in 8/16 cases.Qwen3 performs better on property-based subject shuffles but has mixed object-shuffle results.
  • 4.2 Distortion Exposes Severe Cross-Lingual Disparities in Asian Languages: 26–28 percentage points separate German and Korean under LLaMA 3.1 property-based shuffles, corresponding to a 47–49% relative reduction.German scores 0.556 PSS and 0.580 POS, whereas Korean scores 0.294 and 0.298.
  • 4.2 Distortion Exposes Severe Cross-Lingual Disparities in Asian Languages: 21–23 percentage points separate European and Japanese languages for Mistral Nemo under property-based shuffles, while Gemini and Qwen3 do not show this disparity.The pattern is therefore model-specific rather than inherent to the task or language family.
  • 4.2 Distortion Exposes Severe Cross-Lingual Disparities in Asian Languages: Asian-language degradation intensifies with problem difficulty for LLaMA and Mistral, whereas Gemini remains consistent across difficulty levels.Qwen3’s apparent consistency is attributed to a False-response bias rather than genuine cross-lingual stability.
  • 4.3 Distortions Reveal Language Family Clustering in Model Responses: Under distortion, LLaMA 3.1 and Mistral Nemo develop Western/Asian language-family blocks, while Gemini maintains weak clustering across conditions.Qwen3’s clustering changes are interpreted in light of its response bias toward rejecting statements.
  • 4.3 Distortions Reveal Language Family Clustering in Model Responses: Distortions produce particularly low cross-model agreement for LLaMA 3.1 in non-English languages, most strongly in Korean and Japanese.This exposes model-specific reasoning patterns that remain invisible on original statements.

5 Discussion

SWORD shows that true-statement accuracy does not characterize multilingual factual reliability because false-statement rejection exposes hidden asymmetries. These patterns support diagnostic, distortion-based evaluation alongside existing benchmarks and remain relevant for systems with or without retrieval augmentation.

  • 5 Discussion: SWORD separates accepting true statements from rejecting false ones, revealing model-specific patterns hidden by aggregate accuracy.The benchmark uses simple factual manipulations to increase diagnostic power rather than merely problem difficulty.
  • 5 Discussion: Plausible-over-nonsensical distortion accuracy suggests reliance on distributional familiarity with entity co-occurrences rather than genuine factual verification.This is the paper’s interpretation of the counterintuitive distortion result.
  • 5 Discussion: SWORD complements RAG-focused benchmarks by isolating cross-lingual variation in parametric knowledge, which remains relevant for systems operating without retrieval.Parametric knowledge also supports factual consistency assessment of retrieved content.
  • 5 Discussion: False-statement rejection, difficulty-controlled distortions, and cross-lingual consistency measures can enrich evaluation before deployment.SWORD provides a scalable, fully automated way to surface these failure modes.

6 Limitations

The study identifies scope, data, prompting, model-support, and generalizability limits that qualify its multilingual distortion findings. Despite these constraints, the findings replicate broadly and motivate expanded evaluation.

  • Scope and language coverage: The main results cover eight languages, leaving generalization to lower-resource languages an open question.Preliminary Arabic results suggest similar patterns, but broader systematic analysis is still needed.
  • Language support: Limited official language support may partly explain degradation for some languages, although similar patterns in Mistral Nemo suggest model-specific training factors also matter.LLaMA 3.1 lacks official Korean and Japanese support, while Mistral Nemo claims support for both.
  • Generation and comparison caveats: Gemini-generated statements may contribute to Gemini’s strong performance through self-preference effects, so cross-model comparisons should emphasize relative language patterns over absolute accuracy.Generation quality was directly assessed only in languages accessible to the authors.
  • Generalizability: Specific numerical findings may not generalize to other triple sets or model families, although key findings replicate across five models and eight languages.Random-seed variance was negligible, with std. ≤0.002, suggesting 500 triples provide a stable signal for the reported effect sizes.
  • Temporal baseline: The fixed September 2024 Wikidata dump improves replicability but may yield different behavior for time-sensitive facts on newer model iterations.The controlled temporal baseline does not capture all changes in time-sensitive knowledge.
  • Knowledge representation: Wikidata’s regional density, encyclopedic entity-centric format, and binary ground truth may interact with results, leaving transfer to procedural or contextually dependent facts unresolved.Disentangling model-specific factual decay from systemic data imbalances remains future work.
  • Future directions: The findings expose multilingual evaluation challenges beyond aggregate accuracy and support future work on lower-resource languages and complementary notions of semantic plausibility.The stated goal is evaluation that better reflects multilingual reliability and diverse linguistic communities.

A AI Disclosure

The authors used AI assistants for language polishing and coding assistance while retaining responsibility for the paper’s methods, results, and conclusions.

  • AI use: AI assistants Claude and Gemini were used for language polishing and coding assistance.The tools supported writing refinement and implementation code.
  • Responsibility: The authors state that all methods, results, and conclusions remain their sole responsibility.

B Prompt Templates

The prompt templates generate concise multilingual statements from Wikidata triples and then ask models to classify each testing statement as True, False, or Unsure.

  • Statement generation: The generation prompt asks a target-language writing expert to express a supplied relation in one concise, grammatically correct sentence without changing facts.The prompt fills subject, property, and object placeholders in the target language.
  • Statement generation: Generated answers must use the target language and be enclosed within <answer> and </answer> tags.The template prohibits explanations, examples, or text outside the tags.
  • Classification: The evaluation prompt asks whether the testing statement is True or False and permits the labels True, False, or Unsure.The answer must also be enclosed within <answer></answer> tags.

C Computational Setup

The computational setup combines Gemini-2.5-Flash for multilingual statement generation with local evaluation of three open-source models and CPU-based node2vec embeddings.

  • Statement generation: Gemini-2.5-Flash generated multilingual statements through its API.
  • Model evaluation: LLaMA 3.1, Mistral Nemo, and Qwen3 were evaluated locally using VLLM on 6× NVIDIA GeForce RTX 4090 GPUs.The system also used dual Intel Xeon Gold 6326 CPUs and 1TB DDR4 ECC memory.
  • Embedding computation: Knowledge graph embeddings were computed with node2vec using CPU resources and Gensim’s models.word2vec implementation.

D Appendix Tables and Figures

The appendix reports multilingual MMLU conventions and supplementary accuracy visualizations, including alternative Qwen3 think-mode evaluations. Figures compare accuracy, language agreement, and model agreement across languages, distortions, and difficulty levels.

  • Appendix tables: Table 3 reports multilingual MMLU accuracy across models, with Qwen values distinguishing think mode from non-think mode.The table also notes that some Korean and Japanese scores come from supplementary leaderboard results rather than official model cards.
  • Response visualizations: Statement-level visualizations encode correct responses as colored cells, incorrect responses as black, and unsure or invalid responses as white.Figure 6 organizes 500 statements by ORG accuracy and displays language-distortion combinations by row.
  • Response visualizations: Qwen3 think-mode results reduce unsure responses for Qwen3-8B but increase them for Qwen3-14B relative to non-think mode.This comparison is reported in the think-mode version of the statement-level visualization.
  • Difficulty analyses: Accuracy trends are plotted from easiest to hardest difficulty across original, shuffle, and property-based distortion types for each model-language combination.The think-mode trend visualization reports similar overall behavior, with Asian-language degradation most visible in LLaMA and Mistral.
  • Similarity analyses: Language-language similarity shows stronger Western/Asian blocks for LLaMA 3.1 and Mistral Nemo under distortions, while Gemini and Qwen3 remain comparatively stable.Model-model similarity additionally shows lower LLaMA 3.1 agreement with other models, especially in Korean and Japanese.
Loading 2609.09349v1…