PaperlayerView exact revision on arXiv

Source-linked AI summary

When Scale Meets Diversity: Evaluating Language Models on Fine-Grained Multilingual Claim Verification

Hanna Shcharbakova, Tatiana Anikina, Natalia Skachkova, Josef van Genabith

arXiv:2507.20700v1cs.CL

TL;DR

Multilingual claim verification with fine-grained truth categories remains understudied, despite the need to assess misinformation across languages. This paper benchmarks five language models on X-Fact and finds that XLM-R substantially outperforms the tested larger LLMs.

  • Problem

    Multilingual verification beyond binary and ternary labels remains understudied, although fine-grained categories capture nuanced truth assessment.

    Source: §1 · p. 1

  • Method

    The paper benchmarks five multilingual models on X-Fact’s seven veracity categories, using fine-tuning, prompting, and claims with or without evidence.

    Source: §1 · p. 2

  • Results

    57.7% macro-F1 for XLM-R exceeds the best LLM result, Qwen 2.5’s 16.9% macro-F1.

    Source: §6 · p. 7

  • Takeaways & Limitations

    XLM-R outperforms the tested larger LLMs on fine-grained multilingual claim verification, even with a frozen base and trained classification head.

    Source: §6 · p. 8, §7 · p. 8

    The 7-shot prompting approach may not be equally optimal across all languages.

    Source: §8 · p. 9

Abstract

from arXiv · show

The rapid spread of multilingual misinformation requires robust automated fact verification systems capable of handling fine-grained veracity assessments across diverse languages. While large language models have shown remarkable capabilities across many NLP tasks, their effectiveness for multilingual claim verification with nuanced classification schemes remains understudied. We conduct a comprehensive evaluation of five state-of-the-art language models on the X-Fact dataset, which spans 25 languages with seven distinct veracity categories. Our experiments compare small language models (encoder-based XLM-R and mT5) with recent decoder-only LLMs (Llama 3.1, Qwen 2.5, Mistral Nemo) using both prompting and fine-tuning approaches. Surprisingly, we find that XLM-R (270M parameters) substantially outperforms all tested LLMs (7-12B parameters), achieving 57.7% macro-F1 compared to the best LLM performance of 16.9%. This represents a 15.8% improvement over the previous state-of-the-art (41.9%), establishing new performance benchmarks for multilingual fact verification. Our analysis reveals problematic patterns in LLM behavior, including systematic difficulties in leveraging evidence and pronounced biases toward frequent categories in imbalanced data settings. These findings suggest that for fine-grained multilingual fact verification, smaller specialized models may be more effective than general-purpose large models, with important implications for practical deployment of fact-checking systems.

1 Introduction

Fine-grained multilingual fact verification remains understudied despite the growth of automated verification and language models. The paper benchmarks five models on X-Fact and reports new state-of-the-art results alongside analysis of model behavior.

1 Introduction

  • Most advances in automated verification have focused on English, while multilingual assessment beyond binary or ternary labels remains understudied.Fine-grained labels better capture nuanced truth assessment.Source: §1 · p. 1
  • The benchmark evaluates XLM-R, mT5, Llama 3.1, Qwen 2.5, and Mistral Nemo on X-Fact’s seven veracity categories across multiple languages.It uses fine-tuning for smaller models and prompting and parameter-efficient fine-tuning for larger models, with claims alone or paired with evidence.Source: §1 · pp. 1–2
  • A 15.8% macro-F1 improvement over the previous best establishes new state-of-the-art results on seven-category multilingual X-Fact verification.The study also reports a substantial gap between encoder-based and decoder-only architectures despite the latter’s greater size and general capabilities.Source: §1 · p. 2
  • The paper analyzes model behaviors and error patterns across architectures to identify factors that may influence multilingual verification performance.The observations are presented as potentially informative for future research on verification across diverse languages.Source: §1 · p. 2

2 Related Work

Prior multilingual fact-verification datasets vary in language coverage and label schemes, while prior methods report mixed results across tasks and model types. The cited work also highlights cross-lingual generalization challenges and the limits of conclusions from binary-classification settings.

2.1 Multilingual Fact Verification Datasets

  • X-Fact provides 31K claims across 25 languages and 11 language families, with seven labels; varied annotation schemes complicate cross-dataset evaluation.Other datasets described use binary, three-category, four-category, or eleven-category schemes.Source: §2.1 · p. 2

2.2 Methods for Fact Verification

  • Gupta and Srikumar (2021) reported 41.9% F1 on X-Fact’s in-domain test, falling to 16.2% out-of-domain and 16.7% zero-shot.The authors characterize these drops as highlighting cross-lingual generalization challenges.Source: §2.2 · p. 2
  • Cao et al. (2023) found carefully crafted prompts with explicit output-format instructions and task definitions improved fact-checking; Hu et al. (2023) found few-shot gains beyond a threshold; majority voting across LLM runs and iterative self-refinement also improved performance.Hu et al. (2023) found CoT prompting significantly improved performance across all tested models on English data compared with standard prompting.Source: §2.2 · pp. 2–3
  • Prior comparisons find that model rankings vary by task: GPT-4 variants can outperform traditional models on binary English LIAR, while traditional models do better on multi-way classification.Other cited work reports strong binary-classification results from Turkish-fine-tuned LLaMA-2 and the highest NewsPolyML performance from mBERT.Source: §2.2 · p. 3

3 Dataset

The paper uses X-Fact, a 31,189-claim multilingual dataset with seven standardized veracity labels and separate tests for in-domain, source-shifted, and zero-shot evaluation. Its subsets vary in label distribution, and the training data also varies substantially by language.

Dataset construction

  • X-Fact contains 31,189 claims across 25 languages and 11 language families, with up to five evidence pieces per claim and 4.75 non-empty pieces on average.Claims include metadata such as language, fact-checking source, claim and review dates, and claimant information.Source: §3 · p. 3

Label scheme

  • Seven standardized labels range from true to false and include partly true/misleading, complicated/hard to categorize, and other.The scheme is intended to provide more nuanced veracity assessment than less fine-grained alternatives.Source: §3 · p. 3

Dataset splits

  • The dataset separates in-domain testing (3,826 claims), out-of-domain testing (2,368), and zero-shot testing (3,381 claims from 12 unseen languages).Training contains 19,079 claims across 13 languages, and development contains 2,535 claims across 13 languages.Source: §3 · p. 3, Table 1 · p. 4

Label distribution

  • False dominates the training labels at 7,515 instances (39.4%), while other is least represented at 576 (1.9%).The label distribution varies across subsets.Source: §3 · p. 4, Figure 1 · p. 4

Language distribution

  • Portuguese accounts for 5,601 training claims (29.4%), while Serbian has the lowest representation at 624 (3.3%).The authors note that these imbalances may affect learning, especially cross-lingual transfer and fine-grained label learning.Source: §3 · p. 4

4 Experiments

The experiments compare multilingual models of different architectures using fine-tuning, prompting, and evidence inputs. Small models are fine-tuned, while large models are evaluated with both few-shot prompting and LoRA.

4.1 Experimental Setup

  • The study compares SLMs and LLMs on X-Fact’s seven-category task, selecting multilingual models with architectural diversity and open-source availability.The set includes XLM-R and mT5 as smaller models and Llama 3.1, Qwen 2.5, and Mistral Nemo as decoder-only LLMs.Source: §4.1 · p. 4, Table 2 · p. 4

4.2 Small Language Models Experiments

  • XLM-R and mT5 receive either full-model fine-tuning or classification-head-only fine-tuning with the base model frozen.Both setups use claims with evidence; claim-only experiments were omitted after preliminary results confirmed worse performance.Source: §4.2 · pp. 4–5

4.3 Large Language Models Experiments

  • LLMs are evaluated with 7-shot prompts and LoRA in both claim-only and claim-plus-evidence configurations.Prompts contain examples for each veracity category, task instructions, and category definitions.Source: §4.3 · p. 5

5 Results

On X-Fact, XLM-R substantially outperforms the tested models, while LLM performance varies with model, input, and evaluation subset. The results also show larger macro-/micro-F1 gaps under fine-tuning than few-shot prompting.

5.1 SLMs Performance

  • 57.7% macro-F1 on the test set lets fully fine-tuned XLM-R exceed the previous mBERT state of the art, 41.9%, by 15.8%.XLM-R also maintains relatively strong performance across evaluation subsets.Source: §5.1 · p. 5, Table 3 · p. 5
  • 47.6% test-set macro-F1 for mT5 falls to 22.2% out-of-domain and 19.2% zero-shot, unlike XLM-R’s stronger generalization.Source: §5.1 · p. 5

5.2 LLMs Performance

  • 16.9% test-set macro-F1 is the best LLM result, from Qwen 2.5 claim-only fine-tuning, 40.8 points below XLM-R.Table 4 presents LLM macro- and micro-F1 scores across configurations; bold values mark the highest scores for each model-subset combination.Source: §5.2 · p. 5, Table 4 · p. 6
  • LoRA fine-tuning generally beats few-shot prompting, but Mistral Nemo has configurations favoring few-shot, including 16.1% versus 12.1% out-of-domain with claim+evidence.Qwen shows the largest fine-tuning gains, while Mistral Nemo shows minimal differences between methods.Source: §5.2 · p. 6
  • Evidence effects differ by model: Qwen’s claim-only fine-tuning beats claim+evidence, Mistral Nemo’s few-shot result reverses that pattern, and Llama 3.1 is inconsistent.Llama 3.1 falls to 9.6% macro-F1 in its worst zero-shot configuration, claim+evidence fine-tuning.Source: §5.2 · p. 6
  • All models decline from test to out-of-domain and zero-shot under fine-tuning; Qwen’s test-to-zero-shot drop is 5.2%, while Llama falls from 15.5% to 9.6%.Mistral Nemo varies least and performs best out-of-domain at 16.1%.Source: §5.2 · p. 6
  • Across LLMs, macro-/micro-F1 gaps are larger under fine-tuning than few-shot; Qwen’s test-set claim+evidence gap is 23.6% versus 12.2%.Qwen’s fine-tuned test-set scores are 39.5% micro-F1 and 15.9% macro-F1.Source: §5.2 · pp. 6–7

6 Discussion

The discussion highlights XLM-R’s advantage over larger decoder-only models on fine-grained multilingual verification, alongside challenges in evidence use, category imbalance, and generalization. The authors offer possible explanations and identify further analysis needed.

Performance Gap Between Model Types

  • XLM-R reaches 57.7% macro-F1 versus Qwen 2.5’s best 16.9%, despite the LLMs’ much larger parameter counts.The authors suggest masked-language-model pre-training may align better with classification than LLMs’ next-token objectives, but frame this as a possible contributor.Source: §6 · p. 7
  • Frozen XLM-R still scores 51.4% macro-F1 versus the best LLM’s 16.9%, suggesting the gap extends beyond full fine-tuning versus LoRA.The authors call for per-label analysis to identify model biases and difficult veracity categories.Source: §6 · p. 7

Evidence Integration Patterns

  • Added evidence often fails to help LLMs; the authors hypothesize autoregressive attention or input formatting may contribute, while ruling out evidence truncation.The authors say evidence-based reasoning is fundamental to reliable verification, and suggest LLMs may not use the added information effectively or may be confused by input complexity.Source: §6 · p. 7

Cross-Lingual and Cross-Domain Generalization

  • Models may develop language-specific biases that hinder generalization, particularly to languages underrepresented in training.Source: §6 · p. 8

Fine-Grained Classification Challenges

  • The authors link frequent-category dominance to poor rare-category prediction; X-Fact’s seven labels require distinguishing subtle gradations of partial truth.The passage also reports Portuguese at 29.4% and Serbian at 3.3% of training data.Source: §6 · pp. 7–8

Cross-Lingual and Cross-Domain Generalization

  • XLM-R shows the most robust cross-lingual transfer, while sharper LLM declines may indicate weaker robustness to underrepresented languages or domain shifts.The authors point to XLM-R’s relatively stable performance on unseen languages and suggest its multilingual pre-training provides effective cross-lingual representations.Source: §6 · p. 8

7 Conclusion and Future Work

The evaluation finds that fully fine-tuned XLM-R leads the tested models on seven-category multilingual fact verification, while LLMs show difficulties using evidence and handling imbalanced labels. The results suggest smaller specialized models may outperform general-purpose large models for this task.

7 Conclusion and Future Work

  • The study evaluates diverse model architectures and sizes on X-Fact’s seven-category multilingual fact-verification task.Its findings concern how these models handle fine-grained veracity classification across languages.Source: §7 · p. 8
  • 57.7% macro-F1 for fully fine-tuned XLM-R exceeds the best LLM’s 16.9% and improves 15.8% over the previous state of the art.The best LLM was Qwen 2.5; frozen XLM-R also reached 51.4% macro-F1 against the best LLM’s 16.9%.Source: §7 · p. 8
  • LLMs often perform worse with claim-evidence pairs than with claims alone, indicating difficulty leveraging external information for verification.The authors note that evidence-based reasoning is fundamental to reliable fact-checking, while the underlying causes need further exploration.Source: §7 · p. 8
  • Models favor frequent categories and struggle with rare veracity labels, with this imbalance bias appearing more pronounced in LLMs.The paper relates disparities between micro- and macro-F1 scores to imbalanced fine-grained categories.Source: §7 · p. 8
  • For fine-grained multilingual fact verification, smaller specialized models may perform better while requiring fewer computational resources than LLMs.The paper presents this as an implication of its results, not as a claim about all NLP tasks.Source: §7 · p. 8

8 Limitations

The authors qualify their comparison by noting differences in training methods, possible language-specific prompt disadvantages, and limits on interpreting evidence use. They also frame the comparison around practical computing constraints and identify output analysis as future work.

8 Limitations

  • XLM-R is fully fine-tuned, whereas LLMs use LoRA, a methodological difference that may partially explain the performance gap.The authors caution that this difference could affect how well LLMs adapt to the task.Source: §8 · pp. 8–9
  • The seven-shot prompts balanced veracity labels but prioritized label coverage over linguistic diversity, potentially favoring languages represented in the examples.Language-specific prompt optimization could potentially narrow the performance gap but would require substantial additional engineering for each target language.Source: §8 · p. 9
  • Small performance differences between claim-only and claim-plus-evidence settings do not establish that LLMs cannot use evidence.The authors note that task difficulty, evaluation choices, prompting, evidence formats, dataset size, or search-snippet quality may affect results.Source: §8 · p. 9
  • The comparison reflects limited-compute conditions: fully fine-tuning billion-parameter models may be prohibitive for many researchers.The authors contrast this with LoRA, which can be applied with modest computational resources.Source: §8 · p. 9
  • Detailed analysis of released LLM outputs, including confusion matrices and error patterns, remains future work.The repository includes model predictions and responses to support further investigation.Source: §8 · p. 9

A Model Implementation Details

The experiments compare two small multilingual models with three larger decoder-only models. X-Fact examples are presented with translations for reference.

A Model Implementation Details

  • XLM-R (270 million parameters) was tested with either a fine-tuned classification head over frozen parameters or full fine-tuning.The selected XLM-R base model was pretrained on text in 100 languages.Source: A Model Implementation Details · p. 11
  • The evaluated models also included encoder-decoder mT5 base (580 million parameters), instruction-tuned Llama 3.1 8B, Qwen 2.5 7B, and Mistral Nemo 12B.The listed models differ in parameter count and stated language support.Source: A Model Implementation Details · p. 11

B Details on X-Fact

  • X-Fact examples are shown with translations for reference.The figure presents examples from the dataset as presented in Gupta and Srikumar (2021).Source: B Details on X-Fact · p. 11

C.1 Small Language Models

The small-language-model experiments used Bayesian hyperparameter optimization and standard regularization and scheduling choices; Table 5 lists key settings.

C.1 Small Language Models

  • The XLM-R and mT5 experiments used Bayesian hyperparameter optimization, with 90 sweeps for classification-head training and 60 sweeps each for full fine-tuning.The passage reports separate sweep counts for the two training approaches.Source: §C.1 · pp. 11–12
  • AdamW, a polynomial learning-rate scheduler, and early stopping were used in the small-model experiments.Early stopping was used to prevent overfitting.Source: §C.1 · p. 12
  • Table 5 presents key hyperparameter values for the small language model variants.Source: Table 5 · p. 12, §C.1 · p. 12

C.2 Large Language Models

The LLM experiments used LoRA fine-tuning and prompt-based evaluation, with figures and a table organizing macro- and micro-F1 comparisons across models and evaluation subsets.

C.2 Large Language Models

  • LoRA used rank 16 and adapter alpha 32, targeting attention projections and feedforward layers.Lower rank and alpha settings performed worse, while higher settings gave negligible gains and substantially increased memory requirements.Source: §C.2 · p. 12
  • Prompt temperatures from 0.3 to 0.5 provided the best balance between confident predictions and appropriate uncertainty handling.Lower temperatures were overly deterministic, while higher temperatures produced inconsistent classifications.Source: §C.2 · p. 12

D Performance Comparison across Models

G LLMs Performance Summary Table

  • Table 6 compares claim+evidence macro-F1 across Test, OOD, and Zero-shot subsets, including small-model results alongside LLM results.The table combines claim+evidence results from Table 4 with the small language model results; its caption identifies highest LLM scores across Few-shot and LoRA methods.Source: G LLMs Performance Summary Table · p. 13, Table 6 · p. 14

E Prompt Template

H Micro- and Macro-F1 Scores Comparison across LLMs

Related papers

Loading 2507.20700v1…