PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual vs Crosslingual Retrieval of Fact-Checked Claims: A Tale of Two Approaches

Alan Ramponi, Marco Rovera, Robert Moro, Sara Tonelli

arXiv:2505.22118v2cs.CL

TL;DR

Because task-specific strategies for crosslingual retrieval remained unexplored, this paper compares supervised negative sampling and unsupervised reranking without translation. It finds that LLM-based reranking performs best overall, while the preferred approach differs between multilingual and crosslingual settings.

  • Problem

    Strategies tailored to crosslingual PFCR remained unexplored despite new multilingual embedding models and LLMs and prior gains from fine-tuning with multiple negatives.

    Source: §1 · p. 2

  • Method

    The study compares supervised negative-example selection and unsupervised reranking for multilingual and crosslingual PFCR without translation.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    LLM-based reranking yields the best results overall; supervised similarity-based negatives perform better in mono- or multilingual settings, whereas unsupervised reranking is more effective crosslingually.

    Source: §7 · p. 9, §6.3 · p. 9

  • Takeaways & Limitations

    PFCR systems can be tailored to fact-checkers’ multilingual or crosslingual needs and available computational resources.

    Source: §7 · p. 9

    Some posts and fact-checks that should be positive pairs may be unannotated, potentially affecting model performance and sampled negatives.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Retrieval of previously fact-checked claims is a well-established task, whose automation can assist professional fact-checkers in the initial steps of information verification. Previous works have mostly tackled the task monolingually, i.e., having both the input and the retrieved claims in the same language. However, especially for languages with a limited availability of fact-checks and in case of global narratives, such as pandemics, wars, or international politics, it is crucial to be able to retrieve claims across languages. In this work, we examine strategies to improve the multilingual and crosslingual performance, namely selection of negative examples (in the supervised) and re-ranking (in the unsupervised setting). We evaluate all approaches on a dataset containing posts and claims in 47 languages (283 language combinations). We observe that the best results are obtained by using LLM-based re-ranking, followed by fine-tuning with negative examples sampled using a sentence similarity-based strategy. Most importantly, we show that crosslinguality is a setup with its own unique characteristics compared to the multilingual setup.

1 Introduction

Previously fact-checked claim retrieval can support fact-checkers by reusing verified knowledge, but their needs range from same-language retrieval to finding claims across languages. This paper compares supervised and unsupervised strategies for multilingual and crosslingual retrieval, addressing strategies tailored to crosslingual retrieval that prior work left unexplored.

1 Introduction

  • Fact-checkers may need same-language retrieval, claims in any language for global narratives, or claims in a different language from the input.The paper distinguishes multilingual retrieval from crosslingual retrieval, where the input and retrieved claim differ in language.Source: §1 · pp. 1–2
  • Prior work mostly studied monolingual PFCR; only two works addressed crosslingual setups, where translation into English showed potential, especially for low-resource languages.Recent multilingual embedding models and LLMs, and prior gains from fine-tuning with multiple negatives, left task-specific crosslingual strategies unexplored.Source: §1 · p. 2
  • The study compares supervised and unsupervised PFCR without translation, asking which models and negative-sampling strategies work and how crosslingual retrieval differs from multilingual retrieval.Experiments use a curated MultiClaim subset.Source: §1 · p. 2
  • The curated dataset covers 283 post–fact-check language combinations and splits both posts and fact-checks to prevent data contamination.The authors publish the subset and splits for reproducibility.Source: §1 · p. 2

2 Related Work

Prior PFCR research spans datasets and retrieval methods, but identified post–fact-check pairs are often same-language, leaving crosslingual retrieval comparatively underexplored. Methods include embedding-based retrieval, fine-tuning, reranking, and LLM prompting, without one strategy established as best overall.

2 Related Work

  • PFCR is a standard manual and automated fact-checking task, also used as a content-credibility signal, with shared tasks at CLEF and SemEval.The passage names claim matching and claim detection as alternate names.Source: §2 · p. 2
  • Multilingual datasets vary in scale, language coverage, pair identification, and post sources; crosslingual retrieval remains underexplored because identified pairs are often same-language.MuMiN, MultiClaim, and MMTweets are among the linguistically diverse resources described.Source: §2 · pp. 2–3
  • PFCR systems commonly use text embeddings for similarity search, sometimes adding reranking or fine-tuning; LLM prompting has not yielded one best overall strategy.Prior work also used multiple negative examples during fine-tuning, and this paper compares a wider range of selection strategies and fine-tuned models.Source: §2 · p. 3

3 Dataset

The study curates a multilingual and crosslingual PFCR subset from MultiClaim v2, using original-language post text and fact-checks linked to posts. Its splits separate fact-checks as well as posts, and the evaluation covers 47 languages and 283 language combinations.

3 Dataset

  • MultiClaim v2 contains post–fact-check pairs, potentially in different languages, sourced from fact-checking articles, ClaimReview links, and platform fact-check labels.The subset is drawn from the publicly available dataset.Source: §3 · p. 3
  • Curation keeps original-language post text, omits potentially noisy OCR text, and includes languages with at least 200 posts and claims paired with at least one post.The additional filtering results in a cutoff of 180 posts to ensure non-overlapping fact-checks across splits.Source: §3 · p. 3
  • Posts, fact-checks, and pairs are split by language into training, development, and test sets, with 10% allocated to each held-out set.Fact-checks do not cross splits, preventing data contamination and supporting a less biased retrieval-performance estimate.Source: §3 · p. 3
  • The dataset covers 47 languages—30 in posts and 46 in fact-checks—and 283 language combinations, which the authors describe as the most linguistically diverse PFCR dataset to date.That assessment is qualified as being to the authors’ best knowledge.Source: §3 · p. 3

4 Methodology

The paper frames PFCR as ranking fact-checked claims for an input post, using unsupervised dense retrieval with reranking or supervised embedding fine-tuning. Its supervised methods sample challenging negatives by topic or semantic similarity to address rapid saturation from random negatives.

4 Methodology

  • PFCR ranks a set of fact-checked claims so the gold claim for each post appears as high as possible, using embeddings for posts and claims.The paper designs both unsupervised and supervised approaches.Source: §4 · p. 3
  • Unsupervised retrieval first ranks fact-checks by dense similarity, then reranks them using cross-encoders or LLMs.Cross-encoders score query–document pairs, while LLM-based rerankers generate a ranking of documents for a query.Source: §4 · pp. 3–4
  • Randomly sampled negatives can become easy to distinguish after a few training iterations, so the supervised approach explores challenging negative pairs.Positive examples come from the dataset’s paired posts and fact-checks.Source: §4 · p. 4
  • The negative-sampling strategies select claims from the post’s topic cluster or the top-k most semantically similar claims, excluding existing positive pairs.Topic negatives are sampled within the post’s cluster; similarity negatives are selected by cosine similarity. Random sampling is the baseline, and the number of negatives varies across experiments.Source: §4 · p. 4
  • Negative examples are generated offline and serialized for training to avoid costly computation during fine-tuning and support reproducibility.The experiments compare varying numbers of negatives across the sampling strategies and the random baseline.Source: §4 · p. 4

5 Experimental Setup

The experiments compare unsupervised retrieval and re-ranking with supervised retrieval across multilingual and crosslingual settings, using retrieval metrics that account for whether the paired claim appears in the top ten and its rank.

5 Experimental Setup

  • The test evaluation compares unsupervised retrieval with and without re-ranking against supervised retrieval using three negative-sampling strategies.The multilingual setting includes monolingual and crosslingual pairs; the crosslingual setting includes only pairs whose post and fact-check are in different languages.Source: §5 · p. 4

5.1 Unsupervised Text Embedding Models

  • The study evaluates multilingual embedding models spanning diverse architectures, pre-training techniques, parameter counts (278M–7B), and data, including synthetic data, on both the test set and full claim collection.The test set contains over 6,100 claims, versus over 52,000 per post in the full set; reported re-ranking uses top-n = 30.Source: §5.1 · pp. 4–5
  • RankGPT re-ranking is tested on the three strongest retrieval models, while a cross-encoder provides a second re-ranking approach.The three models were selected for strong retrieval and comparison with the supervised multilingual-e5-large model.Source: §5.1 · p. 5

5.2 Supervised Text Embedding Models

  • Supervised experiments fine-tune multilingual-e5-large and paraphrase-multilingual-mpnet-base-v2, using multiple negatives ranking loss and tuned learning rate, batch size, and warm-up steps.The models span 278M–560M parameters and were selected for fine-tuning without costly computing infrastructure.Source: §5.2 · p. 5
  • k = 10 negatives is selected for test evaluation because it provides the best overall development-set performance across strategies and models.For multilingual-e5-large, similarity sampling outperforms topic and random sampling on development-set S@10, with the best results at k = 10.Source: §5.2 · p. 5

6 Results and Discussion

Across the evaluated approaches, LLM-based re-ranking leads overall, while supervised similarity-based negative sampling and re-ranking show different strengths across multilingual and crosslingual settings. Results also indicate that crosslingual retrieval has distinct performance patterns and trade-offs.

6.1 Unsupervised Results

  • Multilingual and crosslingual leaders often differ, and smaller encoder-only models can challenge or outperform larger decoder-only models, especially crosslingually.The analysis found no correlation between embedding dimension or parameter count and performance.Source: §6.1 · pp. 6–7
  • In multilingual retrieval, text-embedding-3-large leads; crosslingual retrieval has bge-multilingual-gemma2 leading the test set and gte-multilingual-base the full claim set, outperforming text-embedding-3-large by 3.27 and 3.58 MRR@10 points, respectively.Source: §6.1 · pp. 5–6
  • Cross-encoder re-ranking has clearer benefits crosslingually than multilingually, while LLM-based re-ranking yields the strongest overall results and average MRR@10 gains of 4.54, 13.80, and 6.93 points across the three settings.The gains refer respectively to multilingual, crosslingual, and crosslingual-without-English settings.Source: §6.1 · p. 6, Figure 3 · p. 7

6.2 Supervised Results

  • Similarity sampling consistently beats topic and random sampling; multilingual-e5-large improves over random by 1.45 MRR@10 points multilingual and 5.31 crosslingually.Topic sampling also beats random but trails similarity sampling in both settings.Source: §6.2 · pp. 7–8
  • With k = 10 similarity-sampled negatives, crosslingual MRR@10 reaches 0.4947 (+6.24 points) and S@10 reaches 0.7076 (+7.52 points) over unsupervised multilingual-e5-large.Topic sampling is more effective with at most five negatives, while similarity sampling is most effective with more negatives.Source: Figure 4 · p. 8, §6.2 · p. 8

6.3 Comparing Unsupervised and Supervised Approaches

  • Re-ranking and negative sampling provide more substantial benefits crosslingually; English- and Hindi-focused evaluations show supervised similarity sampling stronger in monolingual or multilingual settings, but re-ranking stronger crosslingually.The authors identify crosslingual retrieval as a distinct context requiring further investigation.Source: §6.3 · pp. 8–9
  • 0.8042 and 0.6140 MRR@10: LLM-based re-ranking leads in multilingual and crosslingual settings, while similarity-tuned retrieval surpasses cross-encoder re-ranking in multilingual evaluation.Cross-encoder re-ranking reaches 0.5442 MRR@10 in crosslingual evaluation.Source: §6.3 · p. 8, Table 2 · p. 9
  • Fine-tuning needs sufficient training data and adds computation; unsupervised retrieval needs no training data and scales to larger datasets, but LLM-based re-ranking is computationally costly.LLM-based re-ranking was tested on a limited number of models.Source: §6.1 · p. 7, §6.3 · p. 9

7 Conclusion

The experiments find that multilingual and crosslingual retrieval behave differently, with crosslingual retrieval more challenging. Unsupervised LLM-based re-ranking performs best, even outperforming the best supervised approach.

7 Conclusion

  • Crosslingual retrieval is more challenging than multilingual retrieval, and unsupervised LLM-based re-ranking yields the best results, outperforming the best supervised approach.Source: §7 · p. 9
  • The study highlights the importance of evaluating embedding models, re-ranking, and negative sampling for retrieval performance.Source: §7 · p. 9
  • The findings could guide PFCR systems tailored to fact-checkers’ needs and available computational resources.Source: §7 · p. 9

Limitations

The study identifies limitations related to incomplete positive-pair annotations, uneven language representation, and automatic language detection. These issues may affect model results and their applicability across different data distributions.

Limitations

  • Some post–fact-check pairs that should be positive may be unannotated, affecting model performance and sampled negatives.This can occur because crosslingual positive-pair annotation is often lacking in fact-checker databases.Source: Limitations · p. 10
  • Language representation is uneven: English posts pair with claims in nearly all dataset languages, while Dutch and Romanian have few crosslingual pairs.Findings could change with a different distribution of languages and crosslingual pairs.Source: Limitations · p. 10
  • Automatic language detection may leave some noise in assigned language labels despite combining approaches and correcting outliers.The paper gives Latin being recognized as a language as an example of an outlier case.Source: Limitations · p. 10

Ethics Statement

The work reuses MultiClaim v2 under its research-only terms and conditions. It publishes dataset IDs and associated metadata, splits, negative examples, and code, and positions its models as assistance for human reviewers rather than automated fact-checking.

Ethics Statement

  • The MultiClaim v2 subset is reused under the dataset’s research-only terms and conditions.Source: Ethics Statement · p. 10
  • The authors publish IDs, language metadata, data splits, negative-example lists, and code, but do not redistribute their data subset.Source: Ethics Statement · p. 10
  • The fine-tuned embedding models are intended to assist fact-checkers or moderators, not to ascertain claim veracity automatically.Source: Ethics Statement · p. 10

A Dataset

The dataset subset supports multilingual and crosslingual fact-checked claim retrieval, and the paper reports unsupervised results for retrieval and re-ranking across both settings.

A Dataset

  • The dataset uses original-language post and claim texts and keeps pairs linked by claim_review or backlink relationships.OCR-extracted image text and fact-check titles are omitted.Source: A Dataset · p. 13
  • The authors replace MultiClaim v2’s Google Translate language identification with an open-source pipeline combining four language detectors.They filter rare detector outputs, discard averaged scores below 0.5, and select the remaining language with the highest average score.Source: A Dataset · p. 13
  • Posts are filtered by language frequency, while all associated claims are retained regardless of language; rare claim-language labels were manually corrected after review.The post-language cutoff was at least 200 posts before further filtering, and labels appearing fewer than 10 times were checked.Source: A Dataset · p. 13
  • The subset covers 47 languages: 30 in posts and 46 in fact-checked claims, yielding 283 language combinations.All post languages appear in claims except Urdu, which is represented only in posts.Source: A Dataset · p. 13

B Unsupervised Approach

  • The unsupervised section reports retrieval and cross-encoder re-ranking results on the test and full claim sets, with graphical results also including LLM-based re-ranking.It also reports S@10 and MRR@10 differences between retrieval and cross-encoder re-ranking, plus an embedding-dimension/model-size correlation analysis.Source: B Unsupervised Approach · p. 13

C Supervised Approach

The supervised section examines negative-sampling strategies and the number of negative examples, reporting multilingual and crosslingual S@10 results for selected models and splits.

C Supervised Approach

  • The hyperparameter search varied learning rate, batch size, warm-up steps, and the number of negatives k.The tested k values were 1, 2, 3, 4, 5, and 10.Source: C Supervised Approach · p. 13
  • The reported supervised results include multilingual S@10 for fine-tuned paraphrase-multilingual-mpnet-base-v2 on development data and crosslingual S@10 for multilingual-e5-large on test data.Both figures average three runs with different random seeds and plot results across negative-sampling strategies and values of k.Source: C Supervised Approach · pp. 13–14
  • Table 4 lays out language combinations with post languages in columns and fact-checked claim languages in rows across all data splits.Languages are represented by ISO 639-1 two-letter codes.Source: Table 4 · p. 14

D Additional Results

The additional results compare translated and original-language data and report performance across multilingual and crosslingual settings. Translation performs differently across those settings in the reported results.

D Additional Results

  • 0.6996 MRR@10 and 0.7647 S@10 were achieved by the English-translated multilingual-e5-large model on the multilingual test set, which was outperformed by using original-language data.The authors present the translated-data comparison despite focusing on multilingual and crosslingual retrieval.Source: D Additional Results · p. 14
  • 0.6017 MRR@10 and 0.7514 S@10 were achieved with translated data in the crosslingual test setting; these were respectively the second-best MRR@10 and best S@10 results in the cited comparisons.The passage states translation can still help when the post and fact-check differ in language.Source: D Additional Results · p. 14
  • Tables 5 and 6 report retrieval and cross-encoder re-ranking performance across models on test and full claim sets in multilingual and crosslingual settings.Figures 5 and 6 show S@10 performance across models for retrieval and re-ranking in the respective settings.Source: Table 5 · p. 15, Table 6 · p. 15, Figure 5 · p. 16
  • Figures 7–12 report S@10 or MRR@10 differences (re-ranking minus retrieval) for multilingual test and full sets and crosslingual test sets, using bge-reranker-v2-m3 with top-n = 30.Source: Figure 7 · p. 17, Figure 8 · p. 17, Figure 9 · p. 17
  • Table 8 reports Pearson correlations and p-values between embedding dimension or model size and MRR@10 in the unsupervised setting.The analysis covers 16 embedding models.Source: Table 8 · p. 19
  • Table 9 compares test-set S@10 across multilingual, crosslingual, and crosslingual-without-English settings, covering negative-sampling strategies and unsupervised retrieval and re-ranking approaches.Supervised results are averages with standard deviations over three runs using different seeds.Source: D Additional Results · p. 14, Table 9 · p. 20
Loading 2505.22118v2…