Source-linked AI summary
Novel Claim or Déjà Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking
Haorui He, Xinwen Chen, Dacheng Wen, Reynold Cheng, Francis C. M. Lau, Yupeng Li
TL;DR
Dynamic MAFC benchmarks are assumed to avoid contamination, but this paper empirically tests that assumption across static and newly collected dynamic claims. It finds that dynamic evaluation reduces contamination without eliminating it, with 17.09%–29.30% of claims potentially contaminated and performance inflation reaching 11.34 Macro-F1 points.
Problem
Existing dynamic MAFC benchmarks assume post-cut-off claims are uncontaminated, but empirical evidence is needed to assess residual contamination and its effect on evaluation.
Method
The study compares AVeriTeC with the newly curated ClaimReview2025Q4 benchmark and empirically analyzes contamination sources and evaluation effects across SOTA LLMs.
Results
17.09%–29.30% of dynamic-benchmark claims remain potentially contaminated, while pre-cut-off public knowledge can enable verification and contamination can inflate Macro-F1 by up to 11.34 points.
Takeaways & Limitations
Timestamp-based filtering alone is insufficient for trustworthy MAFC evaluation, motivating stronger controls for claim novelty and contamination.
Takeaways & Limitations
The curated dynamic benchmark is created solely to enable the contamination analysis, while the paper’s main contribution is empirical rather than benchmark introduction.
Abstract
from arXiv · showhide
Multimodal automated fact-checking (MAFC) verifies claims by retrieving and reasoning over external evidence. However, most existing static benchmarks risk contamination: they primarily consist of outdated claims verifiable using an LLM's internal knowledge without external evidence. This can inflate performance estimates and fail to reflect true capability on novel claims that require up-to-date information. To address this, emerging dynamic benchmarks collect claims published after LLMs' knowledge cut-off dates, assuming they are uncontaminated. This work revisits this assumption by empirically studying contamination risks in both the state-of-the-art (SOTA) static AVeriTeC benchmark and our newly constructed dynamic ClaimReview2025Q4 benchmark, as well as their impact on MAFC evaluation. Our experiments yield 16 findings, highlighting three key results: (1) Dynamic evaluation reduces but does not eliminate contamination risks, as 17.09\%--29.30\% of post-cut-off claims remain potentially contaminated; (2) Many newly published claims can be verified either directly or by synthesizing multiple pieces of public knowledge available before the cut-off; and (3) Contamination can induce statistically significant inflation in MAFC performance, increasing Macro-F1 by up to 11.34 points and distorting system rankings. In light of these findings, we re-evaluate SOTA LLMs under a strictly contamination-controlled setting. Our study provides practical guidelines for trustworthy MAFC evaluation.
1 Introduction
MAFC addresses the societal challenge of widespread multimodal misinformation by retrieving and reasoning over online evidence, but static benchmarks risk contamination from outdated claims. This study investigates contamination in static and dynamic evaluation and shows that post-cutoff claims can remain potentially contaminated.
- Motivation: MAFC systems use multimodal LLMs to retrieve and analyze textual or visual online evidence for claim verification.This automation is motivated by the scale of misinformation, which professional fact-checkers cannot address alone.
- Contamination problem: Static benchmarks such as AVeriTeC are not updated, so claims about past events may be verifiable from LLM pretraining rather than external evidence.Their outdated coverage creates a contamination risk for MAFC evaluation.
- Contamination problem: Dynamic benchmarks may still contain déjà vu claims because post-cutoff claims can be verified using information available before the cutoff.This can overestimate performance on truly unseen claims and distort MAFC evaluation.
- Research questions and methods: Potential contamination is measured by comparing evidence sufficiency from an LLM’s internal knowledge with oracle evidence used by human fact-checkers.The pipeline is applied to six SOTA LLMs across static and dynamic benchmarks.
- Research questions and findings: 17.09%–29.30% of claims remain potentially contaminated under dynamic evaluation.The study compares AVeriTeC3 with ClaimReview2025Q4, which contains claims published in Q4 2025 to simulate post-cut-off evaluation.
2 Related Work
Automated fact-checking retrieves evidence to classify claims, while multimodal misinformation has motivated research beyond predominantly text-based systems. Existing evaluations rely largely on contamination-prone static benchmarks, prompting dynamic alternatives and leaving autonomous-agent evaluation comparatively underexplored.
- AFC systems retrieve relevant evidence and predict verdicts such as Supported, Refuted, or Not Enough Evidence, often with explanatory justifications.
- Static benchmarks including AVeriTeC, MOCHEG, VERITE, and AVerImaTeC risk contamination, compromising evaluation reliability.
- 28.68% of AVeriTeC claims either contain multimodal content or require multimodal reasoning for fact-checking.This manual evaluation underscores the multimodal scope of an otherwise static benchmark.
- Dynamic evaluation continuously introduces unseen, time-stamped data to improve robustness and mitigate contamination risks; MAFC-specific examples include XFACTA and LiveBench.LiveBench updates evaluation data monthly, while XFACTA is no longer actively maintained and was last updated in August.
- Prior MAFC contamination research examined human-specified workflows on static benchmarks, whereas this work studies fully autonomous SOTA agents and emerging dynamic benchmarks.
3 Contamination Detection for MAFC
This section defines contamination in multimodal automated fact-checking as the ability to generate sufficient oracle-like evidence from an LLM’s internal knowledge. It operationalizes detection through evidence generation, extraction, Hungarian similarity matching, and evaluation on static and dynamic benchmarks.
- Task Definition: MAFC first retrieves multimodal evidence from external sources and then aggregates and reasons over it to assign a claim-veracity label.Evidence may contain textual and/or visual information.
- Contamination Detection Pipeline: The detection pipeline prompts a target LLM to generate a fact-checking article, eliciting the model’s internal parametric knowledge for each benchmark claim.The generated article is compared against a human-written oracle fact-checking article crawled from fact-checking agencies.
- Contamination Detection Pipeline: GPT-4o-Mini extracts claim-relevant textual evidence from both oracle and generated articles, producing oracle E and generated Ẽ evidence sets.Although extracted evidence is textual, it originates from source articles used in the fact-checking comparison.
- Contamination Detection Pipeline: Hungarian one-to-one matching compares generated and oracle evidence, normalizes by |E|, and yields a claim-level contamination score for each claim and target LLM.The normalization prevents artificial inflation from over-generation while penalizing omissions of oracle evidence.
- Benchmarks: The evaluation uses AVeriTeC’s 500-claim development set as the static benchmark and a newly curated ClaimReview2025Q4 dynamic benchmark sourced from post-cut-off fact-checks.ClaimReview2025Q4 collects claims published in Q4 2025 relative to GPT-5.2’s August 2025 knowledge cut-off and retains 901 precise, self-contained claims after filtering.
4 Experiments
Experiments quantify contamination across static and dynamic MAFC benchmarks, investigate its sources, and measure its effects on evaluation. Contamination lowers the reliability of benchmark scores and rankings, while contamination-controlled evaluation identifies substantial remaining performance gaps.
- Experimental Setup: The evaluation compares AVeriTeC and ClaimReview2025Q4 using textual and semantic pairwise similarities between LLM-generated and human-written oracle evidence.Textual similarity uses METEOR10; semantic similarity uses Gemma-Emb-0.3B and Qwen3-Emb-0.6B embeddings.
- Results: ClaimReview2025Q4 consistently exhibits lower contamination than AVeriTeC across the evaluated metrics and models.The benchmark reduces contamination but does not eliminate it.
- Results: 17.09%–29.30% of dynamic-benchmark claims remain potentially contaminated when contamination is assessed across the three metrics.The intersection criterion identifies claims exceeding all three metric thresholds, showing that post-cut-off publication does not ensure uncontamination.
- Results: 35.18% (METEOR), 55.94% (Gemma-Emb-0.3B), and 50.94% (Qwen3-Emb-0.6B) are Qwen3.5-122B-A10B’s contaminated-claim ratios on ClaimReview2025Q4.This is the highest contaminated-claim ratio across all three metrics among the evaluated models.
- Results: 10.88 percentage points is Qwen3.5-122B-A10B’s Accuracy drop, while Gemini-3.0-Flash shows an 11.34 percentage-point Macro-F1 drop from contaminated to uncontaminated claims.Accuracy and Macro-F1 decrease for every model, with statistically significant drops for four evaluated models; contamination also distorts rankings.
- Results: 55.93% is DeepSeek-V3.2’s highest Macro-F1 on the unified contamination-controlled set, while every model remains below 56%.The evaluation set is label-imbalanced toward Refuted claims, motivating Macro-F1 as the primary metric.
5 Conclusion
The conclusion re-examines whether dynamic MAFC evaluation is inherently contamination-free and frames the study around contamination, its causes, evaluation effects, and contamination-controlled SOTA LLM performance. It positions the work as a foundation for trustworthy evaluation while calling for stronger novelty controls, latent-contamination detection, and broader multilingual research.
- Conclusion: The study re-examines the assumption that dynamic evaluation is inherently contamination-free for multimodal automated fact-checking.It investigates contamination extent, contamination sources, evaluation effects, and SOTA LLM performance under contamination-controlled settings.
- Conclusion: The work lays the foundation for trustworthy evaluation of MAFC systems in dynamic media environments.The conclusion presents trustworthy evaluation as the broader contribution of the study.
- Future directions: Future research should move beyond timestamp filtering through stronger claim-novelty controls, latent-contamination detection, and studies beyond English.The passage notes that similarity-based contamination estimates may be conservative and connects broader research to digital media literacy.
A Definitions of the Verdict Categories
ClaimReview2025Q4 adopts three FEVER-aligned verdict categories: Supported, Refuted, and Not Enough Evidence.
- Definitions of the Verdict Categories: The benchmark labels claims as Supported when evidence supports them, Refuted when evidence contradicts them, or Not Enough Evidence when available evidence cannot establish either verdict.The third category applies when relevant evidence cannot be found or when aspects of the claim remain unresolved.
B Example Prompts
This section presents example prompts for generating fact-checking articles and extracting claim-relevant evidence from oracle and LLM-generated articles.
- Evidence Generation: Evidence Generation uses a prompt to generate a fact-checking article based on the claim.The prompt instructs the system to write an article verifying the claim.
- Evidence Extraction: Evidence Extraction uses a prompt to extract evidence from both oracle and LLM-generated articles.The prompt targets evidence sentences directly related to the claim’s core factual content.
- Evidence Extraction: The extraction instructions prohibit inference, summarization, added information, claim restatements, and duplicated evidence.They require extracted content to appear in the generated text and address the claim’s main factual assertions.
C Impact of LLM Choice on Evidence Extraction
Contamination scores on ClaimReview2025Q4 remain consistent across the tested evidence-extraction models, indicating robustness to LLM choice in the contamination detection pipeline.
- C Impact of LLM Choice on Evidence Extraction: Table 9 reports average contamination scores across different extraction models on ClaimReview2025Q4.The comparison covers GPT-4o-Mini and GPT-5-Nano.
- C Impact of LLM Choice on Evidence Extraction: Contamination scores remain consistent across GPT-4o-Mini and GPT-5-Nano for evidence extraction.GPT-4o-Mini is the default extractor in the pipeline.
- C Impact of LLM Choice on Evidence Extraction: The contamination detection pipeline is robust to the choice of LLM extractor.This robustness assessment uses the average contamination scores reported in Table 9.
D Threshold Selection
The contamination threshold was selected through a labeled threshold-sweep analysis on a representative sample of ClaimReview2025Q4 claims.
- D Threshold Selection: 180 randomly sampled claims—20% of ClaimReview2025Q4—were used for the threshold-sweep analysis.Threshold values were evaluated over τ∈[0, 1].
- D Threshold Selection: Two postgraduate-level annotators with journalism and/or fact-checking expertise independently labeled each claim as contaminated or uncontaminated.Their disagreements were adjudicated before evaluation.
- D Threshold Selection: The adjudicated labels were treated as ground truth for evaluating semantic-similarity threshold values.Fig. 4a reports the resulting classification accuracy.
E Impact of Non-Contamination Factors
The study controls for topical and stylistic confounding by drawing contaminated and uncontaminated subsets from the same ClaimReview2025Q4 sources and time frame, then comparing their embedded claim characteristics.
- Controlling Non-Contamination Factors: Both subsets use the same ClaimReview2025Q4 sources and time frame, reducing potential topical or stylistic confounding.This design addresses whether performance differences reflect contamination rather than variation in claim origins or timing.
- Controlling Non-Contamination Factors: Claims are embedded with Qwen3-Emb-0.6B in 768 dimensions to capture topical and stylistic features, including language complexity and ambiguity.The embeddings provide an empirical basis for examining differences in claim characteristics across the two subsets.