PaperlayerView exact revision on arXiv

Source-linked AI summary

ClaimIQ at CheckThat! 2025: Comparing Prompted and Fine-Tuned Language Models for Verifying Numerical Claims

Anirban Saha Anik, Md Fahimul Kabir Chowdhury, Andrew Wyckoff, Sagnik Ray Choudhury

arXiv:2509.11492v1cs.CLcs.AI

TL;DR

Numerical and temporal claims are difficult to verify because they require precise quantitative reasoning over retrieved evidence. The paper compares zero-shot prompting and LoRA fine-tuning with alternative evidence-selection strategies; LoRA-fine-tuned LLaMA performs best on English validation, while its test-set drop highlights generalization challenges.

  • Problem

    Numerical and temporal claims require precise reasoning over quantities, dates, and trends, while LLM effectiveness in structured verification over retrieved evidence remains an open problem.

    Source: §1 · p. 1

  • Method

    The paper compares zero-shot prompting and LoRA fine-tuning for numerical claim verification using full-document and BM25- or MiniLM-selected sentence evidence.

    Source: §1 · p. 2, §4.4 · p. 5, §7 · p. 8

  • Results

    0.945 macro-F1: LLaMA fine-tuned with LoRA on full-document evidence performed best on validation, but performance dropped on the test set.

    Source: §6.1 · p. 6, §6.3 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    The findings underscore the importance of model architecture and evidence quality for robust fact verification, with future work proposed on domain adaptation, reasoning, and evidence selection.

    Source: §6.3 · p. 8

    The model’s test-set performance, particularly for True and Conflicting, was substantially below validation performance, indicating a generalization challenge.

    Source: §6.3 · p. 7

Abstract

from arXiv · show

This paper presents our system for Task 3 of the CLEF 2025 CheckThat! Lab, which focuses on verifying numerical and temporal claims using retrieved evidence. We explore two complementary approaches: zero-shot prompting with instruction-tuned large language models (LLMs) and supervised fine-tuning using parameter-efficient LoRA. To enhance evidence quality, we investigate several selection strategies, including full-document input and top-k sentence filtering using BM25 and MiniLM. Our best-performing model LLaMA fine-tuned with LoRA achieves strong performance on the English validation set. However, a notable drop in the test set highlights a generalization challenge. These findings underscore the importance of evidence granularity and model adaptation for robust numerical fact verification.

1. Introduction

Numerical and temporal claims are challenging to verify because they demand precise quantitative reasoning, often over noisy or contradictory retrieved evidence. This paper compares prompted and fine-tuned LLMs with different evidence-selection strategies for numerical claim verification.

1. Introduction

  • Verifying numerical or temporal claims requires precise reasoning over quantities, dates, and trends, not just factual knowledge.Such claims can be persuasive despite appearing deceptively simple.Source: §1 · p. 1
  • CLEF 2025 Task 3 classifies numerical and temporal claims as True, False, or Conflicting from a small set of retrieved evidence.That evidence may be noisy, partly relevant, or contradictory, and claims can require implicit or contextual numerical reasoning.Source: §1 · p. 1
  • LLMs’ effectiveness in structured fact verification over multiple retrieved passages remains an open research problem.The paper also identifies factual correctness and computational efficiency as considerations for adapting LLMs.Source: §1 · p. 1
  • The study compares zero-shot prompting and LoRA fine-tuning alongside full-document input and top-k sentence selection with BM25 or MiniLM.The stated aim is to examine the balance between generalization and supervision and investigate adaptation for numerical fact-checking.Source: §1 · p. 2

2. Related Work

Prior work combines retrieval and language models for grounded generation or claim verification, while this paper brings sentence-level retrieval together with fine-tuned LLaMA and focused supervision.

2. Related Work

  • RAG combines document retrieval with LLMs for grounded outputs; RARG uses scientific literature and document-level supervision, while RAFTS generates supporting and refuting responses from retrieved passages.The cited work describes RARG’s reinforcement learning and RAFTS’s emphasis on interpretability and parameter-efficient models.Source: §2 · p. 2
  • FactGenius uses DBPedia and fuzzy matching for zero-shot fact-checking, ClaimMatch uses NLI and paraphrase detection, and MiniCheck verifies sentences with synthetic training data.The passage reports MiniCheck approaching GPT-4 performance and notes competitive results from appropriately fine-tuned compact models.Source: §2 · p. 2
  • Full-context RAG frameworks have also been used for LLM-based claim extraction and fact-checking [20].Source: §2 · p. 2
  • This approach combines BM25 and MiniLM sentence-level retrieval with fine-tuned LLaMA and multilingual claim-evidence alignment, emphasizing simpler architectures and focused supervision.The authors state that strong performance can be achieved without decomposition-heavy pipelines.Source: §2 · p. 2

3. Task Description

Task 3 evaluates whether systems can verify numerical or temporal claims from retrieved evidence, including ambiguous and conflicting cases. The paper uses the English portion of a dataset sourced from fact-checking reports.

3.1. Task Overview

  • Task 3 assigns True, False, or Conflicting labels to numerical or temporal claims using top-100 BM25-ranked evidence passages.Conflicting covers ambiguous, partly supportive, or contradictory evidence; systems may use reranking, filtering, and generation.Source: §3.1 · p. 2

3.2. Dataset Summary

  • The dataset is drawn from Google Fact Check Explorer reports, filtered for numerical or temporal expressions, and includes ranked evidence retrieved with BM25 and claim decomposition.The dataset overview is for the English subset; although the dataset supports multiple languages, these experiments use English only.Source: §3.2 · p. 3, Table 1 · p. 3

4. Methodology

The method frames numerical claim verification as three-way classification, then compares prompted and fine-tuned models across alternative evidence inputs. Evidence selection ranges from full documents to sentence-level retrieval, with model performance assessed using macro-F1 and class-wise F1.

4.1. Problem Formulation

  • The task is modeled as a function that maps a claim and retrieved evidence set to one of three labels, with evidence varying by experimental configuration.The evidence may be a full BM25 document, top-k relevant sentences, or an LLM-generated summary; the function may be a prompted generative model or fine-tuned classifier.Source: §4.1 · p. 3

4.2. Prompting with LLaMA

  • Zero-shot LLaMA receives an instruction prompt containing the claim and selected evidence, and its generated response is post-processed into one of the three labels.Prompting uses no gradient updates and permits testing evidence setups without task-specific fine-tuning; ambiguous outputs are mapped to the nearest predefined label, typically Conflicting.Source: §4.2 · pp. 3–4

4.3. Evidence Selection Strategies

  • Evidence selection addresses irrelevant or redundant content in retrieved documents by testing full documents, top-3 BM25 sentences, and top-3 MiniLM sentences.BM25 selects lexical matches, while MiniLM uses cosine similarity to capture semantic matches beyond surface overlap; each evidence type is paired with prompted and fine-tuned models.Source: §4.3 · p. 4

4.4. Model Architectures

  • The fine-tuning setup includes RoBERTa and LLaMA with LoRA, alongside zero-shot prompted LLaMA; each model classifies a claim with selected evidence into True, False, or Conflicting.LoRA fine-tuning uses claim-evidence prompts and label responses, with adaptation on LLaMA-3.1-8B’s query, key, value, and output projections.Source: §4.4 · pp. 4–5

4.5. Evaluation Metrics

  • Evaluation uses macro-averaged F1 across the three labels and reports class-wise F1 to examine behavior under class imbalance, especially for Conflicting cases.Results are computed on the official English validation and test splits using a consistent preprocessing and evaluation pipeline.Source: §4.5 · p. 5

5. Experiments

The experiments evaluate numerical-claim verification on the English CLEF 2025 Task 3 dataset using prompted and fine-tuned models with different evidence-selection strategies. The setup reports macro-F1 and class-wise behavior, while the training and inference protocols cover zero-shot LLaMA, fine-tuned RoBERTa, and LoRA-adapted LLaMA.

5.1. Experimental Setup

  • The English dataset contains 15,514 claims labeled True, False, or Conflicting, each paired with up to 100 BM25-retrieved evidence documents.The supervised-learning split uses stratified sampling, with 90% for training and 10% for validation.Source: §5.1 · p. 5
  • Macro-averaged F1 is the official metric, supplemented by class-wise F1, confusion matrices, and sample-error analysis.The class-wise analysis targets behavior under class imbalance and difficulty predicting Conflicting cases.Source: §5.1 · p. 5

5.2. Training and Inference Setup

  • Prompted LLaMA is evaluated zero-shot with claim-and-evidence instructions, while RoBERTa is fine-tuned on claim-evidence sequence pairs.The prompting uses temperature 0.3 and top-p 0.9; RoBERTa input is truncated to 512 tokens and trained for three epochs with early stopping on validation macro-F1.Source: §5.2 · pp. 5–6
  • LLaMA fine-tuning uses LoRA on query, key, value, and output projections, with prompt-response training and memory-saving settings.LoRA uses rank 8, scaling factor 16, and dropout 0.05; training uses FP16, gradient checkpointing, and three epochs.Source: §5.2 · p. 6

6. Results

Results vary by evidence configuration and evaluation split: fine-tuned LLaMA with LoRA leads validation, while its best test performance comes from sentence-level evidence. Discussion highlights a validation–test gap and persistent challenges with True and Conflicting claims.

6.1. Validation Results

  • Zero-shot LLaMA reached 0.609 macro-F1 with full-document input, but struggled significantly with the Conflicting class.Fine-tuned models consistently outperformed prompted ones; RoBERTa achieved its best Conflicting-class F1, 0.510, with MiniLM evidence.Source: §6.1 · p. 6
  • 0.945 macro-F1 was achieved by fine-tuned LLaMA with LoRA using full-document evidence on validation, with balanced performance across all three classes.Top-3 MiniLM also provided strong results, improving precision on harder examples while reducing irrelevant context.Source: §6.1 · p. 6

6.2. Test Set Performance

  • On test, prompted LLaMA scored 0.40 macro-F1 with full documents and 0.41 with Top-3 BM25, while fine-tuned RoBERTa scored 0.35 and 0.34 with those sentence filters.The results indicate zero-shot prompting generalized better than fine-tuned RoBERTa.Source: §6.2 · p. 7
  • 0.43 macro-F1 was the highest English test-set score, achieved by fine-tuned LLaMA with LoRA using either Top-3 BM25 or Top-3 MiniLM evidence.Full-document evidence reached 0.42 macro-F1, with Conflicting-class F1 of 0.32.Source: §6.2 · p. 7, Table 3 · p. 7

6.3. Discussion

  • Test-set F1 was higher for False claims across configurations, while True and Conflicting claims remained challenging; Conflicting F1 was higher with full-document fine-tuned LLaMA.The discussion notes a substantial validation–test performance gap, particularly for True and Conflicting categories, and suggests overfitting or shifts in evidence structure and language style as possible explanations.Source: §6.2 · p. 7, §6.3 · p. 7, Table 4 · p. 7
  • Preliminary error review linked failures to numerical reasoning challenges, ambiguous or contradictory evidence, and missing key supporting facts.These patterns underscore the complexity of verifying numerical claims with noisy or incomplete context.Source: §6.3 · pp. 7–8
  • The discussion concludes that robust verification depends on model architecture and evidence quality, and proposes domain-adaptive training, reasoning-aware approaches, and improved evidence selection as future work.The authors describe the test-set drop as a generalization challenge, likely related to domain shift and nuanced real-world evidence.Source: §6.3 · p. 8

7. Conclusion

The paper reports its Task 3 approach and finds that LLaMA fine-tuned with LoRA on full-document evidence performed best on validation, while the test-set drop highlights generalization challenges.

7. Conclusion

  • The Task 3 approach combined zero-shot prompting and parameter-efficient fine-tuning with multiple evidence-selection strategies, including BM25 and MiniLM sentence filtering.Source: §7 · p. 8
  • LLaMA fine-tuned with LoRA on full-document evidence achieved the best validation performance, but test-set performance dropped and highlighted generalization challenges.Sentence-level filtering improved performance for ambiguous claims, especially in the Conflicting class.Source: §7 · p. 8
  • Future work will target robustness through domain-adaptive training, improved retrieval filtering, and reasoning-aware modeling.The authors describe LLMs combined with structured evidence processing as a promising foundation for scalable, accurate fact verification.Source: §7 · p. 8

Declaration on Generative AI

The authors used ChatGPT-4o and Grammarly for language revision, not for analysis, figures, or experiments, and reviewed and edited the content afterward.

Declaration on Generative AI

Loading 2509.11492v1…