Source-linked AI summary
Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Method
Xuan Zhang, Wei Gao
TL;DR
The paper studies whether LLMs can verify news claims despite limited prior exploration of this task. It introduces HiSS, which decomposes claims and progressively verifies subclaims, and reports stronger performance than supervised and few-shot baselines on two datasets.
Problem
The paper addresses the limited understanding of LLM performance for news claim verification, a task outside the core of LLM pre-training.
Method
HiSS prompts LLMs to decompose claims into subclaims and verify each through progressively generated questions with search-based evidence when needed.
Results
4.95% average improvement in macro-average F1 over traditionally strong fully supervised models is reported for HiSS on two public datasets.
Takeaways & Limitations
HiSS outperforms fully supervised state-of-the-art models and strong few-shot ICL baselines while producing more fine-grained and readable explanations.
Takeaways & Limitations
The method relies on textual information and high-capability backbone LLMs, creating scope, computational, access, control, and API-cost constraints.
Abstract
from arXiv · showhide
While large pre-trained language models (LLMs) have shown their impressive capabilities in various NLP tasks, they are still under-explored in the misinformation domain. In this paper, we examine LLMs with in-context learning (ICL) for news claim verification, and find that only with 4-shot demonstration examples, the performance of several prompting methods can be comparable with previous supervised models. To further boost performance, we introduce a Hierarchical Step-by-Step (HiSS) prompting method which directs LLMs to separate a claim into several subclaims and then verify each of them via multiple questions-answering steps progressively. Experiment results on two public misinformation datasets show that HiSS prompting outperforms state-of-the-art fully-supervised approach and strong few-shot ICL-enabled baselines.
1 Introduction
The paper examines LLMs for news claim verification and identifies weaknesses in vanilla reasoning, then proposes HiSS to decompose claims and verify them more thoroughly. Across experiments, few-shot LLM prompting performs strongly, while HiSS improves verification and explanation quality.
- Misinformation claims create confusion and wrong beliefs, while manual fact-checking struggles to match the pace of misinformation production.
- LLM fact verification remains under-explored because factual verification is not central to LLM pre-training.
- Vanilla CoT can miss necessary claim details and hallucinate unsupported facts, producing worse performance than standard prompting.
- HiSS decomposes complex claims into subclaims and verifies them with search-supported reasoning to reduce omissions and hallucinations.
- With four-shot demonstrations, LLMs can outperform most supervised methods for news claim verification.
- 4.95% average improvement in macro-average F1 over strong fully supervised models establishes HiSS as a new state-of-the-art few-shot approach.
- HiSS-prompted explanations are more fine-grained and easier to follow than those from previous methods under automatic and human evaluation.
2 Related Work
Prior explainable fake-news methods generate explanations from evidence, summaries, or encoded knowledge, while LLM-based reasoning for misinformation tasks remains limited. The paper positions HiSS as reasoning from raw evidence to form verification thoughts.
- Earlier explainable detection methods use knowledge bases, attention, or extracted and abstractive summaries to produce human-comprehensible explanations.
- Summary-based approaches predominantly explain decisions by condensing fact-checking articles rather than reasoning directly from raw evidence.
- Prior language-model work has used internal model knowledge, retrieval augmentation, and few-shot capabilities for fact verification.
- LLM reasoning combined with external knowledge has helped reasoning-intensive NLP tasks, but its use in misinformation remains limited.
3 Our HiSS Prompting Method
HiSS addresses omitted reasoning and hallucinated facts by decomposing claims into explicit subclaims, generating progressive verification questions, and selectively obtaining external evidence. The process ends with a final prediction based on the accumulated verification steps.
- 3 Our HiSS Prompting Method: HiSS targets two LLM problems in claim verification: omission of necessary thoughts and fact hallucination.
- 3.1 Research Questions: The first research question asks how to prevent CoT from overlooking crucial explicit and implicit points in a claim.
- 3.1 Research Questions: HiSS addresses hallucination by supplying contextual external knowledge when the model lacks necessary information.
- 3.1 Research Questions: The method uses three processes: claim decomposition, subclaim step-by-step verification, and final prediction.
- 3.2 Claim Decomposition: At Level 1, few-shot demonstrations guide the LLM to split each claim into subclaims covering all explicitly check-worthy points.
- 3.3 Subclaim-level Step-by-Step Verification: At Level 2, the LLM generates progressive probing questions for each subclaim, adapting later questions to earlier answers and acquired context.
- 3.3 Subclaim-level Step-by-Step Verification: The LLM assesses confidence for each question and invokes Google Search when it is not confident, then feeds search results into answer generation.
4 Experiments and Results
Experiments evaluate few-shot prompting for claim verification on RAWFC and LIAR, comparing LLM-based methods with supervised and few-shot baselines. HiSS achieves stronger verification performance, with gains linked to claim decomposition, step-by-step verification, search, and more detailed explanations.
- Experimental Setup: The experiments use RAWFC and LIAR, real-world news-claim datasets with three-class and six-class label schemes, respectively.RAWFC labels are based on Snopes articles, while LIAR labels are based on PolitiFact articles.
- Experimental Setup: The evaluation reports macro-average precision, recall, and F1, while comparing HiSS with supervised and few-shot prompting baselines.The baselines include standard prompting, vanilla CoT, and search-augmented CoT.
- Claim Veracity Classification: HiSS exceeds the previous supervised state of the art by 1.9% F1 on RAWFC and 8% F1 on LIAR.The comparison is against CofCED, using the experimental results summarized in Table 2.
- Error Analysis: HiSS addresses hallucination and thought omission by decomposing claims, checking subclaims step-by-step, and acquiring evidence through search when necessary.Vanilla CoT can omit noteworthy claim details and generate unreliable facts, while HiSS is designed to cover explicit and implicit points.
- Claim Veracity Classification: HiSS surpasses standard prompting, vanilla CoT, and ReAct by 7.95%, 11.65%, and 5.3% in average F1, respectively.The results show that few-shot ICL performance varies substantially with the prompting method.
- Ablation Study: Removing claim decomposition lowers performance by 1.5%, while replacing subclaim step-by-step verification with direct verification causes a 2.9% drop.These ablations support the contributions of both claim decomposition and progressive subclaim verification.
5 Conclusion and Future Work
The paper studies LLM prompting for news claim verification and introduces HiSS to perform fine-grained verification. It reports improved performance and explanation quality, while proposing conversational fact-checking with human involvement as future work.
- HiSS prompts LLMs to verify news claims through fine-grained steps designed to mitigate omitted thoughts and fact hallucination.
- HiSS improves LLM performance over fully supervised state-of-the-art models and strong few-shot ICL counterparts on two public datasets.
- HiSS explanations show superior coverage and readability.
- Future work will develop a user-friendly conversational fact-checking model incorporating human fact-checkers in the loop.
6 Limitations
The study is limited to textual news-claim verification and depends on capable, costly backbone LLMs and searchable external information. Its scope excludes other misinformation formats and multimodal evidence.
- The experiments use a unimodal textual approach and do not consider assistance from images or videos.
- The study focuses on news claims, which represent only a subset of broader misinformation formats such as rumors, fake news articles, and spam.
- The prompting method relies on capable backbone LLMs, creating substantial computational costs and challenges from API access and limited model control.
- Search-based verification assumes pertinent information is accessible online, while unavailable or incomplete knowledge can require capabilities beyond current technologies.
A Case Study
A LIAR case study contrasts vanilla CoT with HiSS on a claim containing percentage information. CoT reaches the correct prediction but hallucinates facts and omits a necessary verification step, whereas HiSS examines each claim component with search support.
- CoT generates inaccurate percentages, 57 and 1 percent, and omits verification of the claim about 1 percentage to food and agriculture.
- Although CoT reaches the correct final prediction, it still omits necessary thoughts by failing to verify the percentage claim.
- HiSS examines each claim component and uses a search engine to gather comprehensive information.
B Demonstration Examples
The appendix presents demonstration examples for standard prompting, vanilla CoT, and HiSS, alongside a case-study table highlighting labels, predictions, hallucinations, and repetition.
- The appendix includes demonstration examples for standard prompting, vanilla CoT, and HiSS.
- The case-study table marks labels and predictions in pink, fact-hallucination sentences in purple, and repeated sentences in yellow.
- The HiSS demonstration examples highlight the CoT process in green.