Source-linked AI summary
Automated Fact-Checking for Assisting Human Fact-Checkers
Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, Giovanni Da San Martino
TL;DR
The spread of misleading claims and the scale of available information make fact-checking increasingly difficult. This paper surveys intelligent technologies that support human fact-checkers across claim selection, prior-check retrieval, evidence retrieval, and verification, finding only partial alignment between fact-checkers’ needs and available technology.
Problem
The spread of misinformation and the volume of available information make it difficult for fact-checkers to identify, investigate, and verify claims efficiently.
Method
The paper surveys intelligent technologies supporting human fact-checkers in selecting claims, finding previously checked claims, retrieving evidence, and verifying claims.
Results
The analysis finds a partial disconnection between what fact-checkers want and what technology currently offers.
Takeaways & Limitations
Fact-checking tools are needed to find claims and relevant evidence while demonstrating practical value in fact-checkers’ day-to-day work.
Takeaways & Limitations
General verification of arbitrary claims still requires real-world understanding beyond current AI, so most methods assist human fact-checkers rather than replace their assessment.
Abstract
from arXiv · showhide
The reporting and the analysis of current events around the globe has expanded from professional, editor-lead journalism all the way to citizen journalism. Nowadays, politicians and other key players enjoy direct access to their audiences through social media, bypassing the filters of official cables or traditional media. However, the multiple advantages of free speech and direct communication are dimmed by the misuse of media to spread inaccurate or misleading claims. These phenomena have led to the modern incarnation of the fact-checker -- a professional whose main aim is to examine claims using available evidence and to assess their veracity. As in other text forensics tasks, the amount of information available makes the work of the fact-checker more difficult. With this in mind, starting from the perspective of the professional fact-checker, we survey the available intelligent technologies that can support the human expert in the different steps of her fact-checking endeavor. These include identifying claims worth fact-checking, detecting relevant previously fact-checked claims, retrieving relevant evidence to fact-check a claim, and actually verifying a claim. In each case, we pay attention to the challenges in future work and the potential impact on real-world fact-checking.
1 Introduction
The survey examines how intelligent technologies can support human fact-checkers amid widespread misinformation and limited trust in fully automated systems. It organizes this support around the fact-checking process, from selecting claims to retrieving evidence and reaching verdicts.
- Misinformation and disinformation on the web and social media have become urgent social and political issues, prompting manual fact-checking initiatives.
- Automated fact-checking research includes systems, datasets, and evaluation campaigns, but credibility concerns and limited human-in-the-loop work remain.
- The survey addresses four fact-checking steps: extracting claims, constructing questions, obtaining evidence, and reaching verdicts.
- The survey limits its scope to automated fact-checking of text, although misleading and manipulated images and videos are also widespread.
- Its distinctive focus is comparing fact-checkers’ desiderata with research attempts intended to meet them.
2 What Fact-Checkers Want
Fact-checkers need practical tools that help prioritize claims, find prior checks, and access evidence, while fully automated verification remains constrained by credibility and complexity. The section emphasizes timely, scalable assistance rather than replacing human judgment.
- AI practitioners face a twin challenge: developing practical tools for fact-checkers and demonstrating their value in day-to-day work.
- 2.1 Finding Claims Worth Fact-Checking: Fact-checkers balance potential harm, verification effort, non-partisanship, and data availability when choosing claims to check.
- 2.1 Finding Claims Worth Fact-Checking: Check-worthiness is harder to estimate than virality, so historical checked and unchecked claims are not reliable indicators for similar claims.
- Rapid detection of previously fact-checked claims can support timely action because viral claims may spread rapidly before manual fact-checking is completed.
- 2.3 Evidence Retrieval: Automatic transcription, summarization, translation, and search can expose evidence in large, audiovisual, or unfamiliar-language sources.
- 2.4 Automated Verification: Fully automated verification is difficult because claims may be partially correct or misleading without context, and incorrect fact-checks can damage organizational reputation.
3 What Technology Currently Offers
Current technology supports fact-checkers by prioritizing claims, finding prior checks, retrieving evidence, and assisting verification, while remaining limited for complex claims and fully automatic decisions.
- 3.1 Finding Claims Worth Fact-Checking: Fact-checking systems identify claims worth checking by ranking them with check-worthiness scores that help prioritize or filter claims.Fact-checkers can provide feedback to tune how well scores reflect actual check-worthiness.
- 3.1 Finding Claims Worth Fact-Checking: Full Fact used a fine-tuned BERT model to classify political claims, enabling rapid identification of check-worthy claims during an election.The model distinguished numerical claims, predictions, and personal beliefs, among other categories.
- 3.2 Detecting Previously Fact-Checked Claims: Previously fact-checked claims can be detected by matching new tweets or debate claims against Snopes and PolitiFact articles using BERT and BM25.The approach formulates prior-check detection as a learning-to-rank task.
- 3.3 Evidence Retrieval: Evidence retrieval ranks relevant documents or extracts supporting snippets from closed collections, with methods combining document- and sentence-level similarity.INEX retrieved evidence snippets from 50k books, where entity matching was among the most important features.
- 3.3 Evidence Retrieval: Evidence systems can also generate background briefs by retrieving passage, entity, and question-answering information decomposed from a claim.This extends retrieval beyond direct support or refutation snippets to contextual information for fact-checkers.
- 3.4 Automated Verification: Automatic verification shows promising results for explicit numerical or simple claims when reliable evidence is available, but general verification remains difficult.Success depends on reliable, consistently formatted data, while complex claims may require expert interviews, contextual understanding, and multiple sources.
- 3.5 Some Real-World Systems: Real-world systems increasingly involve fact-checker feedback and explainable decisions, positioning automation as assistance rather than replacement.Examples include user and fact-checker feedback for misinformation detection and systems designed to support several fact-checking steps.
4 Lessons Learned
The survey finds a partial disconnection between fact-checkers’ needs and available technology. It identifies support opportunities across the workflow but emphasizes credibility, scalability, collaboration, and incomplete human-in-the-loop development.
- 4 Lessons Learned: The analysis finds a partial disconnection between what fact-checkers want and what technology offers.The paper presents this as the main lesson from its analysis.
- 4 Lessons Learned: Automated systems raise credibility concerns when they lack supporting evidence, while manual fact-checking faces scalability limits.These limitations affect both automated and manual processes.
- 4 Lessons Learned: Automation can help identify check-worthy claims, locate prior checks, retrieve and summarize evidence, translate content, and detect rapidly spreading claims.The listed support functions span prioritization, retrieval, multilingual access, summarization, and monitoring.
- 4 Lessons Learned: Researchers and practitioners have not collaborated sufficiently to define tasks and develop datasets for automated fact-checking.The paper identifies human-in-the-loop fact-checking as an ideal setting that remains incompletely explored.
5 Challenges and Future Forecasting
The survey identifies challenges spanning multilingual resources, ambiguity, bias, context, multimodality, collaboration, usability, interpretability, integration, and real-time deployment. It forecasts research directions aimed at making automated fact-checking more useful, accountable, scalable, and adoptable by human fact-checkers.
- Major Challenges: Multilingual verification resources remain uneven, especially because low-resource languages lack the resources available in English.Claims can spread across regions and languages, requiring alignment and coordination of verification resources.
- Major Challenges: Existing systems are biased by datasets curated by small groups and annotations from non-experts.These systems can reflect developers’ perceptions of factuality and the way annotation tasks were specified.
- Major Challenges: Current automated fact-checking makes limited use of contextual information such as comments, linked sources, and social-network data.The survey identifies such information as useful signals for enriching current models.
- Major Challenges: Single-modality approaches are vulnerable when misinformation combines text, images, speech, video, temporal information, user profiles, and network structure.The survey points to multimodal approaches when evidence can be gathered from multiple modalities.
- Future Forecasting: The survey forecasts closer collaboration, integrated open-source resources, usable interfaces, explainable models, and efficient real-time systems.These directions include reducing technical knowledge and cognitive load, supporting ethical accountability, and embedding systems within social networks and technology companies.
6 Conclusion
The paper surveys intelligent technologies that can support human experts throughout multiple steps of manual fact-checking and considers future challenges and real-world impact. It concludes that research priorities only partially overlap with fact-checkers’ needs and proposes directions for future work.
- Conclusion: The survey covers technologies for identifying claims, finding previously fact-checked claims, retrieving evidence, and verifying claims.It examines challenges in future work and potential impact on real-world fact-checking.
- Conclusion: Fact-checkers’ needs and research priorities currently overlap only partially.The paper also discusses lessons learned, major challenges, and research directions expected to emerge in the near future.