Source-linked AI summary
Explainable Automated Fact-Checking: A Survey
Neema Kotonya, Francesca Toni
TL;DR
Explainable automated fact-checking remains underdeveloped despite progress in fact-checking accuracy and datasets. This survey synthesizes existing explanation methods, evaluates them against desirable properties, and concludes that current systems mainly explain individual predictions or pipeline components, motivating holistic, journalistic-informed approaches.
Problem
Despite progress in automated fact-checking models and datasets, relatively little work has addressed explanations for their predictions.
Method
The survey synthesizes explainable fact-checking methods and qualitatively evaluates them against eight desirable explanation properties, including impartiality.
Results
Most surveyed systems use extractive explanations for individual predictions, while existing methods explain only one fact-checking component.
Takeaways & Limitations
The survey recommends holistic, journalistic-informed approaches that generate more relevant, useful, and insightful fact-checking explanations.
Takeaways & Limitations
Current explainable fact-checking systems do not provide process-driven or model explanations spanning multiple or all fact-checking subtasks.
Abstract
from arXiv · showhide
A number of exciting advances have been made in automated fact-checking thanks to increasingly larger datasets and more powerful systems, leading to improvements in the complexity of claims which can be accurately fact-checked. However, despite these advances, there are still desirable functionalities missing from the fact-checking pipeline. In this survey, we focus on the explanation functionality -- that is fact-checking systems providing reasons for their predictions. We summarize existing methods for explaining the predictions of fact-checking systems and we explore trends in this topic. Further, we consider what makes for good explanations in this specific domain through a comparative analysis of existing fact-checking explanations against some desirable properties. Finally, we propose further research directions for generating fact-checking explanations, and describe how these may lead to improvements in the research area.
1 Introduction
Automated fact-checking has advanced substantially through stronger models and datasets, but explanation functionality remains comparatively underdeveloped. This survey examines explainable automated fact-checking, evaluates desirable explanation properties, and proposes future research directions.
- The survey focuses on explanations for automated fact-checking predictions, including posthoc explanations and methods integrated directly into fact-checking models.
- Although fact-checking models and datasets have improved, substantially less work has addressed obtaining explanations for their outputs.
- Justifying claim-verification judgments is important because such justification is a central part of manual journalistic fact-checking.
- The survey covers automated fact-checking progress, computational and journalist-provided explanations, explanation quality, and future research directions.
2 Automated Fact-Checking: State of the Art
Automated fact-checking has advanced through larger, richer datasets and increasingly complex systems, but limited training data, dataset artifacts, and reduced transparency remain important challenges. Current systems combine neural, non-neural, and heuristic techniques across distinct fact-checking subtasks, increasing the need for explanations.
- Datasets: Larger fact-checking corpora now include evidence, contextual metadata, multiple domains, and sometimes natural-language explanations, extending beyond early small claim datasets.LIAR provided 12.8K claims but lacked verification evidence; later resources include real-world claims across domains such as politics, science, and health.
- Datasets: Dataset annotation choices determine task formulations and produce substantially different veracity-label schemes, including binary, graded, three-way, six-point, and multidimensional factuality judgments.Examples include LIAR’s six-point system, FEVER’s three-way classification, and FactBank’s polarity and certainty dimensions.
- Limitations: Synthetic or human-crafted datasets can encode exploitable idiosyncrasies, causing model bias and weak performance on new data.Schuster et al. examined FEVER with a symmetric dataset to expose artifacts that enabled high-accuracy fact-checking models.
- Systems: Shared-task and standalone systems represent the main development contexts, with the first FEVER challenge producing 24 submissions and a top accuracy above 60%.The second FEVER challenge additionally invited adversarial claims through a Build it, Break it, Fix it format.
- Systems: Fact-checking architectures combine deep neural networks, non-neural models, and heuristics, with techniques varying by subtask such as retrieval, evidence selection, and claim verification.Heuristics are typically used for document retrieval, while neural models such as ESIM and decomposable attention support evidence selection.
- Limitations: Increasing reliance on complex neural architectures and joint subtask training creates a trade-off with transparency and strengthens the case for explanations across the fact-checking pipeline.The survey identifies multiple fact-checking subtasks as relevant targets for explanation, not only final veracity prediction.
3 Explanations for fact-checking
The survey distinguishes human-readable explainability from model interpretability and reviews how automated fact-checking systems generate explanations. Most surveyed systems use extractive explanations, while rule-based, knowledge-graph, and natural-language generation approaches offer alternatives with different trade-offs.
- Definitions: Explainability delivers a rationale for a model’s decision, whereas interpretability provides mechanisms for analyzing or visualizing its decision-making.The survey focuses primarily on natural-language, human-readable explanations, while treating interpretations as necessary but not sufficient for human comprehension.
- Survey trends: Most surveyed automated fact-checking systems produce extractive explanations, despite progress in fact-checking accuracy and the existence of alternative task formulations.Extractive explanations select content from evidence or input materials rather than generating fully novel explanations.
- Attention-based explanations: Attention-based DNNs explain predictions by highlighting salient words or excerpts, sometimes incorporating article comments, propagation patterns, user characteristics, or credibility scores.These methods include attention and co-attention architectures for extracting evidence-related tokens and contextualizing them with user or source information.
- Rule-based explanations: Horn-rule and knowledge-graph systems mine more comprehensive explanations than attention-based systems, but they have different disadvantages.The survey contrasts the comprehensiveness of rule-based explanations with the advantages and limitations of attention-based neural approaches.
- Natural-language generation: Natural-language explanation generation has been formulated as text summarization, with separately trained post hoc explainers slightly outperforming jointly trained predictive-explanation models.Atanasova et al. compare separate and joint training objectives for generating fact-checking explanations.
4 Comparative Analysis of Explainable Systems
The survey qualitatively evaluates explainable fact-checking systems against eight desirable explanation properties, combining seven prior usability requirements with a journalism-informed property. Systems generally satisfy context-fullness, while actionability, causality, impartiality, and chronology remain limited or absent.
- 4 Comparative Analysis of Explainable Systems: All generated explanations are judged context-full because they are presented with larger inputs, including claims and sometimes user context.Context-fullness means the explanation’s surrounding context can be fully understood.
- 4 Comparative Analysis of Explainable Systems: Only CredEye and XFake satisfy interactiveness, through user agreement ratings and visualizations of multiple fact-checking assessments, respectively.CredEye allows binary yes-or-no feedback, while XFake visualizes assessments such as linguistic analysis.
- 4 Comparative Analysis of Explainable Systems: None of the examined systems explicitly addresses unbiasedness or impartiality, although rule-based approaches are viewed as a step toward it.Rule-based systems output triples rather than text, which may reduce exposure to biased language in explanations.
- 4 Comparative Analysis of Explainable Systems: Rules, tokens, and word clouds are considered parsimonious explanations, while Atanasova et al. evaluate explanations for non-redundancy.Parsimony concerns communicating the necessary information with minimal redundant text.
- 4 Comparative Analysis of Explainable Systems: No examined system considers actionability, causality, or chronology, as summarized in Table 4.These properties are explicitly absent from the comparative analysis.
5 Future Directions
The survey identifies limitations in current explainable fact-checking systems and proposes possible approaches to overcome them. Key directions concern unverifiable claims, process- and model-level explanations, and explanation evaluation.
- 5 Future Directions: Current explainable fact-checking approaches have limitations that motivate future research directions for overcoming them.The survey frames these shortcomings as additional limitations beyond those emerging from Table 4.
- Unverifiable Claims: Fact-checking must handle claims for which insufficient evidence exists to verify or refute them.This issue affects both manual and computational fact-checking, motivating additional outcomes such as NOT-ENOUGH-INFO in FEVER.
- Predictions vs. Models: Existing systems emphasize outcome-driven explanations and lack process- or model-level explanations spanning multiple fact-checking subtasks.Such explanations could clarify how earlier subtask decisions affect later outputs and why decisions are made.
- Evaluating explanations: Explanation quality requires established evaluation metrics alongside the computational and human evaluations currently used by surveyed systems.Several surveyed approaches employ human annotators to assess generated explanation quality.
6 Conclusion
The survey concludes that explainable machine learning for fact-checking is promising despite the domain’s challenges, while current methods and task formulations remain limited in scope.
- Explainable machine learning for fact-checking shows considerable promise despite the problem’s particularly challenging nature.
- Existing methods explain only one fact-checking component, focusing on relation prediction or entailment.
- Current systems explain predictions rather than addressing broader limitations of the fact-checking pipeline.