Source-linked AI summary
Fully Automated Fact Checking Using External Sources
Georgi Karadzhov, Preslav Nakov, Lluis Marquez, Alberto Barron-Cedeno, Ivan Koychev
TL;DR
As deceptive information proliferates online, automatic fact checking seeks to distinguish false claims from true ones. This paper retrieves Web evidence and combines LSTM encodings with task-specific embeddings and source reliability information; it reports strong performance across rumor detection and cQA answer factuality.
Problem
The proliferation of deceptive information online has increased the need for automatic methods to validate information from the Internet.
Source: §1 · p. 1Method
The framework retrieves potentially relevant Web text for a claim and combines LSTM encodings, task-specific embeddings, and source reliability information to classify its veracity.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The method performs well using only search-engine snippets and is presented as lightweight and potentially applicable across different fact-checking domains and task formulations.
Source: §6 · p. 8Direct comparison with Ma et al. (2016) is not possible because the datasets and class distributions differ, and their task does not take claims as input.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
Given the constantly growing proliferation of false claims online in recent years, there has been also a growing research interest in automatically distinguishing false rumors from factually true claims. Here, we propose a general-purpose framework for fully-automatic fact checking using external sources, tapping the potential of the entire Web as a knowledge source to confirm or reject a claim. Our framework uses a deep neural network with LSTM text encoding to combine semantic kernels with task-specific embeddings that encode a claim together with pieces of potentially-relevant text fragments from the Web, taking the source reliability into account. The evaluation results show good performance on two different tasks and datasets: (i) rumor detection and (ii) fact checking of the answers to a question in community question answering forums.
1 Introduction
The spread of deceptive information online has made automatic fact checking important across applications such as rumor detection and information verification. This paper proposes a fully automatic Web-based framework and evaluates it on rumor detection and factuality in community question-answering forums.
1 Introduction
- The system retrieves Web evidence for a claim, then combines LSTM text encodings, task-specific embeddings, and source reliability information for fact checking.The framework is designed to be general-purpose and lightweight, with evidence derived from the Web.Source: §1 · p. 2
- The authors create and release a high-quality dataset for studying factuality of answers in community question-answering forums.They describe it as the first dataset specifically targeting factuality in a cQA setting.Source: §1 · p. 2
- The framework achieves strong results on two fact-checking tasks: rumor detection and answer factuality in community question-answering forums.The authors present this cross-task performance as evidence of the approach’s generality and potential applicability to different task formulations.Source: §1 · p. 2
2 The Fact-Checking System
The system verifies claims by retrieving Web support, representing claims and retrieved text, and predicting veracity. It uses neural and kernel-based classifiers, including a combined model that feeds neural task-specific representations to an SVM.
2 The Fact-Checking System
- The system’s three stages are external support retrieval, text representation, and veracity prediction.It starts from a claim and searches the Web for information to assess whether the claim is likely true.Source: §2 · p. 2
2.1 External Support Retrieval
- Queries are shortened from claims and sent to search engines to retrieve supporting documents, with no claim-verification-specific features such as temporal indicators.The system ranks terms with tf-idf, keeps verbs, nouns, and adjectives, adds named entities, and produces 5–10-token queries; it skips results from manually identified unreliable domains and relaxes queries when no results are returned.Source: §2.1 · pp. 2–3
2.2 Text Representation
- Claims are compared with snippets and Web pages using tf-idf cosine, embedding cosine, and containment similarities, alongside averaged-word and LSTM encodings.For long pages, the system scores rolling sentence triplets; it also uses the best-matching snippet and triplet, plus a task-specific embedding of the claim and evidence.Source: §2.2 · p. 3
2.3 Veracity Prediction
- The neural classifier uses five source-specific LSTM sub-networks and a compact hidden layer, while the SVM uses averaged embeddings; SVM+NN adds the neural hidden-layer values as SVM features.Figure 2 depicts the overall architecture and the bi-LSTM structure used in each blue box.Source: Figure 2 · p. 3, §2.3 · p. 4
3 Experiments and Evaluation
The experiments evaluate rumor detection on a recovered-claim subset and report results for models using Web evidence. The combined SVM+NN improves on standalone models, while comparisons with prior work have important scope differences.
3.1 Dataset
- The rumor dataset uses 761 recovered Snopes claims, labeled 34% true and 66% false, split into 509 training, 132 development, and 120 test examples.The study ignores tweets and adds Web-retrieved snippets, pages, and sentence triplets; the split was newly created.Source: §3.1 · p. 4
3.2 Experimental Setup
- The neural network uses a bidirectional LSTM with 25 units, while the SVM uses an RBF kernel selected through five-fold cross-validation and grid search.The neural network was tuned on the development set; the selected SVM parameters were c = 16 and γ = 0.01.Source: §3.2 · p. 4
3.3 Evaluation Metrics
- Evaluation reports precision, recall, and F1 for true and false claims, plus macro-average recall, macro-average F1, and accuracy.Source: §3.3 · p. 4
3.4 Results
- The combined SVM+NN improves over both standalone models in avgR, avgF1, and accuracy, and is more balanced between true and false claims.The NN consistently outperforms the SVM on all measures; both standalone models overpredict false claims.Source: Table 1 · p. 5, §3.4 · p. 5
- Direct comparison with Ma et al. (2016) is not possible because this study uses only 761 of 993 examples, a different class distribution, and a different task formulation.In 21% of Ma et al.’s examples there was no claim input.Source: §3.4 · p. 5
- NN task-specific embeddings help the SVM across retrieval settings; snippets and Web pages perform similarly with embeddings, and combining Google and Bing adds little.The baseline SVM performs slightly better with snippets than Web pages; Google versus Bing makes practically no difference.Source: Table 2 · p. 5, §3.4 · p. 5
- The system’s results are competitive with Ma et al.’s when their temporal features are excluded.The system is fully automatic and does not use tweets.Source: §3.4 · p. 5
- 66.7% to 80.0% accuracy corresponds to an almost identical 39.9% error reduction to Popat et al.’s 39.2%, but the comparison is indirect.The corpora and label distributions differ, so a direct comparison is impossible.Source: §3.4 · p. 5
4 Application to cQA
The cQA application evaluates Web-based fact checking of answers using a newly annotated dataset. Its models outperform baselines, while the small dataset limits the standalone neural network.
4 Application to cQA
- The dataset contains 249 question–answer pairs from 71 threads: 128 factually true and 121 factually false answers.Four annotators independently labeled examples, then discussed each to reach a final label; crowdsourcing was not used because pilot annotations found the task difficult to verify reliably.Source: §4 · p. 6, Table 3 · p. 6
- Only factual questions and GOOD answers were retained; answers judged time-sensitive or uncertain were discarded, with present circumstances used as the reference for some time-sensitive facts.The selected questions exclude socializing, advice requests, and questions with multiple sub-questions.Source: §4 · p. 6
- The dataset was split into 185 training, 31 development, and 32 test pairs, preserving the class ratio and preventing question overlap across splits.The annotations cover 249 pairs in total; the listed split sizes sum to 248.Source: §4 · p. 6
- For cQA, the system queried with the concatenated question and answer, used Google snippets from reputed Qatar-related sources, and represented both question and answer with LSTMs.The compared text was the question–answer concatenation; the authors report limiting retrieval to snippets because they had found them sufficiently rich in the earlier task.Source: §4 · p. 6
- The models outperformed all baselines on cQA; task-specific representations combined with SVM improved all measures consistently.Predictions across models were balanced between the two classes; the SVM alone outperformed the neural network, which the authors attribute to the dataset’s smaller size.Source: §4 · p. 7, Table 4 · p. 7
5 Related Work
Prior work addresses credibility and rumor detection through varied sources and signals, while this paper focuses on verifying factuality by gathering external support for cQA answers. The related work also contrasts the paper’s Web-text approach with structured-data and temporal methods.
5 Related Work
- The paper places its work within a broader rise of research on online information credibility, including rumor-detection and claim-verification efforts.The passage cites a 2016 journal special issue, a SemEval-2017 rumor-detection task, and an upcoming CLEF-2018 lab.Source: §5 · p. 7
- Earlier credibility research spans news portals and social media, including work on malicious users and manipulation accounts.Examples include source credibility shaped by topical content and social-network structure, and rumor models using user reputation, writing style, and time-based features.Source: §5 · p. 7
- VERA combines truth-discovery methods to support end-to-end fact checking from the Web, but takes structured data as input rather than the unstructured text used here.Its described pipeline extracts information from multiple sources, combines claim-level information, estimates veracity, and visualizes results.Source: §5 · p. 7
- Popat et al. (2016, 2017) gather Web support for claims, with the latter also modeling stance, source reliability, language style, and temporal information.The authors report similar performance to Popat et al. (2017), while noting that direct comparison is impossible because the datasets differ; their system uses fewer sources, a simpler model, and six times less training data.Source: §5 · pp. 7–8
- Temporal rumor methods use social-context changes over a rumor’s life cycle, whereas this framework queries the Web directly for evidence.Ma et al. (2015) use temporal patterns to detect rumors and predict their frequency; Ma et al. (2016) use recurrent networks for contextual variation over time.Source: §5 · p. 8
- Question-answer credibility research has used answer content, source credibility, sentiment, contradiction, language-model divergence, clicks, links, and user votes.The paper distinguishes credibility as subjective perception from factuality as verification of truth or falsity, and states that prior work had not targeted cQA answer fact-checking with external support.Source: §5 · p. 8
6 Conclusions and Future Work
The paper concludes that its Web-retrieval fact-checking method performs well across two tasks and can work efficiently with search-engine snippets. Future work would test additional datasets and complementary methods, provide justifications, and integrate the method into an application.
6 Conclusions and Future Work
- Strong performance across two fact-checking tasks supports the method’s generality and potential applicability to different domains and task formulations.The method retrieves supporting Web information and compares it with claims using machine learning; snippets alone can yield good performance and make the approach efficient.Source: §6 · p. 8
- Future work includes evaluating other datasets with complementary methods, exploring prediction justifications, and integrating the method into a real-world application.The authors give microblog and temporal-information methods for Twitter rumors as examples of complementary approaches.Source: §6 · p. 8