Source-linked AI summary
Automatic Stance Detection Using End-to-End Memory Networks
Mitra Mohtarami, Ramy Baly, James Glass, Preslav Nakov, Lluis Marquez, Alessandro Moschitti
TL;DR
Stance detection must identify relevant evidence paragraphs and their relationships to judge a claim. This paper uses a memory network to predict stance and extract supporting snippets, performing similarly to more complex state-of-the-art systems.
Problem
Stance detection requires identifying relevant paragraphs and referring across paragraphs, while CNNs and LSTMs are not well-suited to modeling complex dependencies between them.
Source: §6 · p. 9Method
An end-to-end memory network combines convolutional and recurrent networks with a similarity matrix to predict document stance and extract relevant evidence snippets.
Source: §1 · p. 1, §2.1 · p. 2Results
The feature-light model performs similarly to the state of the art; sMemNN improves over Gradient Boosting by 10.62 points in Macro-F1 and 3.77 points in Weighted Accuracy.
Source: §1 · p. 1, §3.6 · p. 6Takeaways & Limitations
The model extracts meaningful document snippets that can explain the factuality of a claim.
Source: §6 · p. 9Future work will extend inference to select optimal explanation sets and explain the model as a whole, beyond instance-level explanations.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We present a novel end-to-end memory network for stance detection, which jointly (i) predicts whether a document agrees, disagrees, discusses or is unrelated with respect to a given target claim, and also (ii) extracts snippets of evidence for that prediction. The network operates at the paragraph level and integrates convolutional and recurrent neural networks, as well as a similarity matrix as part of the overall architecture. The experimental evaluation on the Fake News Challenge dataset shows state-of-the-art performance.
1 Introduction
The paper addresses stance detection as a step toward automatic fact-checking, predicting how a document relates to a claim. It introduces a memory-network model that also extracts supporting snippets and performs similarly to more complex state-of-the-art systems on the Fake News Challenge benchmark.
1 Introduction
- The motivation is the spread of false information online, which has been linked to effects ranging from individual decisions to major events such as elections.Manual fact-checking is tedious, motivating automatic alternatives.Source: §1 · p. 1
- Stance detection classifies a text’s perspective toward a claim as agree, disagree, discuss, or unrelated.The paper presents it as an intermediate step in automatic fact-checking and as a useful stand-alone task for human experts.Source: §1 · p. 1
- The model jointly predicts a document’s stance toward a claim and extracts relevant supporting snippets, using a similarity matrix at inference to improve snippet extraction.The network is based on end-to-end memory networks and incorporates convolutional and recurrent neural networks.Source: §1 · p. 1
- The model performs similarly to state-of-the-art systems on the Fake News Challenge benchmark while remaining feature-light.Its similarity-matrix extension offers sizable performance gains, and the model extracts meaningful snippets that may help human experts assess a claim’s factuality.Source: §1 · p. 1
2 Model
The model treats document paragraphs as candidate evidence for a claim and combines memory-network processing with LSTM, CNN, and similarity-based inference. It predicts stance while extracting relevant evidence snippets.
2.1 Overview of the network
- The memory network combines a new inference component with a pipeline that identifies relevant input parts, updates memory, and produces a stance response.Long-term memory is motivated by the need to use relevant parts of long documents to determine a document’s stance toward a claim.Source: §2.1 · p. 2, §2 · p. 2
2.2 Input Representation Component
- Documents are segmented into paragraphs treated as candidate evidence, and each paragraph is represented in a 3D space using words and their neural representations.The input component encodes each paragraph with an LSTM; the same LSTM is applied across evidence pieces.Source: §2.2 · pp. 2–3
- CNN and LSTM paragraph representations are kept separate in the memory network, which can use either representation for the target task.A Maxout layer pools across CNN feature-map channels while retaining computed n-grams for later inference and response components.Source: §2.2 · p. 3
2.3 Inference Component
- The inference component computes claim–paragraph similarity with learned matrices applied to LSTM and CNN representations.The LSTM path accounts for word order and long-range dependencies, while the CNN path captures n-grams and local dependencies.Source: §2.3 · pp. 3–4, Figure 3 · p. 4
2.4 Memory and Generalization Components
- Memory representations are updated using claim–evidence similarity, with the stated goal of filtering out most unrelated evidence.The updated representations are used in subsequent inference steps for the LSTM and CNN paths.Source: §2.4 · p. 4
2.5 Output Representation Component
- The output representation combines the average CNN paragraph representation with maximum and average claim–evidence similarities.The maximum similarity indicates the most similar part of a paragraph, while the average measures its overall similarity to the claim.Source: §2.5 · p. 4
2.6 Response and Output Generation
- A multilayer perceptron and softmax use the output and claim representations to predict the document’s stance toward the claim.The response component also extracts snippets that best indicate the document’s perspective, using CNN similarities and the similarity matrix.Source: §2.6 · p. 4
- The model’s CNNs, LSTMs, similarity matrices, and multilayer perceptron are jointly learned during training.The full architecture is summarized as a memory-network algorithm for stance detection.Source: Table 1 · p. 2, §2.6 · p. 4
3 Experiments and Evaluation
Experiments on the Fake News Challenge evaluate stance models across several measures and show that sMemNN outperforms the baselines, with gains associated with its similarity matrix. Results also compare model variants, the challenge winner, and performance across stance classes.
3.1 Data
- The Fake News Challenge data contains 75.4K claim–document pairs linking 2.5K articles and 2.5K claims, with four labels: agree, disagree, discuss, and unrelated.Source: §3.1 · p. 5
3.2 Settings
- The models use 100-dimensional GloVe embeddings pretrained on two billion tweets, 100-unit LSTMs, 100 CNN filters of width five, and the first nine document paragraphs.Source: §3.2 · p. 5
3.3 Evaluation Measures
- Evaluation includes accuracy, Macro-F1, and Weighted Accuracy, which gives 0.25 credit for related/unrelated classification and an additional 0.75 for the correct related stance label.Source: §3.3 · p. 5
3.4 Baselines
- Baselines include always-unrelated and always-discuss predictions, CNN and LSTM variants, and the challenge’s Gradient Boosting model with word and n-gram overlap, refutation, and polarity features.Source: §3.4 · p. 5
3.5 Our Models
- sMemNN feeds separate CNN and LSTM outputs into a memory network; variants replace similarity matrices with dot products or add bag-of-words and TF.IDF cosine-similarity features.Source: §3.5 · pp. 5–6
3.6 Results
- LSTM outperforms CNN, LSTM+CNN beats either single network, Gradient Boosting beats the neural baselines, and BOW-enriched sMemNN outperforms sMemNN.Source: §3.6 · p. 6, §3.5 · p. 6
- sMemNN beats all baseline models, improving Macro-F1 by 10.62 points and Weighted Accuracy by 3.77 points over Gradient Boosting; replacing its similarity matrix with a dot product lowers performance to Gradient Boosting’s level.Source: §3.6 · p. 6, Table 2 · p. 6
- The challenge winner’s Macro-F1 of 57.79 is not significantly different from the full model’s at the 0.05 level; disagree examples are hardest, while agree and disagree are often labeled discuss.Source: §3.6 · p. 6
4 Discussion
Class-balanced sampling eventually observes almost all training examples, with validation loss reaching its minimum after nearly all related examples have been seen. The model also extracts evidence snippets for its predictions, though measured precision varies with the evaluation granularity.
4.1 Training Data Coverage
- At iteration 31, validation loss is lowest after almost all related examples have been observed, while only about half of unrelated examples have been seen; sampling eventually covers almost all training data.Source: §4.1 · p. 7
4.2 Explainability
- For 100 sampled agree/disagree examples, the model correctly classified 76 and supplied arguably adequate top-five evidence snippets; its inference component ranks claim-related text snippets.Source: §4.2 · pp. 7–8
- Evidence precision at ranks 1–5 is 0.28, 0.32, 0.35, 0.25, and 0.33 for five-grams, rising to 0.6, 0.58, 0.55, 0.62, and 0.57 under sentence-level evaluation.The paper attributes low five-gram precision mainly to the absence of gold evidence snippets for training.Source: §4.2 · p. 8
5 Related Work
Stance detection is presented as one component of automatic fact checking and also as a stand-alone task. Prior work covers different stance settings and feature-rich Fake News Challenge systems, while this paper uses a feature-light memory network that also highlights evidence.
5 Related Work
- Automatic fact checking has been framed as a multistep process from identifying check-worthy statements through inferring their veracity.Source: §5 · p. 8
- Stance detection also serves media monitoring and supports fact checking and veracity inference, although the tasks typically use separate datasets.Source: §5 · p. 8
- SemEval-2016 Task 6 labels stance toward propositions as infavor, against, or neither, while later work detects stance toward rumors and hot topics in conversational threads.Source: §5 · p. 8
- For the 2017 Fake News Challenge, the leading systems used gradient-boosted trees with rich features and CNNs, or neural networks with similarity features and other representations.The cited systems include Baird et al. (2017), Hanselowski et al. (2017), and Riedel et al. (2017).Source: §5 · p. 8
- Unlike prior Fake News Challenge systems, the paper uses a feature-light memory network that jointly infers stance and highlights relevant evidence snippets.Source: §5 · p. 8
6 Conclusion
The paper extends memory networks for stance detection with similarity-based inference and stance filtering, reporting performance gains and meaningful evidence snippets. Its current explanation scope is instance-level, with selecting optimal explanation sets and explaining the model globally left for future work.
6 Conclusion
- Stance detection requires focusing on relevant paragraphs and relating information across paragraphs, while basic memory networks struggle with irrelevant, noisy information in large documents.Source: §6 · p. 9
- The similarity-matrix and stance-filtering extension, applied at inference time, offers sizable performance gains and makes memory networks competitive.The model also extracts meaningful snippets that can explain a claim’s factuality.Source: §6 · p. 9
- Future work will select optimal explanation sets for each prediction and explain the model as a whole rather than only at the instance level.Source: §6 · p. 9