PaperlayerView exact revision on arXiv

Source-linked AI summary

Combining Similarity Features and Deep Representation Learning for Stance Detection in the Context of Checking Fake News

Luís Borges, Bruno Martins, Pável Calado

arXiv:1811.00706v1cs.LGcs.IRstat.ML

TL;DR

News veracity is difficult to assess, and FNC-1 tests whether an article agrees with, disagrees with, discusses, or is unrelated to a headline. This paper combines neural text representations, similarity features, and NLI-data pre-training; the resulting model slightly outperforms the previous state of the art.

  • Problem

    FNC-1 systems must classify an article’s stance toward a headline, while assessing news veracity remains difficult and standard neural models have struggled with long article bodies.

    Source: §1 · p. 2

  • Method

    The model combines bi-directional RNN representations of headlines and article text with similarity features, and explores pre-training components on SNLI and MultiNLI sentence pairs.

    Source: §1 · pp. 2–3, §5 · p. 21

  • Results

    6.92 percentage points: external features increased weighted accuracy, and the model slightly outperformed the previous state-of-the-art.

    Source: §5 · p. 21

  • Takeaways & Limitations

    The results confirm that external similarity features are essential to the proposed model’s strong FNC-1 performance.

    Source: §5 · p. 21

    The NLI pre-training results are not directly comparable with previous NLI systems because training combined SNLI and MultiNLI data and did not optimize NLI performance.

    Source: §4.3 · p. 18

Abstract

from arXiv · show

Fake news are nowadays an issue of pressing concern, given their recent rise as a potential threat to high-quality journalism and well-informed public discourse. The Fake News Challenge (FNC-1) was organized in 2017 to encourage the development of machine learning-based classification systems for stance detection (i.e., for identifying whether a particular news article agrees, disagrees, discusses, or is unrelated to a particular news headline), thus helping in the detection and analysis of possible instances of fake news. This article presents a new approach to tackle this stance detection problem, based on the combination of string similarity features with a deep neural architecture that leverages ideas previously advanced in the context of learning efficient text representations, document classification, and natural language inference. Specifically, we use bi-directional Recurrent Neural Networks, together with max-pooling over the temporal/sequential dimension and neural attention, for representing (i) the headline, (ii) the first two sentences of the news article, and (iii) the entire news article. These representations are then combined/compared, complemented with similarity features inspired on other FNC-1 approaches, and passed to a final layer that predicts the stance of the article towards the headline. We also explore the use of external sources of information, specifically large datasets of sentence pairs originally proposed for training and evaluating natural language inference methods, in order to pre-train specific components of the neural network architecture (e.g., the RNNs used for encoding sentences). The obtained results attest to the effectiveness of the proposed ideas and show that our model, particularly when considering pre-training and the combination of neural representations together with similarity features, slightly outperforms the previous state-of-the-art.

1 INTRODUCTION

Because assessing news veracity is difficult, FNC-1 frames stance detection as classifying whether an article agrees with, disagrees with, discusses, or is unrelated to a headline. The paper combines neural representations with similarity features and reports a slight improvement over the previous state of the art.

1 INTRODUCTION

  • FNC-1 asks systems to classify an article’s stance toward a headline, while evaluating news veracity remains demanding even for trained specialists.Stance detection can help a human checker retrieve related articles that agree, disagree, or discuss an allegation or title.Source: §1 · p. 2
  • FNC-1’s winning systems combined similarity features with text representations, while standard neural models struggled to model semantic relations across long article bodies.The organizers’ feature-engineering baseline was competitive with leading systems and outperformed several widely used deep-learning architectures.Source: §1 · p. 2
  • The model combines headline, first-two-sentence, and whole-document representations using bi-directional RNNs, then compares them with hand-crafted similarity features.It also explores hierarchical document encoding, max-pooling, neural attention, and pre-training sentence encoders with SNLI and MultiNLI data.Source: §1 · pp. 2–3
  • The complete method slightly outperforms the previous state-of-the-art FNC-1 approach, while pre-training and external similarity features improve results.The experiments also indicate that matching the headline against the article’s first two sentences adds no useful information.Source: §1 · p. 3

2 CONCEPTS AND RELATED WORK

This section reviews neural-network methods for text representation and prior systems for FNC-1 stance detection. It also reports accuracy figures from earlier models evaluated on different data splits.

2.1 Deep Neural Networks for Natural Language Processing

  • For text representation, the section covers sequence encoders and attention, and notes that NLI-trained sentence representations can initialize models for other NLP tasks.Related work includes hierarchical bi-directional GRU attention for documents and sentence encoders trained on SNLI or MultiNLI for transferable representations.Source: §2.1 · pp. 6–7

2.2 Stance Detection in the Context of the Fake News Challenge

  • On FNC-1, the baseline scored 75.20% weighted accuracy, while SOLAT, Athene, and UCL reached 82.02%, 81.97%, and 81.72%, respectively.The baseline used a gradient-boosting classifier with hand-crafted similarity features; all three top teams combined hand-crafted features with neural approaches.Source: §2.2 · p. 8
  • Tosik et al. [45] scored 78.63% weighted accuracy, while Pfohl and Legros achieved 80.8% with CEA-LSTM, which did not outperform simpler approaches.Tosik et al.’s ablations favored n-gram and word overlap, lexicon-based refutation, and distance features; sentiment and Hamming distance did not help.Source: §2.2 · pp. 8–9
  • Bhatt et al. [2] reported 83.08% weighted accuracy, exceeding the FNC-1 winner and, to the authors’ knowledge, setting the task’s state of the art.Their MLP combined skip-thought representations, TF-IDF statistical features, and heuristic features such as word overlap and cosine similarity.Source: §2.2 · p. 9
  • On custom, non-comparable FNC-1 splits, Chaudhry et al. reported 95.3% weighted accuracy and Bourgonje et al. reported 89.59%.Chaudhry et al.’s best model used a bi-directional encoder; Bourgonje et al. used a two-stage approach to classify relatedness and then stance.Source: §2.2 · pp. 9–10
  • Zeng et al. tested six neural encoders, including separate, chained, and concatenated bi-directional GRUs and two attention-based variants.The attention models compared body states with either the headline’s final state or all headline states.Source: §2.2 · p. 10
  • Mohtarami et al.’s [30] memory network encoded article paragraphs and the headline with LSTMs and CNNs, then used learned similarities and TF-IDF cosine similarity as input features.The article is represented as paragraphs, words, and embeddings; paragraph representations are compared with headline representations using trainable similarity matrices.Source: §2.2 · pp. 10–11
  • The memory network iteratively updated article paragraph representations using memory and generalization components before computing its output for stance prediction.Updated LSTM-based similarities were used to update CNN representations, which were then combined with headline CNN representations.Source: §2.2 · p. 11
  • Mohtarami et al. achieved weighted accuracy of 81.23% in their tests on the FNC-1 task, with the response component concatenating its output with LSTM and CNN representations before feed-forward layers and a final softmax prediction.Source: §2.2 · p. 11

3 PROPOSED NEURAL NETWORK ARCHITECTURE

The architecture encodes headlines and article text at sentence and document levels, then matches their representations and combines them with external similarity features to classify stance. Its components include sentence-level summarization, hierarchical document encoding, and optional pre-training on natural-language-inference data.

3 PROPOSED NEURAL NETWORK ARCHITECTURE

  • Headline, full-body, and optional first-two-sentence representations are matched with vector operations, combined with external features, and passed to layers that predict stance.The first-two-sentence branch compares the headline with the article opening.Source: §3 · p. 11

3.1 Encoding Sentences with Bi-Directional RNNs and Summarization Mechanisms

  • A sentence encoder embeds words with pre-trained GloVe vectors, processes them with a bi-directional RNN, and summarizes hidden states using max-pooling and/or inner attention.The encoder can use stacked bi-RNN layers with shortcut connections and concatenate max-pooling with attention.Source: §3.1 · pp. 12–13

3.2 Hierarchical Approach for Encoding Documents

  • The document encoder applies the sentence encoder to each article sentence, then encodes the resulting sentence-vector sequence with a bi-directional RNN and max-pooling and/or inner attention.The document-level encoder can also use two stacked bi-RNNs with shortcut connections.Source: §3.2 · p. 13

3.3 Model Pre-Training with the SNLI and MultiNLI Datasets

  • SNLI and MultiNLI pre-train the sentence encoder and headline-to-opening-sentences matcher, which are reused in the FNC-1 architecture after removing the NLI softmax layer.The headline serves as the NLI hypothesis and the article’s first two sentences as the premise.Source: §3.3 · p. 13

3.4 Combining the Matched Representations with External Features

  • The model combines headline–body neural matches with external similarity features, including text-overlap metrics such as BLEU, ROUGE, and CIDEr.Soft Cosine Similarity uses a sparse GloVe-based term-similarity matrix, allowing related words to contribute even without exact overlap.Source: §3.4 · pp. 14–15

4 EXPERIMENTAL EVALUATION

The evaluation tests the model and its components on FNC-1, including the effects of NLI pre-training and external features. It also examines attention weights as a source of clues about classification decisions.

4.1 The FNC-1 Dataset and Experimental Methodology

  • FNC-1 supplied 49,972 labeled training instances and 25,419 labeled test instances, and its weighted score gives credit for related labels as well as exact stance matches.Correctly identifying an unrelated instance earns 0.25; a correct related stance earns 1.0.Source: §4.1 · p. 16

4.2 Parameters Involved in the Proposed Approach

  • The sentence-matching component was pre-trained on SNLI and MultiNLI, and its NLI results are not directly comparable with prior systems because training combined both corpora and did not optimize NLI performance.The stated focus was pre-training representations of sentence meaning.Source: §4.3 · pp. 17–18, Table 2 · p. 17

4.3 The Obtained Results

  • The complete neural model outperformed an MLP using the full external-feature set on the FNC-1 test split.Table 3 compares prior systems, the external-feature MLP, and the proposed neural architecture.Source: Table 3 · p. 18, §4.3 · p. 19
  • 83.38% weighted accuracy was achieved by the model without the branch matching the headline to the article’s first two sentences, slightly exceeding prior state of the art.The ablations suggest this branch adds no useful information for FNC-1.Source: §4.3 · p. 19, Table 4 · p. 19
  • Without NLI pre-training, weighted accuracy was 81.85% (82.02% when the first-two-sentences branch was also removed); without external features, it was 75.31%.The authors describe both pre-training and external features as essential to good results with this architecture.Source: §4.3 · p. 19, Table 4 · p. 19
  • Attention examples show high weights on stance-relevant wording, including “fake” in an agreeing headline and denial language in the corresponding article.A disagreeing example assigns the highest weight to a sentence denying that climate change is a hoax.Source: §4.3 · pp. 19–21
  • Normalized attention weights for “fraud” and “debunk” tend to be higher in disagreeing than agreeing test instances.Figure 5 contrasts the word-weight distributions across those two stance classes.Source: Figure 5 · p. 20, §4.3 · p. 21
  • The FNC-1 results support the importance of external similarity features for matching long texts and the usefulness of pre-training for deep text representations.Source: §4.3 · p. 21

5 CONCLUSIONS AND FUTURE WORK

The paper combines bidirectional RNN representations and external similarity features for FNC-1 stance detection, and explores pre-training with natural language inference data. Its model slightly outperforms the previous state of the art, while the conclusion outlines several directions for extending the approach.

5 CONCLUSIONS AND FUTURE WORK

  • The method combines bidirectional RNN representations of headlines and article bodies with max-pooling, neural attention, and external similarity features.Source: §5 · p. 21
  • With pre-training and external similarity features, the model slightly outperforms prior state of the art; external features raise weighted accuracy by 6.92 percentage points.Source: §5 · p. 21
  • Pre-training the sentence encoder on SNLI and MultiNLI yields a new state-of-the-art result on the FNC-1 test set.Source: §5 · p. 21
  • Future work could adapt more advanced sentence-modeling and representation-matching methods from natural language inference, including transformer-like approaches.Source: §5 · p. 21
  • The authors propose exploring attention-weight analysis and sparsemax to seek a more selective, compact focus that may facilitate interpretability.Source: §5 · p. 22
  • They also suggest combining text-mining methods with external features capturing news-story characteristics such as diffusion patterns or inconsistencies against outside sources.Source: §5 · p. 22
Loading 1811.00706v1…