PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Fake News Detection

Santiago Alonso-Bartolome, Isabel Segura-Bedmar

arXiv:2112.04831v1cs.CL

TL;DR

Fake news is proliferating and can have effects across different societal segments. This paper studies fine-grained fake-news classification with a multimodal CNN combining text and images, and reports that its multimodal approach outperforms approaches using text alone.

  • Problem

    Fake news is proliferating and its dissemination can affect different segments of society.

    Source: §1 · p. 3

  • Method

    The paper addresses fine-grained fake-news classification with a CNN that combines text and image data.

    Source: §1 · p. 4, §5 · p. 26

  • Results

    The multimodal approach outperforms approaches that exploit only text, while BERT is the best model for text classification.

    Source: §5 · p. 26

  • Takeaways & Limitations

    In this study, using text and image data together yields better results than approaches exploiting text alone.

    Source: §5 · p. 26

    Dynamic training takes much more time than static training.

    Source: §4.2.1 · p. 22

Abstract

from arXiv · show

Over the last years, there has been an unprecedented proliferation of fake news. As a consequence, we are more susceptible to the pernicious impact that misinformation and disinformation spreading can have in different segments of our society. Thus, the development of tools for automatic detection of fake news plays and important role in the prevention of its negative effects. Most attempts to detect and classify false content focus only on using textual information. Multimodal approaches are less frequent and they typically classify news either as true or fake. In this work, we perform a fine-grained classification of fake news on the Fakeddit dataset, using both unimodal and multimodal approaches. Our experiments show that the multimodal approach based on a Convolutional Neural Network (CNN) architecture combining text and image data achieves the best results, with an accuracy of 87%. Some fake news categories such as Manipulated content, Satire or False connection strongly benefit from the use of images. Using images also improves the results of the other categories, but with less impact. Regarding the unimodal approaches using only text, Bidirectional Encoder Representations from Transformers (BERT) is the best model with an accuracy of 78%. Therefore, exploiting both text and image data significantly improves the performance of fake news detection.

1. Introduction

The paper studies unimodal and multimodal methods for fine-grained fake-news classification, addressing a literature that largely uses text alone and typically frames multimodal detection as true versus fake. It motivates automated detection by the societal harms of fake news and the impracticality of manual fact-checking at scale.

1. Introduction

  • Fake-news dissemination can affect different societal segments, with examples including vaccine hesitancy, economic losses, and tragic incidents.The introduction also mentions effects on health behaviour intentions and the Pizzagate incident and mob lynchings in India.Source: §1 · p. 3
  • The volume of digital information makes manual fake-news fact-checking impossible, motivating automated identification for more immediate action.The paper describes fake-news dissemination as harmful across societal segments.Source: §1 · p. 3
  • Most cited detection work uses text alone; multimodal approaches are less explored, and prior multimodal studies typically classify news as true or fake.The introduction notes that multimodal studies have obtained better results than unimodal approaches.Source: §1 · p. 3
  • The paper aims to compare unimodal and multimodal approaches for fine-grained fake-news classification, beyond the binary true-or-fake framing typical of prior multimodal studies.The authors use the Fakeddit dataset for this study.Source: §1 · p. 3
  • The study uses Reddit posts from Fakeddit, which assigns six classes: true, misleading, manipulated, false connection, imposter, and satire.The paper explores CNN, BiLSTM, and BERT architectures for text classification.Source: §1 · p. 4
  • For multimodal classification, the authors propose a CNN architecture combining text and image data.The paper also explores unimodal text approaches, including CNN, BiLSTM, and BERT.Source: §1 · p. 4

2. Related work

The section reviews text-only and multimodal fake-news detection, including the use of text and images and the distinction between binary and fine-grained classification. It situates this work’s fine-grained classification of the full Fakeddit dataset with a simple multimodal CNN among prior multimodal studies.

2. Related work

  • Text-only approaches predominate, although studies have also combined text and images because images may provide useful classification information.Source: §2 · p. 4

2.1. Unimodal approaches for fake news detection

  • The proposed model achieved 99.8% accuracy on the ISOT dataset (binary classification) and 39.5% on the LIAR dataset; DistilBERT achieved 98.41% when pre-trained on a corpus of Covid-19 tweets.Source: §2.1 · p. 5
  • Deep-learning text models achieve very high accuracy on binary fake-news classification but perform much worse on fine-grained classes, and BERT has rarely been used for multiclass fake-news classification.Source: §2.1 · p. 5

2.2. Multimodal approaches for fake news detection

  • 95.18% accuracy was the best result for Singh et al. (2021), who compared traditional machine-learning methods for binary classification on a Kaggle fake-news dataset.Source: §2.2 · p. 6
  • 88.3% accuracy on Twitter and 83.2% on Weibo were reported for Kumari & Ekbal (2021), whose model fuses text and visual features using multimodal factorized bilinear pooling.Source: §2.2 · pp. 6–7
  • 86.4% accuracy on Fakeddit binary classification and 95.2% on BuzzFeed were reported for Kaliy ar et al. (2020)’s DeepNet model.Source: §2.2 · p. 7
  • 95.5% was the best accuracy reported by Kirchknopf et al. (2021) using news text, comments, images, and other metadata for binary classification on Fakeddit.Source: §2.2 · p. 7
  • The Modal Alignment and Fusion Network (EMAF) for binary fake-news detection achieved accuracies of 92.3%, 97.4%, and 80.5 % on Fakeddit, Weibo, and Twitter, respectively.Source: §2.2 · p. 7
  • A stance- and text/image-based multimodal model performs binary classification, reaching 96.63% accuracy on Fakeddit and 76.53% on PHEME.Source: §2.2 · pp. 7–8
  • Kang et al. (2021) use a heterogeneous news-detection graph and HDGCN, reporting F1 scores of 96% on Weibo and 88.5%, 85.8%, and 83.2% on Fakeddit binary, three-class, and six-class tasks.Source: §2.2 · p. 8
  • Most multimodal approaches evaluated on Fakeddit addressed binary classification; only Kang et al. (2021) addressed multiclass classification, using a reduced dataset version.Source: §2.2 · p. 8
  • To their knowledge, this is the first fine-grained fake-news classification using the whole Fakeddit dataset, with a simple multimodal CNN achieving very similar performance to Kang et al. (2021).Source: §2.2 · p. 8

3. Methods

The methods describe text-only fake-news classifiers using CNN, BiLSTM, and fine-tuned BERT, including text preprocessing and encoding. They also specify a text-image CNN that processes paired inputs separately and fuses their representations for six-class prediction.

3.1. Unimodal approaches

  • The text-only unimodal approaches include CNN and BiLSTM models, alongside a BERT classifier.The methods section identifies three text-only unimodal models; the supplied passage names CNN and BiLSTM, and BERT is described in its own subsection.Source: §3.1 · p. 8, §3.1.4 · p. 12

3.1.2. CNN

  • The text CNN embeds 15-token inputs in 300 dimensions, applies four convolutional filters and max pooling, then predicts among six classes.Embeddings use random initialization or pretrained 300-dimensional GloVe vectors, with static and dynamically trained variants.Source: §3.1.2 · p. 10

3.1.3. BiLSTM

  • The hybrid model transforms each length-15 input vector into a 15 x 300 word-embedding matrix, passes it through a bidirectional LSTM with hidden states of length 70, then applies 240 convolutional filters.Source: §3.1.3 · p. 11

3.1.4. BERT

3.2. Multimodal approach

  • The multimodal CNN takes paired text and images, combines their processed representations, and predicts one of six classes.The concatenated vector passes through two dense layers with a ReLU activation between them before classification.Source: §3.2 · pp. 13–14
  • The text branch uses the preprocessing described in 3.1.1 and the CNN from 3.1.2, but removes its last two dense ReLU layers.Source: §3.2 · pp. 13–14
  • Images are reshaped to 560 x 560, then a 3-to-6-channel convolution with 5 x 5 filters produces six 556 x 556 feature maps.The convolution uses stride 1 and no padding.Source: §3.2 · pp. 13–14
  • After ReLU and max pooling, a second convolutional stage yields three 137 x 137 feature maps, flattened into a vector of length 56307.Source: §3.2 · p. 14

4. Evaluation

The evaluation compares text-only and multimodal approaches on Fakeddit across true news and five fake-news categories, with performance varying by class and embedding choice. It also reports how the approaches perform overall and across individual categories.

4.1. Dataset

  • Fakeddit’s unimodal and multimodal datasets have similar class distributions across splits, but True, Manipulated content, and False connection are more represented than the other classes.Satire, Misleading content, and Imposter content are underrepresented, which may make their classification more difficult.Source: §4.1 · pp. 15–16

4.2. Results

  • With random initialization, the CNN achieved 72% accuracy, 57% micro F1, and 49% macro F1; class F1 was highest for True and Manipulated content at 79% and lowest for Imposter content at 13%.Source: Table 1 · p. 18, §4.2.1 · p. 18
  • CNNs initialized with pretrained GloVe outperformed random initialization in both static and dynamic approaches.The authors attribute this to GloVe vectors encoding relationships between words.Source: §4.2.1 · p. 20
  • In the overall comparison of the three models, the best was a CNN with dynamic GloVe vectors, whose training took around 6,000–8,000 seconds more than static training.Dynamic vectors slightly improved accuracy, by roughly one percentage point, while the static model had better micro F1.Source: §4.2.1 · pp. 19–20
  • Random-initialized BiLSTM and CNN had the same accuracy, 0.72, but BiLSTM achieved up to 9 points more micro F1, partly by improving Imposter content scores.The comparison is between the BiLSTM and CNN models with random initialization.Source: Table 1 · p. 18, Table 4 · p. 21, §4.2.1 · p. 21
  • Static GloVe improved BiLSTM class performance over random initialization, raising F1 by 6 points for False connection, 12 for Satire, 3 for Misleading content, and 10 for Imposter.The paper concludes that pretrained GloVe vectors perform better than random initialization for BiLSTM.Source: §4.2.1 · p. 21
  • 75% accuracy with dynamic GloVe vectors is two points above static GloVe, and dynamic vectors improve every class by 1–4 points.Dynamic GloVe also gives BiLSTM its best micro F1, roughly one percentage point above CNN with dynamic GloVe, but requires much longer training.Source: §4.2.1 · p. 22
  • Dynamic GloVe training takes substantially more time than static training.The authors note the longer training time when discussing the dynamic-vector approach.Source: §4.2.1 · p. 22
  • BERT achieves 78% accuracy and 74% micro F1, outperforming all previous unimodal deep-learning approaches and performing better in all classes; True and Manipulated content tie for highest F1 at 83%.Imposter content is the worst-performing class, with F1=38%.Source: §4.2.1 · pp. 23–24

4.2.2. Multimodal approach

  • 87% accuracy and 72% micro F1 were reported for the multimodal approach, with Manipulated content at 100% F1 and Imposter content at 32%.The authors suggest manipulated images may be readily detected by CNN, whereas images provide little information for identifying Imposter content; Imposter F1 is six points below BERT’s 38%.Source: Table 8 · p. 24, §4.2.2 · pp. 24–25
  • 76% F1 for False connection is 8 points above BERT and 15 above text-only CNN, while Satire improves 16 points over both unimodal models.The reported gains indicate image use significantly improves results for these two classes.Source: §4.2.2 · p. 25

4.2.3. Comparison of the best models

  • The multimodal CNN outperforms all unimodal approaches for fine-grained fake-news classification; among unimodal models, BERT leads on accuracy and micro F1.The comparison also places BiLSTM with dynamic GloVe vectors third and reports that all deep-learning approaches outperform the SVM baseline.Source: Table 9 · p. 25, §4.2.3 · pp. 25–26

5. Conclusions

The conclusion argues that fake news can harm politics, health, and the economy, motivating rapid, reliable detection tools. It reports that multimodal approaches outperform text-only approaches in the paper’s fine-grained classification comparison.

5. Conclusions

  • The study compares unimodal and multimodal deep-learning approaches for fine-grained fake-news classification, with the Fakeddit creators’ work as an exception to its novelty claim.The authors describe the comparison as comprehensive and qualify their claim with “to the best of our knowledge.”Source: §5 · p. 26
  • The multimodal approach outperforms approaches that use only text.The conclusion presents this as the central result of the comparison.Source: §5 · p. 26
  • BERT is the best text-classification model, while dynamic GloVe embeddings outperform random initialization for CNN and BiLSTM.The conclusion states these comparisons separately for the text model and the embedding initialization.Source: §5 · p. 26
  • Future work includes using VGG pretrained on a large image dataset and exploring other deep-learning techniques and ways to combine visual and textual representations.The named techniques include LSTM, BiLSTM, GRU, and BERT.Source: §5 · p. 26
  • The current multimodal CNN uses early fusion: it combines textual and visual representations before applying a classifier.The classifier operates on the resulting combined representation.Source: §5 · p. 26
  • The authors plan to study late fusion, using separate text and image classifiers whose predictions are combined for a final prediction.This is proposed as an alternative to the current early-fusion approach.Source: §5 · p. 26
Loading 2112.04831v1…