PaperlayerView exact revision on arXiv

Source-linked AI summary

BanFakeNews: A Dataset for Detecting Fake News in Bangla

Md Zobaer Hossain, Md Ashraful Rahman, Md Saiful Islam, Sudipta Kar

arXiv:2004.08789v1cs.CL

TL;DR

Automated fake-news research has largely focused on English, while the authors report no Bangla resource or computational approach for the task. They present a Bangla dataset and benchmark linguistic and neural methods; their analyses favor linear classifiers with traditional linguistic features over neural-network models.

  • Problem

    The authors report that prior computational fake-news approaches were for English and that, to their knowledge, Bangla lacked a resource or computational approach despite its large speaker population.

    Source: §1 · p. 1

  • Method

    The paper presents an annotated Bangla news dataset and benchmarks traditional linguistic features alongside neural-network architectures and pretrained Transformers.

    Source: §1 · p. 1

  • Results

    Linear classifiers with traditional linguistic features performed better than neural-network models, and character-level features were more significant than word-level features.

    Source: §8 · p. 6

  • Takeaways & Limitations

    The authors present the dataset as a resource for building automated Bangla fake-news detection systems and for further research on low-resource languages.

    Source: §1 · p. 1

    The authors had manually annotated about 8.5K news items and planned to continue annotation toward 50K.

    Source: §8 · p. 7

Abstract

from arXiv · show

Observing the damages that can be done by the rapid propagation of fake news in various sectors like politics and finance, automatic identification of fake news using linguistic analysis has drawn the attention of the research community. However, such methods are largely being developed for English where low resource languages remain out of the focus. But the risks spawned by fake and manipulative news are not confined by languages. In this work, we propose an annotated dataset of ~50K news that can be used for building automated fake news detection systems for a low resource language like Bangla. Additionally, we provide an analysis of the dataset and develop a benchmark system with state of the art NLP techniques to identify Bangla fake news. To create this system, we explore traditional linguistic features and neural network based methods. We expect this dataset will be a valuable resource for building technologies to prevent the spreading of fake news and contribute in research with low resource languages.

1. Introduction

The paper frames fake news as a fast-spreading source of harm and identifies a gap in computational detection for Bangla. It addresses that gap with a dataset, benchmark methods, and comparison with human performance.

1. Introduction

  • Fake news can mislead readers with fabricated information, and its spread has been linked to serious harms, including destruction during Bangladesh’s 2012 Ramu incident.Manual fact-checking websites cannot respond quickly to each fake-news event.Source: §1 · p. 1
  • Earlier computational fake-news approaches discussed by the authors addressed English, while they knew of no Bangla resource or computational approach for the problem.The authors describe Bangla as spoken by around 341 million people.Source: §1 · p. 1
  • The paper releases an annotated dataset of approximately 50K Bangla news items as a resource for automated fake-news detection.Source: §1 · p. 1
  • The authors benchmark linguistic features and neural architectures, analyze methods and results, and compare system performance with human performance.Source: §1 · p. 1

2. Related Works

Prior fake-news and related detection work spans linguistic features, machine learning, neural networks, and fact-checking, but the reviewed work largely concerns English. The paper positions its dataset as the first publicly available Bangla news dataset for fake-news detection and designs it for varied research uses.

2. Related Works

  • Prior English-language work includes linguistic-feature and neural approaches to satire, deception, clickbait, and fake-news detection, as well as fact-checking using knowledge bases or external sources.Source: §2 · p. 2
  • Existing datasets range from satirical and crowdsourced-fabrication collections to Liar’s 12.8K human-labeled statements and a 761-claim fact-checking dataset.Source: §2 · p. 2
  • The review describes limited work in lower-resource languages, including Indonesian and Chinese examples, and presents this as the first publicly available Bangla news dataset for fake-news detection.Source: §2 · p. 2
  • The Bangla dataset is designed for multiple research lines and includes clickbait, satirical, fake, and true news with headlines, articles, domains, and other metadata.Source: §2 · p. 2

3. A New Dataset for Detecting Fake News Written in Bangla

The dataset combines Bangla news from trusted mainstream portals with several kinds of fake news and associated metadata. The authors standardize publisher categories and manually annotate source and headline–article relations for a subset of the data.

3. A New Dataset for Detecting Fake News Written in Bangla

  • The authors select news from 22 popular, trusted mainstream Bangladeshi portals and use Alexa rankings to determine popularity.Source: §3 · p. 2
  • Fake-news items include misleading or false-context news, clickbait, and satire or parody; collected satire duplicates are discarded.False-context items are gathered from Jaachai and BDFactCheck and matched to their original publishing sites; clickbait is manually collected from local or less popular sites.Source: §3 · p. 2
  • The authors map 242 publisher categories into 12 generalized categories, and Table 1 reports the number of news items in each category.Source: §3 · p. 3, Table 1 · p. 3
  • The dataset records the news-site domain and, for manually annotated items, source information and whether the headline is related to the article.The authors had annotated about 8.5K items because those fields required checking each news item.Source: §3 · p. 3

4. Human Baseline

The human baseline tested how five undergraduate annotators judged fake and authentic Bangla news, and found modest fake-class F1 scores and low agreement. Their explanations point to content believability and source credibility as important cues, informing the dataset’s source annotations.

4. Human Baseline

  • The experiment presented headlines and articles for 60 fake and 90 authentic news items, asking annotators first to classify each and then select why.The second question offered four options for each class; the process is shown in the human-baseline flowchart.Source: §4 · p. 3, Figure 2 · p. 4
  • Given only headlines and articles, the human-baseline experiment estimated fake-news detection accuracy; the five annotators’ fake-class F1-scores were 58%, 65%, 70%, 68%, and 63% (range 58%–70%).Source: §4 · p. 3
  • Fleiss’ Kappa was 38.83% and mean pairwise Cohen’s Kappa was 39.05%, indicating annotators gave the same answer on nearly 39% of questions.Source: §4 · p. 3
  • For fake news, annotators most often cited unrealistic content (44%) or no trustworthy source (42%); for true news, they cited believable content (62%) or a reliable source (21%).Source: §4 · p. 3
  • Follow-up interviews identified disguise, repeated reporting, absent sources, and satire that resembles true news as reasons for human errors.Source: §4 · p. 4
  • The authors manually annotated news sources as metadata for possible use as a fake-news detection feature.They report that their analysis indicated source plays an influential role in detection.Source: §4 · p. 4

5. Methodologies

The paper benchmarks traditional linguistic features alongside neural classifiers and multilingual BERT for Bangla fake-news classification. Its feature sets represent lexical and syntactic patterns, embedding-based semantics, punctuation, and article or source metadata.

5.1. Traditional Linguistic Features

  • Traditional lexical and syntactic features combine word and character n-grams weighted by TF-IDF with normalized frequencies of ten part-of-speech tags.Source: §5.1 · p. 4
  • Semantic features represent articles using mean and standard deviation of pretrained word vectors, including FastText vectors and news-trained Word2Vec embeddings.The reported coverage was 55.21% for FastText and 53.95% for the news embeddings.Source: §5.1 · p. 4
  • Metadata and punctuation features include punctuation frequencies, headline and body lengths, and normalized Alexa site ranks, with missing ranks assigned the maximum observed rank.Source: §5.1 · p. 4

5.2. Neural Network Models

  • Neural benchmarks include CNNs with 1–4 token kernels and max or average pooling, plus two-layer BiLSTMs with attention.The CNN used 256 kernels per kernel length; the BiLSTM used 256 LSTM units.Source: §5.2 · pp. 4–5

5.3. Pre-trained Language Model

  • The pretrained-language-model experiment fine-tunes multilingual BERT to classify Bangla news documents.Source: §5.3 · p. 5

6. Experimental Setup

The experiments use preprocessing and class-sensitive metrics, compare against majority and random baselines, and select models using validation performance. The final reported evaluation uses a held-out test set.

6. Experimental Setup

  • Evaluation reports Micro-F1 and fake-class precision, recall, and F1 because the dataset is imbalanced, alongside majority and random baselines.The random baseline is averaged over ten experiments.Source: §6 · p. 5
  • Linguistic-feature models use a 70:30 train-test split and validation-tuned SVM penalty; neural models use Adam, learning rate 0.00002, and batch size 32.Neural models train for 50 epochs, use 10% of test data for validation, and report the best validation fake-class F1 checkpoint on the test set.Source: §6 · p. 5

7. Results and Analysis

The best reported traditional-feature result is SVM with all linguistic features at 91% F1-score; lexical features generally lead the alternatives, while neural models trail linear classifiers.

7. Results and Analysis

  • Lexical features outperform other linguistic features, baselines, and neural models; SVM generally beats LR and RF, except on news embeddings.For news embeddings, RF scores 55% F1, compared with 46% for SVM and 53% for LR.Source: §7 · p. 5, Figure 3 · p. 6
  • Authentic news outnumbers fake news 37.47 to 1, which the authors suggest could explain variation in overall and fake-class results.They therefore focus on fake-class precision, recall, and F1-score in the remaining analysis.Source: §7 · p. 5
  • Fake-class F1-score falls as word and character n-gram size increases, while POS features do not improve over random baselines.MP and word-embedding features outperform POS and the baselines, but remain behind lexical features and neural models.Source: §7 · p. 5
  • 91% F1-score is reported for SVM with all linguistic features, the best result in the experiments.Table 3 presents test-set results for traditional linguistic features and neural networks.Source: §7 · p. 5, Table 3 · p. 6
  • Neural models do not outperform linear classifiers; BERT leads the neural experiments at 68% F1-score.CNN average-pooling and global-max variants score 59% and 54% F1-score, respectively.Source: §7 · p. 6
  • On the human-baseline dataset, SVM with all linguistic features scores 90% F1-score and character 3-gram SVM scores 89%, versus humans’ 64.8%.The models correctly identify 50 and 49 fake stories, respectively.Source: §7 · p. 6

8. Conclusion

The paper presents a labeled Bangla fake-news dataset and finds traditional linguistic features, especially character-level features, useful in its evaluation.

8. Conclusion

  • The paper presents the first labeled Bangla fake-news dataset and reports that around 8.5K items have been manually annotated toward a 50K target.The authors say they will continue expanding the dataset.Source: §8 · pp. 6–7
  • Linear classifiers with traditional linguistic features outperform the neural-network models, and character-level features are more significant than word-level features.The authors plan to incorporate character-level features into neural models.Source: §8 · pp. 6–7
  • The analysis also finds punctuation more frequent in fake news and fake news most often appearing on the least popular sites.The authors identify source information as a future feature to include in experiments.Source: §8 · pp. 6–7

A News Sources

The appendix lists news sources with domain URLs, collected-news statistics, and Alexa rankings dated 08 March 2020.

A News Sources

  • The cited passages list three records with paired numeric values: channeldhaka.news (249033, 436), motikontho.wordpress.com (7291230, 195), and bengalbeats.com (707465, 192).Source: A News Sources · p. 9
  • Other listed sources include prothombarta.news (193562, 2), aparadhchokh24bd.com (4242479, 1), and bengaliviralnews.com (1884782, 3).The paired values are reproduced as listed in the source entries.Source: A News Sources · p. 9
  • The source inventory reports domain URLs and Alexa rankings alongside detailed statistics for collected news.The ranking reference date is 08 March 2020.Source: Table 4 · p. 10
Loading 2004.08789v1…