PaperlayerView exact revision on arXiv

Source-linked AI summary

BanglaMamba: Exploring State Space Models for Bangla Fake News Detection

M. K. Khalidi Siam

arXiv:2608.25190v1cs.CLcs.LG

TL;DR

Fake news detection matters because misinformation spreads rapidly and can undermine trust, while Transformer costs hinder long-sequence processing. This paper evaluates BanglaMamba against BanglaBERT and CustomBERT, finding competitive efficiency and classification results but weaker out-of-domain generalization for models trained from scratch.

  • Problem

    Rapidly spreading misinformation makes Bangla fake news detection important, while Transformer self-attention imposes quadratic time and memory costs and SSM applications remain underexplored.

  • Method

    The paper investigates BanglaMamba and systematically compares it with pre-trained BanglaBERT and similarly configured from-scratch CustomBERT across performance, efficiency, and external-dataset generalization.

  • Results

    BanglaBERT achieves the highest in-domain classification performance, while BanglaMamba remains competitive and provides lower latency, higher throughput, and lower GPU memory usage than BERT-based models.

  • Takeaways & Limitations

    Mamba-based SSMs are a promising, computationally efficient alternative for Bangla fake news detection, especially in resource-constrained settings.

  • Takeaways & Limitations

    BanglaMamba is trained from scratch on the downstream dataset, and its weaker external-dataset performance may reflect missing large-scale pretraining as well as architectural or optimization differences.

Abstract

from arXiv · show

Fake news detection has become an important Natural Language Processing (NLP) task due to the rapid spread of misinformation through online news platforms and social media. While transformer-based models such as BanglaBERT achieve strong performance for Bangla text classification, their quadratic computational complexity makes them less suitable for long-document processing in resource-constrained environments. This paper investigates Mamba-based State Space Models (SSMs) as an efficient alternative for Bangla fake news detection. We propose BanglaMamba and compare it with pre-trained BanglaBERT and a similarly configured BERT model trained from scratch. Experimental results show that BanglaBERT achieves the highest Macro-F1 score (0.9260), while BanglaMamba (0.9029) achieves performance comparable to the from-scratch CustomBERT (0.9057) despite using a different architecture. Meanwhile, BanglaMamba achieves approximately $2.2\times$ higher inference throughput and 49% lower inference peak GPU memory usage than the BERT-based models. Cross-dataset evaluation shows that BanglaBERT generalizes better to an external dataset, highlighting the importance of large-scale pretraining. These findings demonstrate that Mamba-based SSMs can provide a competitive and computationally efficient alternative to Transformer-based architectures for Bangla fake news detection, particularly in resource-constrained settings.

1 INTRODUCTION

Bangla fake news detection matters because misinformation spreads rapidly and can undermine trust, while transformer models face quadratic costs on long sequences. This paper investigates BanglaMamba, a Mamba-based SSM, and systematically compares it with BanglaBERT and CustomBERT.

  • Misinformation spreads rapidly through online news and social media, potentially influencing public opinion and undermining trust in credible sources.
  • Transformer models improve contextual fake news detection, but self-attention incurs quadratic time and memory complexity as input length increases.
  • Mamba-based SSMs offer linear scaling for long-sequence modeling, yet their application to Bangla fake news detection remains largely unexplored.
  • The study investigates BanglaMamba and compares it with pretrained BanglaBERT and similarly configured scratch-trained CustomBERT.
  • The comparison covers classification performance, computational efficiency, and out-of-domain generalization across the three models.

2 RELATED WORK

Prior Bangla fake news research used traditional, deep learning, and Transformer approaches, while newer work motivates efficient SSM alternatives. The paper identifies a lack of Bangla SSM studies and systematic architecture-efficiency comparisons.

  • BanglaBERT outperformed evaluated machine learning methods in one Bangla study, demonstrating the value of contextual language representations.
  • CNN-based systems sometimes matched or exceeded Transformer models, although BanglaBERT remained competitive across reported comparisons.
  • Across English datasets, fine-tuned Transformers achieved a strong balance between classification accuracy and cross-domain generalization.
  • Earlier Bangla fake news studies commonly used BoW, TF-IDF, stemming, or static Word2Vec embeddings, limiting contextual and long-range modeling.
  • Transformers face quadratic sequence-length costs, whereas S4 and Mamba support efficient long-sequence modeling; no published study had applied SSMs to Bangla fake news detection.

3 METHODOLOGY

The study formulates Bangla fake news detection as binary article classification and evaluates BanglaBERT, CustomBERT, and BanglaMamba under a common experimental setup. It combines matched tokenization and dataset procedures with predictive, reliability, and efficiency metrics.

  • 3.1 PROBLEM FORMULATION: The task classifies each Bangla article as Authentic (label 1) or Fake (label 0) from its headline and body text.
  • 3.4 MODELS: The models are compared to separate architectural effects from the effects of large-scale language pretraining.
  • 3.2 DATASETS: Experiments use BanFakeNews-2.0 for training and testing, with BanglaFakeNews2025 providing external evaluation for out-of-domain generalization.
  • 3.3 PREPROCESSING: Articles longer than 512 tokens are truncated to the first 512 tokens, while shorter articles are processed completely.
  • 3.3 PREPROCESSING: All models use the BanglaBERT SentencePiece tokenizer with a 32,000-token vocabulary, padding shorter sequences and truncating longer ones.
  • 3.4 MODELS: BanglaBERT is pretrained, CustomBERT shares its configuration but trains from scratch, and BanglaMamba is a scratch-trained Mamba SSM.
  • 3.4.2 BANGLAMAMBA: BanglaMamba uses an embedding layer, Mamba backbone, masked mean pooling, and a classification head for binary classification.
  • 3.5 TRAINING: Models train with weighted cross-entropy and AdamW for five epochs using BF16 mixed precision on one NVIDIA A100 GPU.

4 RESULT ANALYSIS

Across classification, efficiency, input-length robustness, cross-dataset generalization, and pooling analyses, BanglaMamba offers competitive scratch-trained performance and lower inference cost, but BanglaBERT remains strongest overall and generalizes better externally.

  • Classification Performance Comparison: BanglaBERT achieves the highest classification performance, while BanglaMamba and CustomBERT obtain similar Macro-F1 scores of 0.9029 and 0.9057, respectively.Both scratch-trained models also show similar Fake F1 scores, at 0.8361 for BanglaMamba and 0.8408 for CustomBERT.
  • Classification Performance Comparison: Pretrained BanglaBERT outperforms the scratch-trained models, whereas BanglaMamba performs comparably to a similarly scaled scratch-trained Transformer.The comparison separates the apparent benefit of large-scale pretraining from differences between the underlying architectures.
  • Computational Efficiency Comparison: 2.2× higher throughput and 2.5× lower P50 latency make BanglaMamba more efficient than BanglaBERT during inference.BanglaMamba also achieves approximately 2.2× lower P95 latency, although throughput varies more across runs.
  • Computational Efficiency Comparison: 49% less peak VRAM during inference and nearly 78% less during training distinguish BanglaMamba from the BERT-based models.CustomBERT uses the same architecture and configuration as BanglaBERT, so their efficiency metrics are effectively identical; the reported gains are consistent with linear-time sequence processing.
  • Input Length Robustness Analysis: With inputs capped at 512 tokens, performance differences between short and long articles remain relatively small across the three models.BanglaBERT shows a somewhat larger decrease, while BanglaMamba and CustomBERT exhibit smaller changes; CustomBERT slightly improves on long-article Macro-F1 and Fake F1.
  • Cross-Dataset Generalization: BanglaBERT achieves the strongest external-dataset performance, followed by CustomBERT and BanglaMamba, while BanglaMamba shows lower variability on several metrics.The weaker out-of-domain performance of the scratch-trained models may reflect the absence of large-scale pretraining, though architectural and optimization differences may also contribute.
  • Ablation Study: BanglaBERT remains broadly comparable when using masked mean pooling instead of [CLS], although [CLS] pooling achieves slightly better overall results.BanglaMamba uses masked mean pooling because its causal sequence processing differs fundamentally from BERT-style bidirectional self-attention.

5 LIMITATIONS AND FUTURE WORK

The study is limited by class imbalance and by training BanglaMamba only on the downstream dataset, without large-scale language pretraining. Future work should pretrain BanglaMamba on diverse Bangla text and evaluate it across multiple datasets.

  • Class imbalance in BanFakeNews-2.0 limits evaluation, with fake-news identification remaining harder than real-news identification.Both BanglaBERT and BanglaMamba perform worse on the minority Fake class than on the majority Real class.
  • Training BanglaMamba from scratch without large-scale language pretraining constrains its out-of-domain generalization.On an external dataset, BanglaMamba performs substantially worse than BanglaBERT, although architectural and optimization differences may also contribute.
  • Future work should pretrain BanglaMamba on diverse Bangla text and evaluate it across multiple Bangla fake news datasets.These experiments would provide a stronger assessment of pretrained Mamba-based models’ generalization capability.

6 CONCLUSION

The study finds that BanglaMamba offers competitive in-domain performance with lower inference cost than Transformer-based models, while BanglaBERT generalizes better across datasets. The results support further investigation of pretrained Mamba models for Bangla fake news detection.

  • BanglaMamba achieves competitive in-domain results while providing lower inference latency, higher throughput, and lower GPU memory usage than the compared BERT models.BanglaBERT still achieves the highest in-domain classification performance.
  • Similar in-domain performance between BanglaMamba and CustomBERT suggests that training from scratch can produce competitive classification performance without large-scale language pretraining.
  • Cross-dataset evaluation shows weaker generalization for both models trained from scratch, with BanglaMamba showing the largest performance degradation.The gap cannot be attributed solely to model architecture because architectural and optimization differences may also contribute.
  • Mamba-based SSMs are presented as a promising and computationally efficient alternative to Transformer-based models for Bangla fake news detection.Future work should investigate large-scale pretraining and evaluation across multiple Bangla fake news datasets.

AI USAGE STATEMENT

The authors used AI-assisted tools for writing refinement and code development, while reporting that experiments, findings, interpretations, analyses, and conclusions were produced by the authors.

  • AI tools were used for grammatical correction, language polishing, and assistance with code development.
  • The authors state that all experimental data, results, interpretations, analyses, and conclusions were produced through their own experiments and pipelines.

A EXPERIMENTAL SETUP

The appendix presents the experimental setup, BanglaMamba configuration, and training configurations for the compared models. BanglaMamba uses the official mamba-ssm defaults for its Mamba-specific parameters.

  • Experimental setup: The appendix includes an experimental setup table for the study.
  • Model and training configurations: The BanglaMamba configuration is documented separately from the general experimental setup and training configurations.
  • BanglaMamba configuration: BanglaMamba retains official mamba-ssm defaults, including dstate=16, dconv=4, expand=2, automatic Δrank, use bias=False, and conv bias=True.The automatic Δrank is calculated as ⌈dmodel/16⌉.

C.1 TRAINING DYNAMICS

Across three random seeds, the models generally reached their best validation performance within the first few epochs, with limited or inconsistent gains afterward. BanglaBERT improved more gradually than BanglaMamba and CustomBERT, supporting five training epochs for the reported experiments.

  • All three models generally reached their best validation performance within the first few epochs.
  • BanglaBERT improved more gradually than BanglaMamba and CustomBERT.
  • Later epochs provided limited or inconsistent validation improvement across the models.
  • Five training epochs were used for the reported experiments based on these training dynamics.

D.1 BANGLABERT RESULTS

The supplied tables organize classification and efficiency results across three random seeds for BanglaBERT, BanglaMamba, and CustomBERT.

  • BanglaBERT classification performance is reported across three random seeds.
  • BanglaBERT efficiency metrics are reported across three random seeds.
  • BanglaMamba classification and efficiency metrics are reported across three random seeds.
  • CustomBERT classification and efficiency metrics are reported across three random seeds.

E DATA INTEGRITY AND LEAKAGE PREVENTION

The study applies preprocessing, deduplication, partitioning, tokenizer, class-weight, and checkpoint-selection safeguards to reduce data leakage risks.

  • The corpus was cleaned and deduplicated before a fixed-seed stratified 80/10/10 train–validation–test split.
  • Articles were normalized and exact duplicates were removed using complete normalized headline–body representations.
  • Articles shorter than 20 or longer than 2,000 words were removed before dataset partitioning.
  • The pretrained tokenizer was unchanged, class weights used training data only, and checkpoints were selected solely by validation Macro-F1.
  • These procedures safeguard against duplicate, tokenizer-fitting, label-statistic, and test-set-driven model-selection leakage.
Loading 2608.25190v1…