PaperlayerView exact revision on arXiv

Source-linked AI summary

BharatGather: A Culturally-Informed Benchmark Dataset for Misinformation and Fake News Detection in Indian Public Events

Parth Bramhecha, Smit Deshmukh, Sairaj Bodhale, Adwait Borate, Raviraj Joshi

arXiv:2609.02895v1cs.CLcs.LG

TL;DR

Existing misinformation benchmarks often lack the cultural and linguistic granularity needed for Indian mass gatherings. Bharat-Gather introduces a 14,646-record benchmark built from fact-checking sources, multimedia transcripts, and LLM-generated synthetic examples; its evaluation reports 96.1% accuracy and a macro F1-score of 0.960.

  • Problem

    Existing research largely relies on domain-general or Western-centric datasets that lack cultural and linguistic granularity for Indian event-driven misinformation detection.

  • Method

    Bharat-Gather constructs a binary True/False misinformation dataset from fact-checking websites, news-video transcripts, and controlled LLM-based synthetic augmentation.

  • Results

    96.1% accuracy and a macro F1-score of 0.960 were reported for classification on the dataset.

  • Takeaways & Limitations

    The dataset provides a benchmark for studying misinformation detection in the culturally specific context of Indian mass gatherings.

  • Takeaways & Limitations

    The dataset is predominantly English-language, limiting ecological validity because misinformation also proliferates in Hindi and regional vernaculars.

Abstract

from arXiv · show

Large-scale public events, such as religious festivals, political rallies, and cultural gatherings, are increasingly vulnerable to the rapid dissemination of misinformation, posing substantial risks to public safety and social cohesion. While automated fake news detection has seen significant methodological progress, existing benchmarks frequently fail to capture the socio-cultural nuances and event-specific dynamics characteristic of the Indian context. This paper introduces BharatGather, a curated, multi-source dataset specifically engineered for binary misinformation classification within the ecosystem of Indian mass gatherings. The corpus comprises 14,646 records constructed through a hybrid pipeline involving systematic web scraping of prominent fact-checking platforms, multimedia transcript extraction, and Large Language Model (LLM)-mediated synthetic augmentation to ensure narrative diversity. By providing a resource tailored to the unique complexities of event-aware misinformation in India, this work facilitates the development of culturally informed detection systems and establishes a rigorous benchmark for evaluating their performance in high-stakes public environments.

1 Introduction

BharatGather addresses the lack of culturally and linguistically granular, event-specific Indian misinformation datasets by introducing a curated binary classification benchmark for mass gatherings. Its 14,646 records combine multiple acquisition modalities, synthetic augmentation, and event-aware metadata.

  • Indian mass gatherings are vulnerable to rapidly diffusing false or misleading claims across social media, broadcast news, and video-sharing platforms.
  • Existing misinformation research often relies on domain-general or Western-centric datasets that lack Indian cultural and linguistic granularity.
  • BharatGather provides binary True and False labels for misinformation detection in the specific context of Indian mass gatherings.
  • 14,646 records combine web-scraped fact-checking data, multimedia transcripts, LLM-mediated synthetic augmentation, veracity labels, and event-aware metadata.
  • 3,595 synthetic variants were generated with Qwen3-32B to expose classifiers to subtle deceptive patterns beyond surface-level lexical analysis.

2 Related Work

Prior work approaches misinformation through computational, behavioral, communication, and intervention perspectives, but Indian event-specific benchmarks remain scarce. Existing studies motivate culturally grounded datasets and complementary detection and mitigation strategies.

  • Transformer-based misinformation detection has advanced, but surveys identify persistent limitations in data quality, contextual bias, and domain transfer.
  • Large-scale event-specific misinformation benchmarks in India remain scarce despite practical feasibility demonstrated by NLP-based matching systems.
  • Indian misinformation analyses identify recurring narratives involving health claims, political manipulation, religious polarization, and communal framing.
  • Research also models temporal diffusion, influential users, statement-level conflict, and behavioral forensics for misinformation identification.
  • Complementary mitigation approaches include crowd judgments, synthetic crowd modeling, staged debunker selection, and prebunking.
  • Crisis communication and information-behavior research emphasizes monitoring gaps, dissemination stages, platform vulnerabilities, and timely context-sensitive detection.

3 Dataset Construction

BharatGather is constructed as a culturally grounded, three-tier corpus covering Indian mass gatherings through fact-checking platforms, video transcripts, and controlled LLM augmentation. The pipeline standardizes heterogeneous sources while preserving high-integrity provenance and generating diverse synthetic narratives.

  • Dataset Scope: The corpus targets misinformation surrounding Indian religious festivals, political rallies, and regional cultural events, with each instance anchored in verified reporting.
  • Source Modalities: Five Indian fact-checking organizations provide structured, high-integrity sources containing verified claims and corresponding fact-checked responses.
  • Tier I Acquisition: 10,691 records were obtained through systematic scraping designed for heterogeneous HTML architectures.
  • Tier I Acquisition: 2,368 native fact-claim records from BoomLive, Factly, and NewsMeter formed the highest-integrity tier and exclusive source for synthetic augmentation.
  • Tier I Acquisition: 8,323 AltNews and The Quint records were normalized with Qwen3-32B extraction and excluded from augmentation to preserve provenance boundaries and avoid recursive model bias.
  • Source Modalities: 360 records come from authenticated Indian news-channel transcripts, capturing video-mediated misinformation while introducing inter-channel near-duplicate redundancy.
  • Tier III Augmentation: 3,595 synthetic records were generated from high-integrity records using true and adversarial false variants with bounded lexical overlap checks.

3.4 Data Labeling

BharatGather assigns binary labels through directional semantic matching between extracted claims and verified facts, while tier-specific curation determines the training text used by downstream models. The labeling process also deterministically labels synthetic true and false variants.

  • Claims semantically aligned with verified facts receive label 1, whereas contradictory or materially divergent claims receive label 0.
  • The training_text field is the primary downstream NLP input and is curated according to each acquisition tier’s structural characteristics.
  • Tier I long-form articles are distilled into separate true Fact and false Claim training records to focus learning on core semantic meaning.
  • Tier II transcript summaries are universally labeled true because the videos represent verified information.
  • Tier III augmentation creates false claim records and true fact records from paraphrased native data, supporting consistency across phrasing styles.

3.5 Dataset Schema

The dataset uses a standardized schema to preserve record identity, provenance, labels, source context, and the primary NLP input across acquisition tiers. Its 14,646 records are distributed across three tiers, with Tier I drawing on multiple fact-checking platforms.

  • Schema design: Each record is mapped to a standardized schema that preserves provenance and linguistic context across acquisition tiers.The schema contains eight described fields, including tracking, source, label, model input, full context, verification link, and content type.
  • Schema design: The label field uses 1 for verified truth and 0 for misinformation.
  • Schema design: The training_text field is the primary NLP input, while full_article stores the complete source text or transcript for additional context.
  • Schema design: The url field supports independent verification and auditability, and content_type distinguishes fact-checked, factual-data, and generated records.
  • Dataset organization: 14,646 records are distributed across three acquisition tiers, with Table 1 summarizing the tier-level distribution.

4 Experiments

Experiments evaluate BERT for binary classification on a deduplicated BharatGather corpus, comparing frozen feature extraction with encoder fine-tuning. Fine-tuning substantially improves test performance and produces balanced classification errors across misinformation and true instances.

  • Experimental setup: The corpus was deduplicated using training_text before experiments, and BERT was fine-tuned for binary sequence classification.
  • Experimental setup: Two regimes were compared: a frozen BERT encoder with a trained classification head and full encoder fine-tuning with the classification head.
  • Experimental setup: Both regimes used maximum sequence length 256, batch size 16, learning rate 3 × 10−5, and validation macro F1 for checkpoint selection.
  • Results: 77.3% test accuracy and 0.769 macro F1 were achieved with static BERT embeddings and a trainable classification head.
  • Results: 96.1% test accuracy, 0.960 macro F1, and 0.961 weighted F1 were achieved when encoder layers and the classification head adapted to the domain.
  • Results: The fine-tuned configuration correctly identified 844 of 879 misinformation instances and 1,153 of 1,199 true instances.
  • Results: Validation macro F1 reached 0.975, corresponding to an approximately 1.5-percentage-point gap to the held-out test set.

5 Discussion and Challenges

Fine-tuned BERT substantially outperforms frozen feature extraction, while adversarial misinformation remains difficult to distinguish and external generalization is unresolved. The benchmarks support domain-specific adaptation but motivate multilingual evaluation and testing on real-world event misinformation.

  • Lexical Signaling and Classifier Behavior: Fine-tuning raises macro F1 from 0.380 with frozen BERT to 0.960, showing that deep task adaptation is needed beyond lexical cues.The frozen encoder achieved accuracy 0.586, whereas full fine-tuning produced the higher macro F1.
  • Class Asymmetry and Adversarial Robustness: The frozen model predicts the majority class for nearly all instances and correctly identifies only 10 misinformation cases.This performance collapse reveals greater predictive difficulty for the misinformation class, especially adversarial variants.
  • Generalization and Corpus Provenance: Validation and test macro F1 scores of 0.975 and 0.960 indicate a marginal internal generalization gap.The corpus’s heterogeneous provenance still creates uncertainty about transfer to in situ misinformation during Indian mass gatherings.
  • Generalization and Corpus Provenance: The benchmark provides a reproducible foundation for event-aware misinformation research while showing that transformer performance depends on domain-specific adaptation.Future work should incorporate multilingual encoders for India’s linguistic diversity.

6 Conclusion

The paper introduces BharatGather, a culturally informed benchmark of Indian public-event misinformation built from structured web, video, and LLM-augmented sources. Fine-tuned BERT performs strongly, supporting regional and event-specific training while positioning the dataset for future detection research.

  • Dataset Contribution: BharatGather contains 14,646 records assembled through a structured three-tier process for misinformation detection during large Indian public events.The corpus combines fact-checking websites, news-video text, and LLM-generated synthetic examples in a consistent ten-attribute format.
  • Dataset Contribution: The dataset combines five Indian fact-checking websites, news-video transcripts, and challenging synthetic examples to represent culturally specific misinformation.This multi-source construction provides a benchmark focused on regional and event-based data.
  • Evaluation: 96.1% accuracy and a macro F1-score of 0.960 are achieved by fine-tuning bert-base-uncased on the dataset.The reported result follows limitations in the basic model and demonstrates strong performance after task-specific training.
  • Evaluation: Synthetic fake examples are difficult for models to guess easily, strengthening the dataset’s intended role as a research benchmark.The conclusion presents these examples as useful for future misinformation-detection research.

Limitations

BharatGather’s coverage and validity are bounded by source selection, LLM extraction noise, transcript redundancy, English-language predominance, and a fixed temporal snapshot. These constraints may limit cultural generalizability, ecological validity, and robustness to evolving or closely matched misinformation.

  • Scope of Event Representation: Editorial priorities of five fact-checking platforms may underrepresent localized regional festivals and limit cultural generalizability.Events with limited media coverage are especially vulnerable to underrepresentation.
  • LLM-Mediated Extraction Noise: Qwen3-32B extraction from unstructured sources may introduce approximation errors and residual noise into fact-claim pairs.Excluding these records from synthetic augmentation mitigates potential model collapse but does not remove extraction noise.
  • Inter-Channel Transcript Redundancy: Near-duplicate Tier II transcripts may remain despite deduplication, creating a latent risk of data leakage across training and evaluation partitions.Identical events are often reported by multiple news outlets.
  • Linguistic Granularity: The predominantly English-language dataset limits ecological validity where misinformation spreads in Hindi and regional vernaculars.The missing multilingual coverage is a deployment boundary for Indian mass-gathering settings.
  • Temporal and Semantic Separation: A fixed temporal snapshot may degrade on evolving narratives, while strong lexical and semantic separation leaves future benchmarks needing closer true-false proximity.Increasing class proximity is proposed as a way to create more rigorous transformer evaluations.

Data Provenance and Source Metadata

The dataset’s provenance is documented through five named fact-checking domains and a separate collection of verified YouTube channels. These sources cover Indian festivals, protests, public gatherings, and misinformation narratives, while a repository provides a catalog of collected topics and categories.

  • Fact-Checking Sources: The primary fact-checking sources are AltNews, BoomLive, Factly, The Quint, and NewsMeter.Their domains are listed for auditability and transparency.
  • Multimedia Sources: Tier II transcripts come from verified news, digital-media, public-service, agency, and educational YouTube channels.The listed channels support multimedia acquisition for Indian public-event coverage.
  • Multimedia Sources: The source collection covers cultural festivals, political protests, public gatherings, and misinformation narratives across India.The paper identifies this coverage as part of the Tier II multimedia acquisition pipeline.
  • Dataset Metadata: A repository provides a detailed catalog of the topics, events, and thematic categories identified during web and multimedia acquisition.The catalog encompasses narratives centered on Indian mass gatherings and public events.
Loading 2609.02895v1…