PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Neural Complexity Improve Health Misinformation Detection? A Leakage-Controlled Cross-Corpus Benchmark

Mkululi SIKOSANA

arXiv:2610.04636v1cs.CLcs.LG

TL;DR

Architecture comparisons are difficult to interpret when corpora, preprocessing, splits, and leakage controls differ. This study benchmarks compact neural architectures, an ensemble, and classical baselines under controlled cross-corpus conditions. Rankings changed across datasets, and on CONSTRAINT the linear SVM outperformed the neural ensemble.

  • Problem

    Comparisons of health-misinformation models are difficult to interpret when studies use different corpora, preprocessing pipelines, splits, and leakage controls.

    Source: §1 · p. 2

  • Method

    The study compares five neural architectures, a soft-voting ensemble, and classical baselines across two corpora using controlled protocols and repeated neural runs.

    Source: §1 · p. 3, §4 · p. 8, §8 · p. 15

  • Results

    Model rankings changed across corpora; on CONSTRAINT, linear SVM macro-F1 was 0.9574 versus 0.9272 for the deep ensemble.

    Source: §6 · p. 14, §8 · p. 15

  • Takeaways & Limitations

    Model complexity is an empirical question rather than a quality label, so benchmark performance should be interpreted cautiously and strong classical baselines retained.

    Source: §8 · pp. 15–16

    Both datasets concern COVID-19 and do not establish generalisation to later outbreaks, other health domains, or non-health misinformation.

    Source: §7 · p. 14

Abstract

from arXiv · show

Increasing architectural complexity is often assumed to improve health misinformation detection, yet reported gains are difficult to interpret when studies use different corpora, preprocessing pipelines, data splits, and leakage controls. This study provides a controlled cross-corpus benchmark of five compact neural architectures (1D-CNN, LSTM, BiLSTM, CNN-LSTM, and CNN-BiLSTM), a soft-voting neural ensemble, and three classical machine-learning baselines using COVID19-FNIR and CONSTRAINT. Exact-text duplicate controls were applied before modelling; all neural systems used a common preprocessing and optimisation protocol, and neural results were repeated across three random seeds. On COVID19-FNIR, the deep ensemble achieved a mean macro-F1 of 0.9963 and ROC-AUC of 0.9994, while individual neural models ranged from 0.9945 to 0.9957 macro-F1. On CONSTRAINT, the ensemble achieved macro-F1 of 0.9272 and ROC-AUC of 0.9811, whereas a linear SVM achieved macro-F1 of 0.9574 and ROC-AUC of 0.9931. Architecture rankings changed across corpora, and simple sparse linear models remained highly competitive. The findings show that model complexity does not provide a stable performance advantage and that benchmark construction can dominate architecture choice. The study contributes a reproducible, leakage-controlled basis for evidence-driven model selection in health misinformation classification

1 Introduction

The introduction argues that architecture comparisons are hard to interpret across inconsistent datasets and pipelines, then frames a controlled two-corpus test of whether added neural complexity improves detection. Its results show that model rankings depend on the corpus and that a linear SVM can outperform the neural ensemble on CONSTRAINT.

1 Introduction

  • Architecture comparisons across datasets, preprocessing pipelines, and splits are difficult to interpret, while exact duplicates crossing train-test boundaries can inflate performance estimates.Dataset construction and annotation policy also affect fake-news detection performance.Source: §1 · p. 2
  • The study asks whether more complex neural architectures outperform simpler ones when the data pipeline is held constant, rather than assuming their representational capacity ensures gains.CNNs target local phrase patterns, while LSTMs model longer dependencies and bidirectional recurrence uses both sequence directions.Source: §1 · p. 2
  • The benchmark compares five neural architectures, a soft-voting ensemble, and three classical baselines on COVID19-FNIR and CONSTRAINT, whose construction processes differ.The two-corpus design tests ranking stability, not universal generalisation beyond COVID-era datasets.Source: §1 · p. 3
  • The experiments use compact models, fixed seeds, transparent preprocessing, and CPU-only execution to support reproducibility without requiring large pretrained language models.The paper positions this as a counterpoint to the idea that publication-quality detection necessarily requires very large pretrained models.Source: §1 · p. 3
  • All model families achieved high performance, but the most complex neural model was not consistently best, and linear SVM outperformed the deep ensemble on CONSTRAINT.The study presents this as a challenge to treating complexity as a proxy for quality.Source: §1 · p. 3

2 Related Work

Prior work spans sparse classifiers, neural architectures, and pretrained Transformers, with strong performance reported across these families. This section motivates controlled comparisons because corpus construction, lexical cues, and leakage can shape apparent model performance.

2 Related Work

  • Earlier sparse classifiers can perform strongly when labels correlate with recurring lexical patterns or source conventions, and linear SVMs have remained competitive on CONSTRAINT.A prior CONSTRAINT system reported weighted F1 of 95.19% using a linear SVM [9].Source: §2 · pp. 3–4
  • CNN filters can capture discriminative local phrases, but that same bias may lead them to learn dataset artefacts rather than transferable veracity cues.Global max pooling retains salient activations from feature maps.Source: §2 · p. 4
  • LSTMs model longer dependencies, bidirectional recurrence incorporates both contexts, and CNN-recurrent hybrids combine local and sequential features without guaranteeing better performance.Greater flexibility also increases parameter count and optimisation complexity; prior health-misinformation work reports strong CNN-LSTM results and psychological-feature integration.Source: §2 · p. 4
  • Pretrained Transformer models dominate much of the current NLP benchmark landscape, but comparisons require aligned data splits and contamination controls.Distilled models aim to retain substantial language-model capability in smaller systems [13].Source: §2 · p. 4
  • Sparse baselines remain important because they are interpretable, inexpensive, and can be difficult to beat when datasets have strong lexical separability.A simple model matching or exceeding a more expensive one changes the practical interpretation of the benchmark.Source: §2 · p. 5
  • Health-misinformation corpora differ in style and construction, and exact duplicate controls alone do not establish cross-domain transfer or remove benchmark-specific predictability.The study addresses whether model rankings persist across two differently constructed corpora, not full out-of-domain transfer.Source: §2 · pp. 5–6

3 Research Questions and Expectations

The research questions test whether compact neural models perform strongly, whether ensembling helps, whether neural complexity beats classical baselines, and whether rankings hold across two corpora.

3 Research Questions and Expectations

  • RQ1 tests whether five neural architectures achieve strong classification performance under a shared leakage-controlled protocol, with an expectation of macro-F1 above 0.90 on both datasets.The stated expectation was that FNIR would be easier than CONSTRAINT.Source: §3 · p. 6
  • RQ2 tests whether soft voting improves on individual networks, based on the expectation that partially uncorrelated errors could make the ensemble among the strongest deep systems.Source: §3 · p. 6
  • RQ3 tests whether greater neural complexity consistently beats classical sparse baselines, without imposing a directional hypothesis because linear SVMs can be strong on CONSTRAINT.Source: §3 · p. 6
  • RQ4 tests whether architecture rankings remain stable across COVID19-FNIR and CONSTRAINT, with changing rankings indicating dataset-contingent model quality.The question cautions against choosing architectures based on a single benchmark.Source: §3 · p. 6

4 Materials and Methods

The methods apply exact-text duplicate controls and a common lightweight neural pipeline across two COVID-19 corpora, alongside classical TF-IDF baselines. Neural results are repeated across three seeds and evaluated with multiple classification and ranking metrics.

4 Materials and Methods

  • COVID19-FNIR contained 7,249 unique records after exact duplicate removal and was split by seed into 5,074 training, 1,087 validation, and 1,088 test examples.The records comprised 3,459 misinformation and 3,790 reliable items; vocabulary was built from training data only.Source: §4 · p. 6, Table 1 · p. 7
  • CONSTRAINT retained its native partitions after exact-text separation, leaving 6,417 training, 2,140 validation, and 2,140 test records.The procedure removed overlapping normalised texts across partitions while preserving the native shared-task structure.Source: §4 · pp. 6–7
  • Neural inputs used lowercased, lightweight tokenisation, a training-only vocabulary capped at 25,000 tokens, and sequences truncated or padded to 96 tokens.No pretrained word vectors or language-model weights were used.Source: §4 · p. 7
  • The 1D-CNN used trainable 64-dimensional embeddings, parallel width-3, 4, and 5 convolutions, global max pooling, dropout, and a binary classifier.Its task-specific embeddings were randomly initialised and learned from benchmark data.Source: §4 · p. 7
  • LSTM and BiLSTM models pooled recurrent representations, while CNN-LSTM and CNN-BiLSTM combined local convolutional features with recurrent sequence representations.The hybrid recurrent layers had 48 units, and their pooled convolutional and recurrent representations were concatenated before classification.Source: §4 · pp. 7–8
  • The soft-voting ensemble averaged misinformation probabilities from five neural architecture families within each seed, without fitting a secondary meta-learner.This design tests whether architectural diversity benefits performance while avoiding a separately trained ensemble stage.Source: §4 · p. 8
  • Classical baselines used combined word and character TF-IDF features; logistic regression and linear SVM operated on the sparse representation, while random forest used a 64-dimensional projection.The combined feature space was capped at 40,000 features, and the settings kept the benchmark executable on CPU hardware.Source: §4 · p. 8
  • Neural models used AdamW, weighted binary cross-entropy, validation ROC-AUC checkpoint selection, and three fixed seeds; evaluation reported seven metrics with means and standard deviations.Macro-F1 weighted both classes equally; with three runs, standard deviations were treated descriptively rather than as strong inferential evidence. The reproducibility package records pipeline code and results, and pretrained Transformers were excluded from the primary benchmark.Source: §4 · pp. 8–9

5 Results

All five neural architecture families performed strongly across the two corpora, and the deep soft-voting ensemble provided the strongest neural performance overall.

5 Results

  • The deep soft-voting ensemble provided the strongest neural performance overall across the two corpora.Table 2 reports deep-learning performance averaged across seeds 41, 42 and 43.Source: Table 2 · p. 9, §5 · p. 9
  • For FNIR, the reported values were 0.9945 for BiLSTM and 0.9957 for CNN-LSTM; the passages also report 0.9948 for 1D-CNN and 0.9951 for LSTM.Source: §5 · p. 9
  • On CONSTRAINT, the reported values ranged from 0.9058 for LSTM to 0.9169 for 1D-CNN among individual neural models.BiLSTM was 0.9163 and CNN-LSTM was 0.9093.Source: §5 · p. 9

ROC-AUC mean

The comparison across datasets shows no stable architecture ranking: sparse linear baselines remain highly competitive and outperform the neural ensemble on CONSTRAINT.

ROC-AUC mean

  • Linear SVM achieved macro-F1 0.9574 and ROC-AUC 0.9931 on CONSTRAINT, exceeding the deep ensemble by approximately 3.02 macro-F1 percentage points.Table 3 presents classical TF-IDF baselines.Source: Table 3 · p. 10, ROC-AUC mean · p. 11
  • On FNIR, linear SVM and logistic regression each averaged macro-F1 0.9966, essentially matching the deep ensemble.Their ROC-AUC values were approximately 1.0000 and 0.99998, respectively.Source: ROC-AUC mean · pp. 10–11
  • CNN-LSTM led individual neural models on FNIR but ranked near the bottom on CONSTRAINT, while 1D-CNN was competitive on both.Architecture rankings were not stable across datasets.Source: ROC-AUC mean · p. 11
  • The CONSTRAINT ensemble’s macro-F1 ranged from approximately 0.9246 to 0.9287 across seeds, while individual CNN-BiLSTM runs varied more.Its macro-F1 standard deviation was about 0.0109.Source: ROC-AUC mean · p. 11

6 Discussion

The discussion finds no stable advantage for greater neural complexity: rankings vary by corpus, and classical linear models can outperform neural systems. It frames high FNIR scores as benchmark-specific and emphasizes empirical justification, reproducibility, and deployment safeguards.

6 Discussion

  • Model complexity should be justified by empirical gain rather than layer count or architectural popularity, especially when benchmarks approach saturation.The discussion points toward generalisation, calibration, robustness, uncertainty, and resource efficiency as more informative research questions.Source: §6 · pp. 12–13
  • On CONSTRAINT, the linear SVM’s macro-F1 of 0.9574 is plausible alongside the neural results, as sparse lexical patterns can suit short social-media posts.Character n-grams can also accommodate spelling variation, hashtags, and morphological changes.Source: §6 · p. 12
  • COVID19-FNIR’s different sources for true and false content may create lexical or stylistic separation useful for classification without establishing general veracity reasoning.The supported conclusion is that FNIR is highly predictable under this protocol.Source: §6 · p. 12
  • On CONSTRAINT, averaging five neural probability estimates yields macro-F1 of 0.9272, but the ensemble still does not outperform the SVM.Architectural diversity can help when several models can run in parallel, but does not guarantee the best overall result.Source: §6 · pp. 12–13
  • CNN-LSTM ranks highly on FNIR but is not the strongest neural model on CONSTRAINT, so architecture rankings depend on dataset composition.The same algorithm can shift several ranking positions without an implementation change.Source: §6 · pp. 11, 13
  • Operational screening requires calibration, threshold selection, human review, and governance; the models should not be treated as autonomous arbiters of truth.Accuracy and macro-F1 summarize errors symmetrically, while real applications may assign different costs to false positives and false negatives.Source: §6 · p. 14
  • The executable CPU-based benchmark makes architecture definitions, duplicate checks, fixed seeds, and classical baselines inspectable and repeatable.The paper presents transparency as a contribution, not as evidence of methodological superiority to pretrained Transformers.Source: §6 · p. 14
  • On CONSTRAINT, mean macro-F1 above 0.91 for deep models coexists with neural ensembling gains and stronger classical linear baselines; near-ceiling FNIR results are benchmark-specific.The authors caution that FNIR predictability is not proof of general misinformation understanding.Source: §6 · p. 14

7 Limitations and Future Work

The study’s conclusions are bounded by two COVID-19 datasets, task-trained embeddings, exact-duplicate controls, and a compact evaluation protocol. It assesses predictive discrimination, not explanations or causal mechanisms.

7 Limitations and Future Work

  • Both datasets concern COVID-19, so they do not establish generalisation to later outbreaks, other health domains, or non-health misinformation.The authors suggest adding other health and general-misinformation datasets and evaluating leave-one-dataset-out transfer, with careful label harmonisation.Source: §7 · p. 14
  • The neural models learn embeddings from task data rather than pretrained language models, so the paper does not claim state-of-the-art Transformer performance.This choice supports reproducible architecture comparisons and isolates task-level learning; future extensions could evaluate named pretrained models under the same duplicate-controlled protocol.Source: §7 · pp. 14–15
  • Exact-duplicate removal leaves near duplicates, paraphrases, shared claim templates, and source signatures uncontrolled, particularly warranting further scrutiny of FNIR’s near-ceiling results.The authors suggest semantic-similarity clustering as a stronger leakage audit.Source: §7 · p. 15
  • Only three neural seeds were run, and compact rather than extensively tuned hyperparameters mean rankings describe these implementations, not each architecture family’s theoretical maximum.More repetitions could improve model-variance estimates; broader model-specific searches could change rankings but reduce computational symmetry.Source: §7 · p. 15
  • Predictive discrimination does not explain why people believe or share messages; the authors call for separate work on error analysis, interpretability, and psychologically grounded variables.Text-derived proxies should not be treated as direct measurements of cognition.Source: §7 · p. 15

8 Conclusion

The benchmark finds very high neural performance on COVID19-FNIR, but no stable advantage from increasing architectural complexity across the two corpora. Its conclusion is to treat model choice empirically and retain strong classical baselines while interpreting high scores cautiously.

8 Conclusion

  • The deep ensemble achieved mean macro-F1 of 0.9963 on COVID19-FNIR and 0.9272 on CONSTRAINT, while linear SVM remained strongest on CONSTRAINT at macro-F1 0.9574; architectural ordering was unstable.Source: §8 · p. 15
  • Increasing architectural complexity did not yield a stable ordering, and model complexity is an empirical question rather than a quality label.Strong benchmark performance can arise from CNNs, recurrent networks, hybrids, or sparse linear representations depending on dataset construction.Source: §8 · p. 15
  • The authors recommend retaining strong classical baselines, reporting repeated runs, controlling duplicate leakage, and interpreting near-ceiling scores with caution.These practices are presented as a more credible foundation for future work on domain generalisation, uncertainty, psychological feature fusion, and efficient deployment.Source: §8 · pp. 15–16

A Exact Experimental Configuration

The configuration specifies compact neural-model settings and TF-IDF feature ranges, with source code documenting architecture-specific dimensions and dropout.

A Exact Experimental Configuration

  • Neural models used a 25,000-token vocabulary, minimum token frequency 2, sequence length 96, 64-dimensional embeddings, batch size 128, and two epochs.Optimization used AdamW at 2e-3 learning rate, weight decay 1e-4, gradient clipping 1.0, validation ROC-AUC checkpointing, and CPU execution.Source: A Exact Experimental Configuration · p. 16
  • Neural results used seeds 41, 42, and 43; classical word TF-IDF used 1-2 grams and character-within-word TF-IDF used 3-5 grams.Source: A Exact Experimental Configuration · p. 16
  • The full source code records architecture-specific hidden dimensions and dropout settings.Source: A Exact Experimental Configuration · p. 16

Statements and Declarations

The statements report no funding or relevant competing interests, describe the secondary-data ethics approach and access conditions, and disclose AI-assisted editing and code review.

Funding

Competing interests

Ethics statement

  • The study used secondary public or curated content, recruited no participants, and neither identified private individuals nor inferred protected characteristics.Source: Ethics statement · p. 16

Data availability

  • Access to COVID19-FNIR and CONSTRAINT records remains subject to provider conditions; the package requires locally authorised copies and does not redistribute platform content.Source: Data availability · p. 16

Declaration of generative AI and AI-assisted technologies

  • ChatGPT assisted with language editing and code review, but not experimental observations or numerical results; the author independently checked the analyses and takes responsibility.Source: Declaration of generative AI and… · p. 17

Related papers

Loading 2610.04636v1…