Johnny Still Receives Spam SMS: Assessing the Robustness of SMS Spam Detection

Muhammad Salman; Muhammad Islam; Muhammad Ikram; Mohamed Ali Kaafar

Source-linked AI summary

Johnny Still Receives Spam SMS: Assessing the Robustness of SMS Spam Detection

Muhammad Salman, Muhammad Islam, Muhammad Ikram, Mohamed Ali Kaafar

arXiv:2609.01171v1Sep 1, 2026cs.CR

TL;DR

SMS spam detectors can perform well in controlled settings yet remain unreliable and vulnerable under real-world adversarial manipulation. The paper evaluates deployed and open-weight systems with deliverable attacks, measures held-out generalization, and tests diverse ensembles. It finds that adversarial training transfers mainly within attack families, while minority-voting ensembles improve robustness across attack types.

Problem
SMS anti-spam systems used by end users remain unreliable under benign conditions and vulnerable to adversarial manipulation, including imperceptible perturbations.
Method
The paper evaluates deployed messaging apps, third-party services, and open-weight detectors, then compares ensemble decision rules using diverse detectors and adversarial training.
Results
Adversarial training improves resilience but is class-bounded, while an architecturally diverse ensemble improves robustness against known and unknown attacks.
Takeaways & Limitations
Robust SMS spam detection requires evaluation beyond controlled settings and defenses that combine adversarial training with model diversity.
Takeaways & Limitations
The real-world evaluation used one SIM card on one device in one country, constraining claims about uniform behavior across providers.

Abstract

from arXiv · show

SMS spam detection systems often achieve high accuracy in controlled environments but struggle against adversarial attacks and increasingly sophisticated spam tactics in real-world deployments. In this paper, we evaluate the robustness of SMS anti-spam systems that end users actually rely on, including commercial messaging applications, third-party anti-spam services, and publicly available open-weight models hosted on Hugging Face. We evaluate these systems under both standard and adversarial conditions, considering perceptible and state-of-the-art imperceptible attacks. We include only perturbations that we verify survive real SMS or RCS delivery, rather than lab-only artifacts. Our experiments reveal significant gaps in existing spam detectors' ability to identify adversarially manipulated messages. We further demonstrate that adversarial training alone is insufficient. Using an explicit held-out evaluation protocol, we find that robustness transfers well within a perturbation family but degrades sharply against structurally distinct, encoding-level attacks. To address these weaknesses, we propose a multi-model ensemble that combines adversarial training with spam classifiers diverse in architecture and tokenization. Our results show that this ensemble, particularly when using a minority-voting strategy, substantially improves robustness against both perceptible and imperceptible adversarial attacks while maintaining competitive classification accuracy. We also characterize the resulting precision-recall trade-off and recommend operating points for false-positive-sensitive and recall-critical deployments. These findings highlight the need for comprehensive robustness evaluations and ensemble-based defenses for building more secure SMS spam detection systems in real-world settings.

1 Introduction

SMS spam detectors achieve high benign-data performance yet remain insufficiently robust in realistic adversarial settings. The paper evaluates deployed infrastructure and open-weight models, measures held-out attack generalization, and studies diverse ensembles as a defense.

TPRs exceed 99% and FPRs remain below 1% for many classifiers, but real-world effectiveness is still questionable.
Existing robustness studies often use narrow evaluations, such as one attack for training and the same or similar attacks for testing.This leaves unknown and structurally different attacks insufficiently assessed.
Robustness against one attack does not guarantee defense against others, motivating evaluation across diverse known and unknown attacks.
Tokenization schemes and network architecture influence asymmetric text-attack transferability, supporting ensembles of diverse detectors.The cited work identifies tokenization as the strongest factor among the listed model properties.
The study evaluates end-user messaging apps, third-party services, and Hugging Face open-weight models under benign and adversarial conditions.Messages were transmitted through real services, and imperceptible perturbations were verified to survive SMS/RCS delivery.
FNRs exceeded 5% across evaluated systems, and all tested systems were vulnerable to adversarial manipulation, including delivered imperceptible perturbations.The study also manually assigns misdetected messages to 12 spam categories.
Adversarial training substantially improves robustness on seen attacks but transfers almost completely only within attack families, leaving gaps against encoding-level attacks.Homoglyph substitution is given as an example of a structurally distinct attack.
An ensemble with adversarial training and minority voting improves resilience to known and unknown attacks while exposing a precision–recall trade-off.Higher-recall configurations can elevate false positives into double digits; deployment guidance distinguishes false-positive-sensitive and recall-critical traffic.

2 Background and Related Work

SMS filtering spans network, gateway/content, and on-device layers, but this study focuses on content-based services and applications. Prior work largely examines isolated models or known attacks, leaving deployed-system robustness and held-out attack transfer underexplored.

SMS Message Flow and Filtering Points: SMS messages traverse carrier routing and filtering infrastructure before reaching recipient devices and on-device filtering applications.The flow includes SMSC routing, HLR queries, protocol conversion, carrier firewalls, and delivery to the recipient device.
SMS Message Flow and Filtering Points: Filtering occurs at network/operator, gateway/content, and on-device/application layers, using metadata, content, sender reputation, blacklists, and link reputation.
Evaluation Scope: The evaluation targets on-device applications and content-based services because they classify message content without network signaling or routing metadata.A fresh SIM is used to neutralize sender-reputation and volumetric network controls.
Evaluation Scope: RCS bypasses the SS7/SMSC path and relies more heavily on on-device and server-side content classification of the kind studied here.
Related Work: Prior adversarial-text studies show that imperceptible Unicode insertions and homoglyph substitutions can induce misclassification.
Related Work: Earlier work often reports high accuracy or adversarial-training gains using isolated models, without testing deployed infrastructure or unseen attack families.
Research Gaps and Contributions: The paper addresses these gaps by evaluating deployed apps, services, and open-weight models under perceptible and deliverable imperceptible attacks.It explicitly measures transfer to held-out attack families.
Research Gaps and Contributions: Its ensemble compares minority, majority, and consensus rules and selects members with diverse architectures and tokenization schemes.This contrasts with homogeneous combinations and standard majority or soft voting defenses.

3 Materials and Methods

The study evaluates real-world SMS anti-spam infrastructure and open-weight models using reproducible datasets, black-box attacks, and delivery-verified perturbations. It also defines adversarial-training and ensemble procedures for testing robustness under realistic deployment constraints.

Dataset: The Super SMS Dataset contains 67,010 labeled messages, while a balanced test set uses 5,000 benign and 5,000 spam messages.The dataset aggregates established SMS and public-source collections; 300 spam messages form a holdout set for adversarial-example generation.
Threat model: The threat model assumes black-box access, with attackers modifying messages to induce misclassification while preserving meaning and imperceptibility.Attackers observe only model predictions and cannot access architecture, weights, or parameters.
Delivery verification: Imperceptible perturbations were tested for survival through SMS/RCS delivery, with invisible-character and homoglyph attacks surviving on Android and iOS while RTLO did not.The delivery check excludes perturbations that fail during message composition or transmission.

4 Evaluating Anti-spam Infrastructure and Open-Weight Models

Evaluation of open-weight models, mobile messaging apps, and third-party services reveals substantial imbalances between detecting spam and preserving legitimate messages. Many systems miss large portions of spam, while others achieve higher spam recall at the cost of excessive false positives.

Open-weight models: 94.96% accuracy, 94.73% F1-score, and 9.44% FNR made satish860_sms the strongest evaluated open-weight model, but its FNR remained above 5%.
Mobile text apps: Mobile apps all exceeded the 5% FNR target; SMS Shield and VeroSMS reached 27.39% and 23.65% FNR, while Mezo recorded 8%.
None of the evaluated models achieved both FPR and FNR below the desired 5% threshold.
Misclassification analysis: Misclassification concentrated in promotional, account-alert, and finance-fraud messages, especially when spam cues were subtle or blended with legitimate content.

5 Own Trained Detectors

Custom detectors were trained across architectures to improve SMS spam detection under realistic and adversarial conditions. BERT and fastText provided the strongest clean-set performance, while BERT generalized well to newer Smishtank messages but remained vulnerable to some imperceptible attacks.

Clean performance: 99.33% accuracy, 1.06% FNR, and 0.30% FPR made BERT the strongest custom detector in the initial evaluation.
Clean performance: 98.54% accuracy, 2.66% FNR, and 0.34% FPR made fastText the second strongest reported custom model.
Cross-dataset evaluation: 93.3% of Smishtank messages exceeded cosine similarity 0.5 with the training corpus, while 61 messages were semantically novel.
Cross-dataset evaluation: BERT correctly classified 910 of 913 Smishtank messages, achieving 99.67% recall and FNR of 0.33%.
Robustness evaluation: BERT was resilient to perceptible attacks, including spacing at 94.1%, but performance fell to 41.3% under homoglyph attacks.
Robustness evaluation: Random Forest, SVM, and LightGBM fell below 20% accuracy under all adversarial conditions, while LSTM and fastText also weakened under imperceptible attacks.

6 Adversarial Training

Adversarial training preserved BERT’s clean performance and improved robustness, but its transfer to unseen attacks depended strongly on perturbation structure. Robustness transferred nearly perfectly within attack families, yet remained limited against encoding-level homoglyphs.

Adversarially trained BERT: 98.93% accuracy after adversarial training showed only a marginal performance decrease while remaining within the 5% FPR/FNR benchmark.
Adversarially trained BERT: Adversarial training improved BERT’s resistance to both perceptible and imperceptible attacks.
Held-out generalization: Same-class held-out attacks showed a maximum 2.3-point gap from full adversarial training, indicating near-perfect within-family transfer.
Held-out generalization: Reordering produced a 4.0-point gap, whereas homoglyph accuracy improved from 41.3% to 79.7% but retained a 19.8-point gap.
Held-out generalization: Homoglyphs resisted transfer because they substitute characters through cross-script Unicode encoding rather than directly manipulating Latin characters.
Realistic class imbalance: At 5% spam prevalence, precision remained 86.36%, supported by a 0.82% FPR under class imbalance.

7 Ensemble Detectors

Minority-voting ensembles provide the strongest robustness across the evaluated adversarial attacks, especially when combining diverse and adversarially trained models. Majority and consensus rules degrade as ensemble size increases, while robustness gains remain coupled to false-positive costs.

Open-source ensembles: 100% accuracy on original spam messages is achieved by the open-source-model minority ensemble, with 94.9% against spacing and 98.0% against insert-character attacks.Invisible-character manipulations reach 99.7% accuracy, while Homoglyphs remain a weaker case.
Ensemble comparison: Minority-rule ensembles consistently outperform majority and consensus strategies across evaluated ensemble sizes and attack types.This pattern holds for both own-model and open-source-model ensembles.
Own-model ensembles: 99.74% accuracy on original spam messages is achieved by the own-model minority ensemble, with 96.94% against spacing and 98.21% against insert-character attacks.Swap-character attacks reach 98.72% accuracy in the same arrangement.
Own-model ensembles: 84.95% accuracy against Homoglyphs contrasts with 93.37% against Reordering for the own-model minority ensemble.Imperceptible attacks remain more challenging, although Reordering improves substantially relative to other strategies.
Own-model ensembles: 20.15% and 19.90% accuracy under Homoglyphs and Reordering show the majority rule’s decline as the own-model ensemble grows.The seven-model consensus rule falls to 2.81% against invisible characters and 2.04% against Reordering.
Operating trade-offs: 13.80% and 13.90% false-positive rates accompany stronger spam detection in higher-recall ensemble configurations.The top-7 configuration is robust against attacks but incurs higher false positives on original messages.

8 Deployment Recommendations by Use Case

The recommended ensemble configuration depends on whether deployment prioritizes preserving legitimate messages or maximizing spam recall. Smaller Top-3 minority ensembles suit false-positive-sensitive channels, whereas larger Top-5 or Top-7 configurations suit high-spam traffic.

General principle: The evaluated ensembles provide a spectrum of operating points rather than one universally best detector.Selection depends on the relative operational costs of false positives and false negatives.
False-positive-sensitive traffic: 2.86%–2.94% FPR and FNR ≤0.52% make the smallest Top-3 own-trained or hybrid minority configuration suitable for security-sensitive channels.This configuration prioritizes legitimate transactional-message integrity while retaining adversarial robustness.
Recall-critical traffic: 0.21%–0.23% FNR is achieved by larger Top-5 or Top-7 minority ensembles, at the cost of 11.26%–13.90% FPR.These settings are appropriate only when false positives are relatively tolerable and legitimate traffic is more repetitive or recoverable.
Deployment summary: Transactional and OTP channels should use low-FPR Top-3 settings, while marketing-heavy channels may accept higher-recall Top-5 or Top-7 settings.The paper recommends matching ensemble size to each use case’s false-positive tolerance.

9 Limitations and Future Work

The study’s real-world delivery evaluation is bounded by a single SIM, device, country, and fixed classifier operating points. Future work should test broader delivery ecosystems, datasets, and prevalence-aware cost-sensitive tuning.

Evaluation scope: A single SIM card, device, and country constrain how broadly the real-world evaluation generalizes.Carrier, intermediary-provider, and device behavior may differ across the SMS delivery pipeline.
Class-imbalance analysis: The class-imbalance analysis fixes each classifier’s balanced-set TPR and FPR instead of re-optimizing ensembles for asymmetric deployment costs.It isolates base-rate effects on precision but leaves cost-sensitive operating-point optimization for future work.
Delivery generalization: Cross-carrier, cross-device, and cross-OS experiments are proposed to measure provider variability in delivery and rendering behavior.The study also calls for assessing whether invisible characters, homoglyphs, and reordering survive normalization across network paths.
Future directions: Future work includes characterizing telco intermediaries, tuning ensembles under realistic prevalence and FPR constraints, and expanding SMS datasets.Novel defenses tailored to SMS-specific adversarial tactics are also proposed.

10 Conclusion

The evaluated SMS anti-spam infrastructure is vulnerable to adversarial manipulation, while adversarial training transfers strongly within attack families but poorly to encoding-level attacks. Architecturally diverse ensembles using minority voting improve robustness, with operating points that trade false positives against recall for different deployment needs.

Deployed messaging apps, third-party services, and open-weight detectors are unreliable under benign conditions and highly vulnerable to adversarial manipulation, including perturbations surviving real SMS/RCS delivery.The evaluation covers systems end users rely on and verifies that imperceptible perturbations are not merely lab artifacts.
Adversarial training transfers almost completely within the same transformation family but leaves a large robustness gap against mechanistically distinct encoding-level attacks such as homoglyph substitution.The conclusion characterizes this benefit as class-bounded and states that adversarial training alone is necessary but not sufficient.
An ensemble of architecturally diverse detectors, including an adversarially trained model and governed by minority voting, offers a more effective defense against adversarial text attacks.The design is motivated by asymmetric transferability, making attacks that evade one detector less likely to evade all detectors.
Hybrid ensembles retain strong detection on original spam while substantially mitigating known and unseen attacks, with the hybrid configuration providing the best balance of robustness and clean-data performance.This result supports combining adversarial training with detector diversity rather than relying on adversarial training alone.
Larger, higher-recall configurations reduce false negatives but inflate false positives into double digits, so operators should select low-FPR or higher-recall points according to deployment costs.The paper recommends low-FPR configurations for transactional and OTP/banking traffic and higher-recall configurations for marketing-heavy channels.
Fully optimizing operating points under realistic spam prevalence and closing the remaining gap against encoding-level attacks remain important directions for future work.

A Evaluation Metrics

The paper evaluates SMS spam detectors using accuracy, F1, and rates for correct and erroneous classifications, including changes under adversarial modifications.

Metrics for Model Evaluation: The evaluation reports Accuracy, F1 Score, True Positive Rate, True Negative Rate, False Positive Rate, and False Negative Rate.These metrics assess overall performance and spam, ham, false-alarm, and missed-spam outcomes.
True and False Positives: For detector A, type-I errors are false positives and type-II errors are false negatives.The detector labels an SMS as spam when A(x) = 1 and as ham when A(x) = 0.
True and False Positives: True Positive Rate is defined as the complement of False Negative Rate.This relates correctly detected spam to spam messages incorrectly classified as ham.
Metrics for Model Evaluation: Accuracy and F1 Score are defined from correctly classified spam, incorrectly flagged ham, and missed spam counts.The supplied formulas use the detector’s spam and ham decisions to construct these metrics.
Adversarial Evaluation: Under evasive adversarial technique E, the metrics are recalculated to measure the impact of adversarial modifications.Changes in FPR and FNR indicate how effectively E evades detection.

B Performance Evaluation of open-weight Models

The open-weight-model evaluation is reported in Table 23, which presents the performance results for the evaluated model.

Performance Evaluation of open-weight Models: Table 23 reports the performance evaluation of the open-weight model.The supplied passages provide the table reference but no individual metric values.

C Training Configuration of Our Own Detectors

The study trains classical and neural SMS spam detectors with fixed, architecture-specific configurations to compare adversarial robustness rather than maximize clean accuracy.

Classical Models: SVM, Random Forest, and LightGBM use their respective library defaults, with no hyperparameter search.The classical models are trained with scikit-learn and LightGBM defaults.
Design Rationale: Fixed configurations aim to attribute adversarial-degradation differences to architecture and tokenization rather than unequal tuning effort.The objective is comparative robustness under adversarial perturbation, not maximal clean accuracy.
Environment: Neural models are trained in TensorFlow 2.15 on one NVIDIA Tesla T4 GPU, while classical models run on CPU.Classical training uses scikit-learn and LightGBM.
Classical Models: Classical-model preprocessing lowercases messages, removes punctuation and nonalphabetic content, filters English stopwords, and applies TF–IDF features.The TF–IDF vectorizer uses unicode accent stripping and unigram features.
Neural Models: LSTM, BiLSTM, and CNN inputs use a 14,000-token Keras vocabulary, an out-of-vocabulary token, and sequences fixed to 50 tokens.Sequences are post-padded and post-truncated, while infrequent terms map to <OOV>.
Training Data: All models are trained on Dtrain containing 57,018 messages.The vocabulary cutoff retains the most frequent terms.

D Hugging Face Open-Weight Model List and Exclusion Analysis

The study lists 33 Hugging Face open-weight SMS spam models, but only 11 were successfully executed; 22 were excluded for configuration, serialization, or structural problems.

Model Inventory: 33 Hugging Face open-weight SMS spam detection models were downloaded and documented.The study provides the complete model list for transparency and reproducibility.
Execution Results: 11 of the 33 models were successfully executed and evaluated.Their classification metrics appear in Section 5.1 and Appendix B.
Exclusion Analysis: 22 models could not be run because of configuration, file-serialization, or structural issues.Table 25 records the exact failure reason for each excluded model.