PaperlayerView exact revision on arXiv

Source-linked AI summary

Classifier-Dependent Benefits of Pseudo-Labeling for Semi-Supervised Android Malware Attribution

Md Rafid Islam, Zahid Hasan, Hafiz Abdur Rahman

arXiv:2609.29564v1cs.LG

TL;DR

Android malware-family classification faces scarce expert labels, while prior SSL studies rarely compare benefits across classifier types or report statistical significance. This paper evaluates pseudo-labeling across six classifiers and finds that its effects depend on the classifier, with practical gains concentrated in particular settings and harder-to-classify families.

  • Problem

    Android malware-family classification is challenged by scarce expert labels, and prior SSL work rarely compares benefits across classifier types or reports statistical significance.

    Source: §I · p. 1

  • Method

    The paper systematically assesses pseudo-labeling across six classifiers and five labeled-data ratios on CICMalDroid 2020.

    Source: §VI · p. 6

  • Results

    SSL benefits are classifier-dependent: SVM gains +4.4% accuracy at 5% labels (p = 0.0028), while Random Forest drops −3.1% at 1% labels (p = 0.0048).

    Source: B. SSL Benefit Summary · p. 4

  • Takeaways & Limitations

    At 10% labels, LGBM reaches 90% accuracy without SSL, while per-class analysis finds larger SSL gains for harder-to-classify families such as Adware than for Benign.

    Source: §I · p. 1, §V · p. 6

    The study evaluates only pseudo-labeling with default classifier hyperparameters on CICMalDroid 2020, so results may not apply to other datasets or more recent Android versions.

    Source: §VI · p. 6

Abstract

from arXiv · show

Detecting and classifying Android malware families remains challenging due to high feature dimensionality, class imbalance, and the high cost of expert-labeled data. Semi-supervised learning (SSL) offers a way to leverage unlabeled samples, but prior works rarely test whether SSL benefits generalize across classifier types or report statistical significance. We present a systematic evaluation of pseudo-labeling across six classifiers (LightGBM, XGBoost, Random Forest, Logistic Regression, MLP, and SVM) on the CICMalDroid 2020 dataset, using five-fold stratified cross-validation and paired t-tests across five labeled ratios (1-20%). We find that SSL benefit is strongly classifier-dependent: SVM shows the largest significant gain (+4.4% accuracy at 5% labels, p = 0.0028), LightGBM improves modestly (+0.8 to +1.3% at 2-5% labels), while Random Forest is significantly harmed at low label ratios (-3.1% at 1% labels). Per-class analysis reveals SSL disproportionately benefits the hardest-to-classify families, with Adware F1 improving by +13.8 percentage points versus only +0.8 for the already well-classified Benign class. We further show that approximately 800 labeled samples (10% of the dataset) yield near-optimal performance across all classifiers. These findings offer practical guidance on when and with which classifier pseudo-labeling is worthwhile for Android malware classification.

I. INTRODUCTION

Android malware-family classification is difficult because of feature dimensionality, class imbalance, and scarce expert labels. This paper evaluates pseudo-labeling across classifiers and addresses gaps in prior work on cross-classifier comparison and statistical significance.

I. INTRODUCTION

  • Scarce expert-labeled data, high feature dimensionality, and class imbalance make Android malware-family classification challenging.Family attribution is needed for mitigation beyond binary malware detection.Source: §I · p. 1
  • The study evaluates pseudo-labeling across six classifier families, using five-fold stratified cross-validation and paired t-tests at five labeled-data ratios.The evaluation uses CICMalDroid 2020 and reports results as mean ± standard deviation across folds, with significance tests at α = 0.05.Source: §I · p. 1
  • SSL improves Adware F1 by +13.8%, compared with +0.8% for the already-well-classified Benign class.The reported per-class analysis indicates disproportionate gains for harder-to-classify families.Source: §I · p. 1
  • Approximately 800 labeled samples, or 10% of the dataset, yield near-optimal performance within 6–12% of the fully supervised upper bound.Source: §I · p. 1

II. RELATED WORK

  • Prior studies seldom compare SSL benefits across classifier families or report statistical significance and cross-validation variance.The paper identifies these as gaps in prior Android malware SSL research.Source: §I · p. 1, §II · p. 2

III. METHODOLOGY

The methodology uses CICMalDroid 2020, a dataset with 11,598 samples and 470 dynamic features. Features are standardized, with no feature selection or dimensionality reduction applied.

A. Dataset Description

  • CICMalDroid 2020 contains 11,598 samples and 470 dynamic features spanning system calls, binder calls, and composite behavior.Source: A. Dataset Description · p. 2
  • All numerical features are standardized to zero mean and unit variance using scikit-learn’s StandardScaler.Source: A. Dataset Description · p. 2
  • The study applies no feature selection or dimensionality reduction, preserving the original feature space for comparison across classifiers.Source: A. Dataset Description · p. 2

B. Experimental Setup

The experimental setup separates a held-out test set from the training pool, samples labeled data at five ratios, and compares supervised and pseudo-labeling models through stratified cross-validation.

1) Train-Test Split:

  • The dataset is split into a 70% training pool of 8,118 samples and a stratified 30% held-out test set of 3,480 samples.The test set is excluded from model selection and hyperparameter tuning.Source: 1) Train-Test Split · p. 2

2) Labeled Ratios:

  • Labeled subsets use 1%, 2%, 5%, 10%, or 20% of the total dataset, corresponding to approximately 81, 162, 405, 811, or 1,623 samples.Remaining training-pool samples are treated as unlabeled data for semi-supervised learning.Source: 2) Labeled Ratios · p. 2

3) Cross-Validation Protocol:

  • Five-fold stratified cross-validation compares supervised and pseudo-labeling models on each fold’s validation set at every labeled ratio.Stratification preserves class distribution in each fold; labeled subsets are sampled from each fold’s training partition.Source: 3) Cross-Validation Protocol · p. 2

4) Evaluation Metrics:

The evaluation reports accuracy and macro F1 across folds and uses per-class F1 analysis to examine whether SSL benefits malware families uniformly. The pseudo-labeling procedure retains confident predictions for final model training.

4) Evaluation Metrics:

  • Accuracy and macro F1-Score are reported as mean ± standard deviation over five folds.Macro F1 averages the harmonic mean of precision and recall across classes without weighting by class size.Source: 4) Evaluation Metrics · p. 2

5) Per-Class Analysis:

  • Per-class F1 scores are computed for the classifier and labeled ratio showing the strongest SSL improvement to assess whether gains differ across malware families.Source: 5) Per-Class Analysis · p. 2
  • The procedure trains an initial classifier on labeled data, keeps unlabeled-sample predictions with confidence at least 0.7, then retrains on the combined labeled and pseudo-labeled sets.The algorithm specifies five classes and a confidence threshold τ = 0.7.Source: 5) Per-Class Analysis · p. 3

C. Classifiers

The evaluation uses six classifier families with default hyperparameters to assess whether pseudo-labeling benefits depend on model architecture. Pseudo-labeling generates artificial labels for unlabeled data based on model confidence.

C. Classifiers

  • Six classifiers from varied model families are evaluated, generally with default hyperparameters rather than substantial tuning.This design is intended to assess whether SSL benefits generalize rather than overfit the dataset.Source: C. Classifiers · p. 3

D. Semi-Supervised Learning: Pseudo-Labeling

  • Pseudo-labeling generates artificial labels for unlabeled data from model confidence, with τ = 0.7 selected to balance sample count and label quality.Higher thresholds yield too few pseudo-labeled samples, while lower thresholds introduce noisy labels.Source: D. Semi-Supervised Learning:… · p. 3

E. Statistical Analysis

Paired two-tailed t-tests compare supervised and SSL accuracy across five cross-validation folds. A fully supervised model trained on all 8,118 training labels supplies an upper bound for judging performance with limited labels.

E. Statistical Analysis

  • Paired two-tailed t-tests compare supervised and SSL accuracies across five folds for each classifier and labeled ratio.The null hypothesis is no improvement; p < 0.05 indicates a significant improvement or degradation according to the sign of the mean difference.Source: E. Statistical Analysis · p. 3

F. Upper Bound Estimation

  • The fully supervised upper bound uses 8,118 training labels and is evaluated on a held-out test set of 3,480 samples.The upper bound is used to compare SSL performance and assess whether limited-label SSL approaches it.Source: F. Upper Bound Estimation · p. 3

IV. EXPERIMENTAL RESULTS AND ANALYSIS

Fully supervised performance is strongest for LightGBM, while pseudo-labeling effects vary across classifiers: SVM benefits most, Random Forest is harmed at low label ratios, and several classifiers show no significant effect. Per-class results show larger gains for harder-to-classify families.

A. Upper Bound Performance

  • 94.66% accuracy and 93.61% macro F1 are LightGBM’s fully supervised upper-bound results; SVM has the lowest reported performance at 77.93%.At higher label ratios, LightGBM, XGBoost, and Random Forest exceed 94% accuracy, leaving little opportunity for SSL improvement.Source: A. Upper Bound Performance · pp. 3–4, Table III · p. 4

B. SSL Benefit Summary

  • +4.4% accuracy at 5% labels (p = 0.0028) is SVM’s largest statistically significant SSL gain; LightGBM gains +1.3% at 2% labels and +0.8% at 5% labels.Table IV reports SSL accuracy minus supervised accuracy in percentage points.Source: Table IV · p. 4, B. SSL Benefit Summary · p. 4
  • −3.1% at 1% labels (p = 0.0048) and −1.6% at 2% labels (p = 0.0268) are significant Random Forest degradations with SSL.XGBoost has no significant change at any labeled ratio; MLP’s positive gaps are not significant, and Logistic Regression shows no effect or small negative effects.Source: B. SSL Benefit Summary · p. 4
  • +13.8 percentage points for Adware and +7.8 for Banking contrast with +0.8 for Benign in SVM per-class F1 at 5% labels.Adware rises from 12.5% to 26.3% F1, while the already better-classified Benign class starts at 62.8%.Source: Table V · p. 4, B. SSL Benefit Summary · p. 4

C. Effect of Labeled Data Quantity

On this dataset, approximately 811 labeled samples bring all classifiers within 2–3% of their upper-bound accuracy. SSL gains are concentrated at low label ratios, while MLP results vary substantially across folds.

C. Effect of Labeled Data Quantity

  • Within 2–3% of upper-bound accuracy is reached by all classifiers with 10% labeled data, around 811 samples.The paper describes approximately 800 representative samples as producing near-optimal performance for Android malware classification on this dataset.Source: C. Effect of Labeled Data Quantity · p. 4
  • At 20% labels, no classifier shows statistically significant SSL benefit, as supervised learning already approaches the upper bound.The largest SSL improvements are reported at 1–5% labels.Source: C. Effect of Labeled Data Quantity · p. 4
  • MLP has the highest fold standard deviations, up to ±5.9% at 1% labels, and its +3.3% gain at 5% labels is not statistically significant.The passage attributes this variance to sensitivity to random initialization and data sampling.Source: C. Effect of Labeled Data Quantity · p. 4

D. Upper Bound Gap Analysis

The upper-bound analysis finds that gaps narrow quickly as labeled data increases, while the key findings show that pseudo-labeling’s benefits vary by classifier and class difficulty.

D. Upper Bound Gap Analysis

E. Key Findings

  • At 5% labels, SVM gains +4.4% significantly and LGBM gains +0.8–1.3% at 2–5%, while RF significantly falls −3.1% at 1% and −1.6% at 2%.Source: E. Key Findings · p. 5
  • At 20% labels, no classifier shows statistically significant SSL improvement, suggesting approximately 1,600 labeled samples may obviate SSL.Source: E. Key Findings · p. 5
  • With 10% labels—approximately 811 samples—all classifiers reach within 6–12% of their fully supervised upper bound.Source: E. Key Findings · p. 5
  • With 10% labels, LGBM and XGB exceed 90% accuracy without SSL, making them label-efficient options in low-label scenarios.Source: E. Key Findings · p. 5
  • At 5% labels, SVM’s hardest classes gain most: Adware F1 rises +13.8% and Banking F1 rises +7.8%.Source: E. Key Findings · p. 5

V. DISCUSSION

The discussion links SSL outcomes to classifier and class behavior: hard-to-classify families benefit most, while variance and probability reliability qualify gains for some models.

  • Adware and Banking have the lowest supervised F1 scores, 12.5% and 36.5%, yet gain +13.8% and +7.8% with SSL, respectively.The authors attribute this pattern to SVM margin sensitivity and pseudo-labeled samples filling feature space where the decision boundary is poorly estimated.Source: §V · pp. 5–6
  • RF loses −3.1% and −1.6% at 1% and 2% labels; the authors suggest correlated pseudo-label noise and poorly calibrated probabilities may contribute.Source: §V · p. 6
  • MLP gains +2.4% at 2% and +3.3% at 5% labels, but cross-validation variance above 4% at low ratios prevents statistical significance.The authors cite sensitivity to initialization and sparse-data optimization as sources of instability.Source: §V · p. 6
  • LGBM gains +1.3% at 2% and +0.8% at 5% labels, but reaches 90% accuracy at 10% labels without SSL.The authors suggest its built-in regularization already provides much of the benefit pseudo-labeling offers.Source: §V · p. 6

VI. CONCLUSION

The paper evaluates pseudo-labeling across six classifiers and five label ratios on CICMalDroid 2020. Its results show classifier-dependent effects, alongside scope limits that qualify the practical guidance.

  • The assessment compares SSL with supervised learning for six classifiers across five labeled ratios on CICMalDroid 2020.Source: §VI · p. 6
  • SVM gains +4.4% at 5% labels (p = 0.0028), LGBM gains +0.8–1.3% at 2–5%, RF declines at low ratios, and MLP’s positive increases lack statistical significance.Source: §VI · p. 6
  • Per-class analysis finds the largest gains for Adware (+13.8%) and Banking (+7.8%), the families with the lowest supervised performance.Source: §VI · p. 6
  • The study tests only pseudo-labeling, uses default classifier hyperparameters, and may not generalize to newer Android versions or other malware datasets.Source: §VI · p. 6
  • The findings offer researchers and practitioners guidance on whether pseudo-labeling may be worthwhile and which classifier to use.Source: §VI · p. 6
Loading 2609.29564v1…