PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking Windows Malware Detection: A Comprehensive Evaluation of Problem-Space Adversarial Robustness

Mashal Zainab, Salijona Dyrmishi, Hamid Bostani, Lorenzo Cavallaro, Maxime Cordy

arXiv:2609.34456v1cs.CR

TL;DR

Robustness evaluations of Windows malware detectors are fragmented and constrained by the need for valid, functional executables. This study evaluates attacks and detectors across complementary robustness dimensions, finding that attack success depends on both model and strategy and that hardening does not reliably transfer across attacks.

  • Problem

    Prior robustness evaluations are fragmented, while executable validity, reproducibility, and computational costs constrain broad comparisons across malware detectors and attacks.

    Source: §I · pp. 1–2

  • Method

    The study evaluates nine problem-space attacks against eight Windows malware classifiers and examines attack effectiveness, complementarity, transferability, and adversarial hardening.

    Source: §I · p. 2

  • Results

    Attack effectiveness varies with detector and strategy, and two complementary attacks cover approximately 99% of the evasive examples observed in the benchmark.

    Source: §I · p. 2

  • Takeaways & Limitations

    Attack-by-attack evaluation can underestimate risk, while hardening does not consistently protect against diverse attack strategies.

    Source: §VIII · p. 13

    The models were trained on EMBER, which represents only a subset of the full distribution, and unreproducible published attacks may leave parts of the threat landscape unevaluated.

    Source: §VII · p. 13

Abstract

from arXiv · show

Problem-space evasion attacks have exposed critical weaknesses in machine learning-based malware detectors; yet, their evaluation remains fragmented across models, datasets, and attack methodologies, often neglecting domain-specific requirements such as executability and functionality preservation. We address this gap with a unified, large-scale evaluation of nine state-of-the-art evasion attacks against eight Windows malware detectors, including seven open-source models and one commercial detector, under executability-preserving conditions. Our study analyzes attack effectiveness, complementarity, transferability, and adversarial hardening to evaluate robustness along complementary dimensions. We show that detector vulnerability depends strongly on both model representation and attack type: raw-byte detectors are particularly susceptible to several classes of problem-space manipulation, but no detector family is uniformly robust across all attacks. Importantly, effectiveness is not explained by transformation-space size alone: the strongest attacks can achieve substantially higher success while using fewer distinct transformations and concentrating on a small set of high-impact manipulations. We further show that two complementary attacks are sufficient to cover approximately 99% of the adversarial examples produced by the remaining evaluated attacks. Transferability exhibits a different pattern from direct attack success: attacks with low direct success can produce highly transferable evasions. Finally, adversarial hardening is highly attack- and model-dependent: robustness gains often fail to transfer across attacks and can even increase susceptibility to unseen attacks. These findings highlight limitations in current malware robustness evaluations, establish a comprehensive empirical baseline, and clarify relationships between effectiveness, transferability, and defense robustness.

I. INTRODUCTION

The introduction frames problem-space malware evasion as a realistic threat but notes that robustness evidence is fragmented and constrained by executable-validity, reproducibility, and computational requirements. It proposes a broad evaluation across attacks and detector representations, against a background of distinct static-classifier families and vulnerabilities.

I. INTRODUCTION

  • Prior evaluations are fragmented across detectors and attacks, while executable validity, reproducibility, and computational costs constrain broad robustness testing.These constraints can leave robustness conclusions narrow or over-optimistic.Source: §I · p. 1
  • The study evaluates attack effectiveness, overlap, transferability, and hardening across multiple Windows malware classifiers and problem-space attacks.The evaluation spans raw-byte, image-based, and feature-engineered detectors and diverse attack strategies.Source: §I · pp. 1–2

1) Evasion Attacks:

  • Problem-space attacks modify executable malware while preserving functionality, unlike feature-space attacks that manipulate abstract features without ensuring realistic, functional samples.The paper focuses on black-box problem-space attacks, which operate under limited model knowledge and operational constraints.Source: 1) Evasion Attacks · p. 2

2) Static Windows Malware Classifiers:

  • Static Windows classifiers use engineered features, visualized binaries, raw executable bytes, or graphs to represent malware for detection.The representations range from hand-crafted PE features and byte-derived images to end-to-end byte models and structured program graphs.Source: 2) Static Windows Malware Classifiers · pp. 2–3
  • All four classifier approaches face robustness challenges, from feature-targeted bypasses and binary perturbations to manipulations of program structure.The introduction motivates evaluating diverse static detector categories rather than assuming any representation is robust.Source: 2) Static Windows Malware Classifiers · p. 3

3) Black-Box Problem-Space Windows Malware Evasion Attacks:

Black-box problem-space attacks use varied search and transformation strategies to create functional malware that evades detection. The study evaluates representative attacks from these families against static detectors.

3) Black-Box Problem-Space Windows Malware Evasion Attacks:

  • Reinforcement-learning attacks iteratively select sequences of semantics-preserving modifications, with MAB-malware and Gym-malware among the examples.Other examples include AIMED-RL, MalwareTotal, AMG-VAC, DQEAF, SRL, MalInfo, and gym-malware-mini.Source: 3) Black-Box Problem-Space Windows… · p. 3
  • Randomization attacks apply stochastic changes such as section injection or code padding, while evolutionary attacks evolve candidate binaries through selection, crossover, and mutation.ARMED and Dropper represent randomization; GAMMA, AIMED, MDEA, and AMG-PDG represent evolutionary methods.Source: 3) Black-Box Problem-Space Windows… · p. 3
  • GAN-based attacks generate adversarial payloads or perturbations, whereas heuristic and search-based attacks use targeted modifications to improve evasion probability.GAPGAN and MalFox are GAN examples; MiniMal and Malware-makeover are heuristic or search-based examples.Source: 3) Black-Box Problem-Space Windows… · p. 3
  • Problem-space attacks aim to produce functional malware variants, but format preservation is more common than full executability and retention of malicious behavior.Some recent work adds sandbox or semantic verification to improve realism.Source: 3) Black-Box Problem-Space Windows… · p. 3
  • The evaluation compares representative attacks including AIMED, AIMED-RL, ARMED, GAMMA and variants, MAB-malware, GameUp, and MiniMal across static detectors.The evaluated detectors use diverse feature representations.Source: 3) Black-Box Problem-Space Windows… · p. 3

III. RELATED WORK

The related-work review finds limited breadth in detector and attack comparisons, realistic black-box evaluation, transferability, and attack complementarity. The research questions address these gaps through comparisons of attack efficacy, ensembles, transfer, and adversarial hardening.

III. RELATED WORK

  • The survey identifies four recurring gaps: detector diversity, cross-attack comparison, realistic black-box problem-space evaluation, and analysis of transferability and complementarity.The paper presents a unified large-scale evaluation to enable systematic comparisons across attack strategies and detector representations.Source: §III · pp. 3–4
  • Among 33 surveyed studies, MalConv and LGBM appear in 21 and 19, respectively, while only two studies evaluate more than three model types.The review also reports that 12 studies consider multiple attacks and commercial detectors appear in 14 studies.Source: §III · pp. 3–4
  • Prior attack comparisons remain bounded by narrow detector, dataset, or generator coverage, and some do not test attacks against adversarially hardened models.EXE-Bench broadens benchmark coverage but uses three manipulation families in its adversarial evaluation.Source: §III · p. 4
  • Feature-space and white-box evaluations do not necessarily capture attackers’ constraints to produce valid, functionality-preserving executables without detector access.The review notes prior work on raw-binary classifiers and adversarial training, while emphasizing the realism limits of feature-space or white-box settings.Source: §III · p. 4
  • Earlier transferability studies often use single-step attacks or small datasets, while multi-attack complementarity across models remains largely unexplored.The review identifies open questions about transfer of iterative, functionality-preserving attacks.Source: §III · p. 4

IV. RESEARCH QUESTIONS

  • RQ1 and RQ2 ask which factors shape attack efficacy across classifiers and whether an ensemble can combine attacks’ unique successful evasions.The study evaluates nine attacks on eight representative static classifiers and assesses attack overlap to identify ensemble members.Source: §IV · p. 4
  • RQ3 examines cross-model transfer, and RQ4 tests whether hardening against one attack improves robustness to other attacks.The transfer study tests surrogate-generated examples on target models; the hardening analysis compares performance across attacks.Source: §IV · p. 5
  • The authors aim to turn these evaluations into concrete recommendations for more rigorous malware-classifier robustness assessment.They focus on the roles of evasion attacks and defense mechanisms.Source: §IV · p. 5

V. EVALUATION FRAMEWORK AND SETTINGS

The evaluation framework separates experimental factors, threat model, and robustness outcomes. It organizes comparisons across data, representations, models, and attacks under a black-box setting.

V. EVALUATION FRAMEWORK AND SETTINGS

  • The framework treats data, representation, model, and attack as separate experimental factors, including the transformations and functional constraints used to generate adversarial malware.The factors capture dataset provenance and composition, detector input representation, model architecture, and attack-generation strategy.Source: §V · p. 5
  • Under the black-box threat model, adversaries lack detector internals and training data and can only query outputs; robustness is assessed through direct effectiveness, transferability, and hardening.Figure 1 provides an overview of the adversarial malware generation, evaluation, and retraining pipeline.Source: §V · p. 5, Figure 1 · p. 5

A. Models

The evaluation covers eight pretrained detectors spanning engineered PE features, raw bytes, and image representations, alongside nine diverse black-box problem-space attacks. Models and attacks were selected to support broad, reproducible comparison under common effort constraints.

A. Models

  • Eight pretrained detectors span engineered PE features, raw-byte sequences, and grayscale-image representations.Seven are open-source models and one is a commercial detector.Source: A. Models · pp. 5–6
  • The evaluated models use distinct training datasets, including EMBER versions, SOREL-20M, and a proprietary dataset for the commercial image model.The study calibrates decision thresholds at low false-positive operating points, relaxing from 0.1% to 1% FPR when TPR is below 70%.Source: A. Models · p. 6

B. Attacks

  • Nine black-box attacks cover reinforcement learning, evolutionary and genetic algorithms, greedy search, and random mutation.They use functionality-preserving executable transformations and include methods with different optimization and transformation mechanisms.Source: B. Attacks · pp. 6–7
  • Attacks were selected for problem-space operation, black-box applicability, open-source artifacts, and state-of-the-art relevance.Methods with overlapping transformations or search algorithms, or insufficient reproducibility, were excluded; many excluded attacks build on Gym-Malware.Source: B. Attacks · pp. 6–7
  • All attacks share a 50-query budget, 10-transformation budget, five optimization rounds, and hard-label setting.Method-specific parameters follow the respective authors’ recommendations.Source: B. Attacks · p. 7

C. Datasets

The study uses a unified set of 1,000 functional malware samples for robustness evaluation and supplements it with model-native test samples. For hardening, it generates adversarial examples from a random subset of each model’s training malware.

C. Datasets

  • The Unified Evaluation Set contains 1,000 valid functional PE malware samples, disjoint from all models’ training data.Samples were drawn from an approximately 65k-file pool and selected to be consistently classified as malware by all evaluated models using adapted STAS sampling.Source: C. Datasets · p. 7
  • Native Test Sets add 1,000 correctly classified malware samples from each model’s original dataset test set.These experiments examine whether findings generalize across dataset choices and are reported in Appendix F.Source: C. Datasets · p. 7
  • Adversarial hardening uses attacks to generate examples from a random 15% of each model’s training malware and retrains the models.Further dataset details are provided in Appendix C.Source: C. Datasets · p. 7

D. Sandboxing environment and executability checks

The evaluation counts an adversarial sample as successful only when it evades detection and executes successfully, while noting that execution does not establish preservation of malicious behavior. It also defines metrics for attack success, overlap coverage, transferability, and hardening-related changes in attack success rate.

D. Sandboxing environment and executability checks

  • Every generated adversarial sample is checked for executability, and success requires both evasion and successful execution.Cuckoo sandbox validation is applied across attacks; samples that crash or fail in the sandbox are excluded.Source: D. Sandboxing environment and… · p. 7
  • Executability validates operational validity but does not directly measure behavioral equivalence with the original malware.Source: D. Sandboxing environment and… · p. 7

E. Evaluation Metrics

  • Attack Success Rate measures the proportion of originally detected malware modified into executable variants that evade detection.Source: E. Evaluation Metrics · p. 7
  • Coverage measures how much one attack’s executable evasions overlap with those produced by another attack or an ensemble.It is defined from the intersection of the evaded-sample sets.Source: E. Evaluation Metrics · pp. 7–8
  • Transferability measures whether executable adversarial samples successful on a surrogate model also evade a target model.The transferability rate is conditioned on surrogate evasion, whereas transfer-based ASR measures end-to-end success over eligible originals.Source: E. Evaluation Metrics · p. 8
  • Delta ASR compares attack success before and after adversarial retraining using absolute and relative changes.Source: E. Evaluation Metrics · p. 8

VI. RESULTS

Across the evaluated attacks, MAB-Malware has the highest average ASR, while executability checks sharply reduce Gamma-section’s ASR. Its advantage is not explained by using a broader transformation set, and successful perturbations concentrate on a few transformations.

RQ1: Attack Effectiveness

  • ASR means across three runs have standard deviations below 1 percentage point, and attack trends retain the same ranking across the two evaluation settings.The additional setting’s Native Test Set results are reported in Appendix F, Table XI.Source: §VI · p. 8
  • MAB-Malware’s seven transformations are a strict subset of FAME’s ten, yet its 93.64% average ASR exceeds FAME-based attacks’ 8.35%–19.19%.Successful MAB-Malware evasions average 2.89 transformations and 1.55 distinct types; AIMED averages 10.00 and 6.40, and ARMED 10.00 and 10.00.Source: §VI · p. 8, Table II · p. 9
  • Section addition and overlay append occur in 54.16% and 51.65% of successful MAB-Malware evasions, and account for 47.20% and 40.97% of final actions before first observed evasion.Unlike MAB-Malware, AIMED distributes successful sequences across a broader portion of its action space.Source: §VI · p. 8
  • Gamma-section’s ASR drops by an average of 94% after executability checks, versus decreases below 1% for Gamma-padding, MAB-malware, and MiniMal and no decrease for Gamma-shift.The checks are applied post hoc to the Gamma variants, MAB-malware, and MiniMal because they are not integrated into those attacks.Source: §VI · p. 8
  • MAB-Malware achieves the highest average ASR (93.64%); Gamma-shift and MiniMal average 65.79% and 65.74%, respectively.The three remain the most effective against EMBER-24; MAB-Malware achieves almost complete evasion on the commercial detector.Source: §VI · p. 8, Table I · p. 9

Appendix E.

Across evaluated detectors, attack success and transfer vary with both model representation and attack strategy; hardening gains likewise depend on the model and attack. A compact two-attack ensemble covers nearly all observed evasions, while defenses may not generalize to unseen attacks.

Attack effectiveness

  • MAB-Malware achieves the highest average ASR despite using a strict subset of FAME’s transformation space, and successful evasions typically use few distinct transformations.Section addition and overlay append dominate successful MAB-Malware sequences; identifying and combining high-yield transformations matters more than simply enlarging the action set.Source: Appendix E · p. 9
  • Raw-byte end-to-end detectors are generally more susceptible to the evaluated problem-space manipulations, while engineered-feature detectors resist several attacks.No attack’s effectiveness uniformly characterizes every detector; both attack success and rankings vary with the target model.Source: Appendix E · p. 9

Attack complementarity

  • MAB-Malware and Gamma-shift together cover 98.89% of Gamma-section evasions and 100% of MiniMal, AIMED, and Game-Up evasions in the evaluated samples.The ensemble achieved similarly high coverage for the remaining attacks; the analysis is not intended to exhaust the full attack space.Source: Appendix E · pp. 9–10, Table III · p. 10

Cross-model transferability

  • AIMED, AIMED-RL, and ARMED have the highest transferability among their successful evasions, while MAB-Malware and MiniMal are more moderate.Direct attack success and transferability are distinct: attacks with low ASR can still produce evasions that generalize well across models.Source: Appendix E · p. 10, Figure 2 · p. 10
  • For arbitrary-surrogate transfer, MAB-Malware reaches 81.5% on EMBER-24 but 20.8% on MalConv; MiniMal reaches 86.1% on MalConv but 1.1% on EMBER-24.Transferability depends on the attack and surrogate–target relationship, not just direct attack effectiveness.Source: Appendix E · pp. 10–11, Figure 3 · p. 10
  • Best-surrogate selection raises MAB-Malware transfer ASR from 20.8% to 61.2% against MalConv and from 19.4% to 64.4% against MalConv2.In the best-surrogate setting, MiniMal reaches 99.7% against the commercial detector, compared with 97.23% under direct attack.Source: Appendix E · pp. 10–11

Adversarial hardening

  • Same-attack hardening produces model-dependent ASR reductions: EMBER-17 falls by 95.96% for MAB-Malware, while MalConv falls by 1.27%.The Commercial model’s MAB-Malware hardening produces no ASR reduction; UES results are also shown for defense and test attacks in Figure 4.Source: Table IV · p. 11, Appendix E · p. 12, Figure 4 · p. 12
  • Defense transfer varies by model and attack: Gamma-padding and Gamma-shift hardening transfer strongly for EMBER-17, but weakly for MalConv, while AIMED-hardened EMBER-17 increases ASR under Gamma-padding and Gamma-shift by 69.90% and 71.21%.Shared transformation availability alone does not reliably predict defense transfer; the attacks’ transformation selection and composition differ.Source: Appendix E · p. 12

VII. LIMITATIONS AND FUTURE WORK

The study’s conclusions are bounded by constraints on data availability and reproducibility, and the authors call for research on robust feature representations.

Limitations

  • Training data came from EMBER, which represents only a subset of the full malware distribution, because acquiring original malware and benign executables is difficult.The study also evaluated nine representative attacks, but unreproducible published strategies may leave parts of the threat landscape uncaptured.Source: §VII · p. 13
  • Problem-space attack evaluation inherits computational-complexity and data-availability constraints.Some published strategies could not be reproduced because implementations were incomplete or depended on proprietary disassembly tools.Source: §VII · p. 13

Future work

  • The authors propose prioritizing robust feature representations that are less susceptible to semantics-preserving manipulations.They argue that feature representations can allow such modifications to dictate model decisions.Source: §VII · p. 13

VIII. CONCLUSION

The conclusion presents a systematic evaluation of Windows malware classifiers and a reproducible benchmark, alongside study materials and constraints on what is shared. The included sections also describe prior-work coverage and the problem-space transformations used in the evaluated attacks.

VIII. CONCLUSION

  • The study evaluates attack effectiveness, cross-model transferability, and adversarial hardening, finding substantial variation across models and datasets.It provides a reproducible benchmark and systematic methodology intended to support malware-classifier robustness research and evaluation practices.Source: §VIII · p. 13

OPEN SCIENCE

  • Code, experimental configurations, and trained models are released to support reproduction, but raw malware binaries are not redistributed.The stated reasons are safety and licensing considerations.Source: OPEN SCIENCE · p. 14

LLM USAGE CONSIDERATIONS

  • LLMs assisted with editorial work and code development and debugging, but were not used to generate experimental data or results.The authors report reviewing and validating generated content and retaining responsibility for manuscript accuracy and integrity.Source: LLM USAGE CONSIDERATIONS · p. 14

ETHICAL CONSIDERATIONS

  • The study uses isolated environments and authorized or public samples, does not distribute malware or payloads, and reports responsible disclosure to affected vendors.The authors state that they introduce no new functionality-preserving binary transformations.Source: ETHICAL CONSIDERATIONS · p. 14

A. Overview of Evaluated Models and Attack Methods in Prior Work

B. Problem-Space binary transformations and obfuscation techniques

  • The evaluated attacks use problem-space transformations constrained to preserve runtime executability and functionality.The section describes Gamma-Padding, Gamma-Shift, Gamma-Sections, MiniMal, MAB-malware, and other attacks’ transformation sets.Source: B. Problem-Space binary… · p. 17

C. Datasets

The evaluation recovers runnable binaries for feature-only datasets and reports clean-model performance at calibrated false-positive rates. Attack success can change substantially after executability checks, particularly for Gamma-section, whose PE-header modifications can compromise loading.

C. Datasets

  • For feature-only datasets, the study uses hash-based searches of public malware repositories and archives to recover matching runnable binaries.Runnable samples are needed for executability-preserving problem-space attacks; disarmed binaries cannot be executed in a sandbox.Source: C. Datasets · p. 17

D. Model performance on clean samples

  • Models are generally calibrated at 0.1% FPR, with the constraint relaxed to 1.0% when recall at the stricter point is below 70%.Most models have AUC above 0.95; EMBER-based models and MalConv variants achieve the highest F1-scores and recall at low-FPR settings.Source: D. Model performance on clean samples · p. 17

E. Attack results (RQ1)

  • Gamma-section’s attack success rate drops dramatically after non-executable samples are removed, while Gamma-shift and Gamma-padding show only minor differences.Gamma-section alters PE section-table and header fields; inconsistencies can cause parsers or the Windows loader to fail, unlike padding appended at the file end and minor in-file relocations.Source: E. Attack results (RQ1) · pp. 17–18

G. Coverage of MAB-malware (RQ2)

MAB-malware overlaps strongly with several attacks but covers less of the evasions from Gamma-section, Game-Up, and ARMED. The hardened-model evaluation reports clean-sample metrics and NTS-dataset attack and hardening comparisons.

G. Coverage of MAB-malware (RQ2)

  • MAB-malware covers 99.57% of Gamma-padding, 99.34% of MiniMal, and 98.44% of Gamma-shift evasions, but coverage is lower for Gamma-section (93.37%), Game-Up (96.79%), and ARMED (96.93.45%).The authors conclude that MAB-malware does not fully span the evasive behavior space.Source: G. Coverage of MAB-malware (RQ2) · pp. 18–19, Table XII · p. 20

H. Performance of Hardened Models on clean samples (RQ4)

  • The hardened-model clean-sample evaluation reports recall, precision, F1-score, accuracy, and AUC across four defense strategies at fixed FPR operating points.Most models use 0.1% FPR; the commercial model uses 1.0% because its TPR is significantly reduced under stricter constraints.Source: H. Performance of Hardened Models on… · p. 19

I. Performance of Hardened Model on the NTS dataset (RQ4)

  • For the NTS dataset, the reported tables cover mean ASR with and without executability checks, hardened-model performance, and ASR changes before and after hardening.The captions specify ASR across models and hardened-model metrics at a fixed FPR; they do not provide outcome values in the supplied passages.Source: Table X · p. 20, Table XIII · p. 21, Table XIV · p. 22

Related papers

Loading 2609.34456v1…