Adversarial Attacks against Windows PE Malware Detection: A Survey of the State-of-the-Art

1. Introduction

• The survey focuses on Windows PE malware because file-format-specific detection methods are not universal and PE malware is a prominent threat.The introduction reports that over 90% of malware detected per day by Kaspersky at the end of 2020 was Windows PE malware.Source: §1 · p. 2
• ML/DL-based malware detectors are presented as able to generalize to new and previously unseen malware, but are vulnerable to adversarial examples.Adversarial examples are generated by carefully perturbing legitimate inputs to confuse target models.Source: §1 · pp. 2–3
• Realistic adversarial PE malware must evade detection while preserving its format, executability, and maliciousness, making feature-space perturbations alone insufficient.The survey identifies these as three unique challenges for attacks against Windows PE malware in the wild.Source: §1 · pp. 3–4
• Earlier surveys either formalized problem-space attacks without systematically reviewing the field, offered an incomplete attack taxonomy and omitted defenses, or covered multiple formats too broadly.The authors argue that Windows PE's distinct file-format characteristics warrant a format-specific review.Source: §1 · pp. 3–4
• The survey reviews state-of-the-art attacks and defenses, proposes an attack taxonomy, and discusses other attacks beyond adversarial methods.It also aims to offer principles for generating real adversarial PE malware rather than impractical adversarial features.Source: §1 · p. 4
• The paper also outlines its coverage of PE files and detection frameworks, adversarial challenges, attacks, defenses, other attack types, and future research directions.The sections are organized across §§ 2–7.Source: §1 · p. 4

2. Machine Learning and Deep Learning for PE Malware Detection

• The section introduces PE file layout and PE malware, then outlines the general ML/DL-based framework for PE malware detection.Source: §2 · p. 5

2.1.1. General Layout of PE files

• A PE file contains header information, section information, and unmapped data, with headers describing how the OS maps the file for execution.The section information includes code and necessary data, while unmapped data includes unused bytes such as debug information at the file's end.Source: §2.1.1 · pp. 5–6, Figure 1 · p. 5
• Windows PE files commonly include EXE and DLL types: EXEs can run independently, whereas DLLs provide functions other executables can use.Source: §2.1.1 · p. 6

2.1.2. PE Malware

• PE malware remains a predominant threat to personal and business users, and includes forms such as viruses, trojans, worms, adware, and ransomware.The text attributes PE's prevalence in part to Windows popularity and PE files' ability to contain all data needed to launch an attack.Source: §2.1.2 · p. 6

2.2.1. Overview

• Signature-based detection only recognizes previously collected malware, while ML/DL methods are presented as generalizing to unseen malware through a three-step pipeline.The steps are data acquisition, feature engineering, and learning from models and predictions.Source: §2.2.1 · p. 6

2.2.2. Data Acquisition

• There is no widely used PE malware benchmark, and disagreement among antivirus engines complicates labeling collected samples.Labeling approaches include choosing a credible antivirus or using votes from multiple engines.Source: §2.2.2 · p. 7

2.2.3. Feature Engineering

• Feature engineering maps PE files to numeric representations grouped as static, dynamic, or hybrid features, with Table 1 pairing common features and learning models.Static features include byte sequences and header information; dynamic features are monitored during execution, while hybrid examples include opcodes, API calls, and graphs.Source: §2.2.3 · pp. 7, 9, 10, Table 1 · p. 8

2.2.4. Learning from Models and Predictions

• Detection models map numeric executable features to malware or goodware labels, and the chosen ML/DL architecture must accept the feature representation.Training learns model parameters from samples; prediction computes probabilities for both classes.Source: §2.2.4 · p. 11

3.1. Adversarial Attacks: The General Concept and Taxonomy

• Adversarial examples target a model at test time, and attacks are classified by adversary space and adversary knowledge.The section frames these as two broad dimensions for categorizing attacks.Source: §3.1 · p. 11

3.1.1. Adversary’s Space: Feature-space Attack versus Problem-space Attack

• Feature-space attacks seek a nearby misclassified feature vector, while problem-space attacks modify a real-world input at minimal cost; PE feature mappings often frustrate direct gradient methods.The problem-space mapping φ is usually neither invertible nor differentiable.Source: §3.1.1 · pp. 12–13

3.1.2. Adversary’s Knowledge: White-box Attack versus Black-box Attack

• White-box attackers know the target model and training dataset, whereas black-box attackers know only the model output; gray-box assumptions should specify what is known.The section describes black-box attacks as harder because they have less model knowledge.Source: §3.1.2 · p. 13

3.2. Three Unique Challenges of Adversarial Attacks for PE Malware: From Feature-space to Problem-space

• PE feature mappings vary and may lack an invertible or differentiable inverse, so a feature-space misclassification need not yield adversarial PE malware.In black-box settings, attackers may not know the PE feature mapping; the section therefore points to problem-space attacks that manipulate the executable itself.Source: §3.2 · pp. 13–14

3.2.1. Challenge 1: Follow the format of PE files (format-preserving)

• PE transformations must respect the file specification: adding a section in header information violates the format, while adding one in section information is acceptable.The survey identifies checking each candidate transformation against the PE specification as an approach to this challenge.Source: §3.2.1 · p. 14

3.2.2. Challenge 2: Keep the executability for PE files (executability-preserving)

• Format compliance does not ensure execution: even a one-bit change in the .data section can cause a runtime crash by disrupting required data loading.PE layout rules do not determine whether the content of each element will execute properly.Source: §3.2.2 · p. 15

3.2.3. Challenge 3: Keep the same maliciousness for PE malware (maliciousness-preserving)

• Adversarial PE malware must retain the original malicious behavior as well as execute; runtime consistency in the same sandbox is used as reasonable empirical evidence.The survey says theoretical proof is nearly impossible and notes that many attacks inspect transformations only to a limited extent.Source: §3.2.3 · pp. 15–16

4.1. White-box Adversarial Attacks against PE Malware Detection

• White-box attacks assume full knowledge of the detector and are divided into feature-space and problem-space attacks.Feature-space attacks manipulate representations such as API-call lists, while problem-space attacks modify PE files directly.Source: §4.1 · p. 16, §4.1.1 · p. 16, §4.1.2 · p. 19

4.1.1. Feature-space White-box Attacks against PE Malware Detection

• For MalConv, slack-FGSM outperforms append-FGSM while modifying fewer bytes.The proposed explanation is that appended bytes may exceed MalConv’s 2 MB input limit, whereas slack-FGSM can exploit contextual bytes within the convolution window.Source: §4.1.1 · p. 17
• Enhanced-BFA is ten times more effective than BFA when few bytes are appended, but their performances are comparable when many bytes are appended.BFA selects important benign-file data blocks using Grad-CAM saliency vectors; Enhanced-BFA additionally optimizes the perturbations with FGSM.Source: §4.1.1 · p. 17
• ATMPA produces grayscale-image perturbations that destroy malware structure and prevent execution, limiting its practicality for real-world detection.COPYCAT [23] appends an adversarial image to the malware image, while AMAO inserts semantic NOPs to make the PE malware resemble a generated adversarial image while attempting to preserve executability.Source: §4.1.1 · p. 18

4.1.2. Problem-space White-box Attacks against PE Malware Detection

• Problem-space white-box attacks largely target raw-byte detectors, using payload insertion, header edits, or functionality-preserving binary transformations.AMB appends optimized bytes; Demetrio et al. [27] instead edit DOS-header bytes, and RAMEn [28] uses Extend or Shift manipulations. Sharif et al. guide function-level in-place randomization and code displacement with gradients.Source: §4.1.2 · p. 19

4.2. Black-box Adversarial Attacks against PE Malware Detection

• Black-box attacks assume knowledge is limited to the target detector’s outputs, and are divided into feature-space and problem-space approaches.Outputs may provide only a malicious/benign label or may also include a probability.Source: §4.2 · p. 22

4.2.1. Feature-space Black-box Attacks against PE Malware Detection

• Feature-space black-box methods target API-call lists or sequences, using learned substitutes, feature selection, or transferability to evade detectors.MalGAN [30] adds irrelevant API calls using a substitute detector and generator; Improved-MalGAN uses different API-call lists for training. EvnAttack adds goodware-associated calls or removes malware-associated calls. For API sequences, methods include generative RNN attacks [33], GADGET, which transfers attacks from a surrogate through a proxy wrapper, and ELE, which adds calls via IAT hooking.Source: §4.2.1 · pp. 23–24
• BADGER optimizes API-call insertion positions or uses random positions for black-box evasion, while Rosenberg et al. transfer attacks by modifying important, easily changeable features.BADGER inserts only calls with no or irrelevant effect and reduces queries through logarithmic backtracking; the transfer attack uses a substitute model and predefined feature values.Source: §4.2.1 · p. 24
• SRL [38] uses deep reinforcement learning to insert semantic NOPs into control-flow graphs while preserving semantics, achieving nearly 100% attack success against two graph-neural-network detector variants.The agent iteratively chooses basic blocks and insertion points until the modified malware bypasses the target detector.Source: §4.2.1 · p. 24

4.2.2. Problem-space Black-box Attacks against PE Malware Detection

• Gym-malware [137] defines 10 format- and functionality-preserving actions, while follow-up RL methods address stochastic actions, state size, transformation diversity, and subjective reward design.Its DQN and ACER agents offer limited improvement over random policy; later approaches include deterministic action spaces, variational actor-critic, and inverse RL.Source: §4.2.2 · pp. 25–26
• Compared with, malware samples generated by are verified for Cuckoo sandbox executability and original maliciousness by comparing before-and-after function-call graphs with IDA Pro.Source: §4.2.2 · p. 26
• Problem-space black-box methods include randomized PE modifications and feature-guided byte changes; targeted occlusion can destroy execution, while hill climbing accepts transformations that increase benign probability.ARMED tests randomized modifications in a sandbox, and [49] minimizes action sequences; targeted occlusion replaces important byte regions, whereas queries after transformations.Source: §4.2.2 · pp. 26, 28
• AIMED reduced the time to generate the same number of successful adversarial samples by up to 50% versus random search.GAMMA [52] uses genetic search to optimize evasion while penalizing payload size; MDEA [51] uses genetic optimization for adversarial retraining but does not test generated samples for functionality.Source: §4.2.2 · p. 27
• GAPGAN achieved a 100% attack success rate against MalConv with appended payloads equal to 2.5% of input malware length.MalFox [54] instead uses packing-related perturbations to generate functionally indistinguishable malware against realistic black-box antivirus products.Source: §4.2.2 · pp. 27–28

4.3. Summary of Adversarial Attacks against PE Malware Detection

• Across attack categories, white-box methods generally use constrained gradient-based optimization and depend on specific detectors, whereas problem-space black-box attacks are theoretically detector-agnostic.The survey reports that most attacks guarantee format preservation but not executability or maliciousness; it identifies attacks [49] as experimentally verified for both latter properties.Source: §4.3 · pp. 28–29

5. Adversarial Defenses for PE Malware Detection

• The survey presents defenses as part of an arms race in which new attacks can inspire defenses and new defenses can prompt attacks.It notes that few researchers focus exclusively on PE malware defenses, while attack studies may include corresponding defenses.Source: §5 · p. 29

5.1. Adversarial Training

• Adversarial training aims to improve ML/DL robustness by retraining with generated adversarial examples, with or without original training examples.Source: §5.1 · p. 29
• Adversarial training typically generates adversarial PE malware or features, then retrains a corresponding detector with those inputs, sometimes alongside original data.Examples include training on MalGAN-generated API calls, gym-malware-generated PE files, and EvnAttack-generated API calls; SecDefender also adds a security regularization term for evasion cost.Source: §5.1 · pp. 29–30
• Adversarial training experimentally mitigates one or several attacks, but generating adversarial examples adds significant training costs.Studies retrain detectors using generated adversarial PE malware or features, with or without original training data.Source: §5.1 · pp. 29–30

5.2. Other Defense Methods

• PEberus combines semantic-gap checks, max-voting across detectors, and stateful nearest-neighbor detection, flagging a file if any method predicts malware.Its semantic-gap heuristics check slack space, overlay, and duplicate content; the nearest-neighbor detector compares files with previously detected malware.Source: §5.2 · p. 30
• Lucas et al. describe binary normalization and instruction masking, applying randomized transformations or zero-masking before PE files reach the detector.Normalization iteratively applies in-place randomization to reduce lexicographic representation; masking selects random instruction bytes.Source: §5.2 · p. 30

• After reviewing adversarial attacks and defenses, the discussion covers other PE malware attacks and future research directions.Source: §6 · p. 30

6.1.1. Universal Adversarial Perturbation

• A problem-space universal perturbation is a short sequence of PE transformations that can evade a classifier with high probability while preserving format, executability, and maliciousness.Labaca-Castro et al. [155] use greedy search over transformations such as adding or renaming sections and packing or unpacking.Source: §6.1.1 · p. 31

6.1.2. Training-time Poisoning Attacks

• Training-time poisoning aims to make a poisoned model retain clean-input predictions while assigning an attacker-chosen prediction to inputs carrying a backdoor trigger.An early attack targets a malware family under assumptions of full training-data and algorithm knowledge and control over malware training samples and labels.Source: §6.1.2 · p. 31
• Later clean-label poisoning work allows control of the poison instance but not its label; Shapira et al. [159] propose adding sections to similar goodware as a problem-space trigger.The triggered target malware is then misclassified as benign; Shapira et al. argue feature-space manipulation assumptions in [158] are unrealistic for real-world classifiers.Source: §6.1.2 · pp. 31–32
• A feature-space poisoning setup with a 1% poison rate and 17 manipulated features yielded about 20% attack success rate.Severi et al. [158] use SHAP-based greedy selection of coherent feature combinations and values to create backdoor triggers in poisoned goodware.Source: §6.1.2 · p. 31

6.1.3. Model Steal Attacks

• A black-box substitute PE malware detector achieved nearly 90% similarity to the remote model in predictions on the same inputs.Ali and Eshete use limited training samples, pre-trained image models, raw-byte-to-image feature mapping, and cross-domain transferability.Source: §6.1.3 · p. 32

6.2.1. Strong Demands for Robust PE Malware Detection

• Current PE malware defenses are limited and mostly empirical heuristics effective against only one or a few attacks; no certified defenses are reported.The survey calls for both empirical defenses and theoretically guaranteed robustness, noting possible benefits for evaluating effectiveness and cloud-service pricing transparency.Source: §6.2.1 · p. 32

6.2.2. Practical and Efficient Adversarial Attacks against Commercial Anti-viruses in the Wild

• Future attacks should use more practical, stealthier PE transformations and more efficient search strategies, since reinforcement learning and genetic algorithms are time-consuming.The survey contrasts easily noticed and removed NOP insertion with splitting a control-flow block into multiple iteratively called blocks.Source: §6.2.2 · pp. 32–33
• Because commercial antivirus products commonly combine static and dynamic analysis, the survey calls for practical, efficient attacks that target dynamic analysis and commercial products.Source: §6.2.2 · p. 33

6.2.3. Lack of Benchmark Platforms for Research

• A shared benchmark platform should support comprehensive, fair evaluation with datasets, representative detectors, attack and defense methods, metrics, and environments.The survey links the need to incomplete and biased evaluations that make strengths and limitations difficult to quantify.Source: §6.2.3 · p. 33

7. Conclusion

• The survey organizes research on adversarial attacks and defenses for Windows PE malware detection into taxonomies intended to make the literature more concise and understandable.It presents a comprehensive review of both attack and defense efforts.Source: §7 · p. 33
• Format-preserving, executability-preserving, and maliciousness-preserving are identified as three challenges to maintaining the semantics of adversarial PE malware in practical, realistic attacks.Source: §7 · p. 33
• The paper also discusses attacks against Windows PE malware detection beyond adversarial attacks and points to future research directions.Source: §7 · p. 33
• The authors hope the survey will provide a comprehensive understanding of fundamental issues and serve as a starting point for advancing the research field.Source: §7 · p. 34