Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning

Hyrum S. Anderson; Anant Kharkar; Bobby Filar; David Evans; Phil Roth

Source-linked AI summary

Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning

Hyrum S. Anderson, Anant Kharkar, Bobby Filar, David Evans, Phil Roth

arXiv:1801.08917v2Jan 26, 2018cs.CR

TL;DR

Static PE malware detectors can be challenged without differentiability, model scores, or internal model knowledge. The paper uses reinforcement learning to learn functionality-preserving PE modifications in a black-box setting, reports substantial but dataset-dependent evasion, and finds adversarial training can reduce attack effectiveness while risking overfitting.

Problem
Existing malware machine learning attacks do not provide a general black-box framework that avoids requiring differentiability, model scores, or detailed model knowledge.
Method
A reinforcement-learning agent applies functionality-preserving operations to static Windows PE files and learns evasive action sequences from detector feedback.
Results
The approach produces functioning evasive PE variants, attacks black-box models, and demonstrates that retraining on evasive ransomware variants reduces a subsequent attack's evasion rate from 12% to 8%.
Takeaways & Limitations
The framework provides a way to probe static malware-model weaknesses and generate adversarial examples under realistic label-only API access.
Takeaways & Limitations
Adversarial-training effects are dataset- and model-dependent, and adversarial variants may be idiosyncratic enough to risk poisoning the retraining dataset.

Abstract

from arXiv · show

Machine learning is a popular approach to signatureless malware detection because it can generalize to never-before-seen malware families and polymorphic strains. This has resulted in its practical use for either primary detection engines or for supplementary heuristic detection by anti-malware vendors. Recent work in adversarial machine learning has shown that deep learning models are susceptible to gradient-based attacks, whereas non-differentiable models that report a score can be attacked by genetic algorithms that aim to systematically reduce the score. We propose a more general framework based on reinforcement learning (RL) for attacking static portable executable (PE) anti-malware engines. The general framework does not require a differentiable model nor does it require the engine to produce a score. Instead, an RL agent is equipped with a set of functionality-preserving operations that it may perform on the PE file. Through a series of games played against the anti-malware engine, it learns which sequences of operations are likely to result in evading the detector for any given malware sample. This enables completely black-box attacks against static PE anti-malware, and produces functional evasive malware samples as a direct result. We show in experiments that our method can attack a gradient-boosted machine learning model with evasion rates that are substantial and appear to be strongly dependent on the dataset. We demonstrate that attacks against this model appear to also evade components of publicly hosted antivirus engines. Adversarial training results are also presented: by retraining the model on evasive ransomware samples, a subsequent attack is 33% less effective. However, there are overfitting dangers when adversarial training, which we note. We release code to allow researchers to reproduce and improve this approach.

1 INTRODUCTION

The paper frames static PE malware detection as vulnerable to adversaries that manipulate features correlated with benign behavior. It introduces reinforcement learning as a generic black-box approach for learning evasive modifications and producing functioning malware variants.

Machine learning supports antivirus detection by learning feature relationships that distinguish malicious from benign files and can generalize to similarly distributed new samples.
Sophisticated adversaries can camouflage malware by inducing feature representations correlated with benign behavior without being causally benign.
The paper presents reinforcement learning for bypassing static-feature machine learning antivirus models and generating functioning evasive malware samples.
The approach targets static Windows PE malware while also being applicable to other static machine learning detection engines, including PDFs, Mach-O, and ELF binaries.
Unlike prior approaches, the framework can generate functioning PE malware, attack black-box models without scores, and generalize to samples absent from training.
Contributions: The contributions include a generic black-box attack, evaluation against a machine learning model, adversarial-training experiments, released code, and discussion of practical limitations.

2 BACKGROUND

Static PE detection classifies files before execution, while adversarial research studies how attackers exploit model information and representation weaknesses. Existing attacks range from gradient-based white-box methods to score-based and binary black-box settings, motivating functionality-preserving black-box attacks that can generate valid evasive PE malware.

2.1 Static PE Malware Detection: Static malware detection classifies files without execution and can identify malicious files before they run.It contrasts with dynamic detection based on runtime behavior.
2.1 Static PE Malware Detection: Machine-learning malware detectors commonly use handcrafted features despite advances in end-to-end deep learning.Published malware-detection work still largely relies on manually designed representations.
2.2 Evading Machine Learning Models: Gradient-based attacks require differentiable models with known structure and weights, whereas black-box attacks may receive only malicious/benign labels.The binary-output setting is the minimum information an online classification service could provide.
2.2.1 Direct gradient-based attacks: Valid malware generation requires a mapping onto legitimate PE files that preserve the desired malicious function, a general case prior work had not studied.This requirement distinguishes executable-preserving attacks from approaches that manipulate abstract feature vectors.
2.2.4 How our work differs: The proposed RL attack targets the most difficult and realistic black-box setting, accepts only binary labels, and produces valid evasive PE samples applicable to new malware.The approach is intended both to summarize detector blind spots and to supply samples for model hardening.

3 METHOD

The paper formulates malware evasion as reinforcement learning against a static PE detector, using functionality-preserving mutations and observed state feedback to learn evasive action policies.

3.1 Reinforcement Learning: The attack uses an RL agent to find evasive Windows PE malware through repeated interactions with an anti-malware environment.The implementation applies ACER to a Markov decision process whose state is a malware feature vector and whose actions modify the PE file.
3.4 Action Space: Available actions are PE mutations that preserve file format and intended functionality, including section edits, unused sections, entry-point changes, packing, and metadata manipulation.The action set includes appending bytes, removing signer information, modifying debug information, and changing header checksums.
3.5 Implementation: The malware evasion environment is released as an extensible OpenAI Gym with a default ACER agent and PE parsing and manipulation support.The release is intended to support reproducible training and benchmarking of malware-manipulating agents.
3.2 Environment: The environment returns a binary reward for evasion or detection and a feature vector summarizing the current malware sample.In the experiments, evasion receives R = 10, while detection receives zero reward.
3.2 State Representation: The state representation is a 2350-dimensional vector covering PE metadata, imports and exports, strings, byte histograms, and byte-entropy histograms.The hashing trick fixes the dimensionality of countably infinite feature sets such as section names and imported functions.
3.4 Action Space: Stochastic mutations compress an exponentially large space of specific changes into a few dozen broad actions for reinforcement learning.Randomized names, appended bytes, and packing levels reduce the action-space burden while still modifying features used by static malware models.

4 EXPERIMENTAL SETUP

The experiments evaluate an ACER-based black-box attack against a gradient-boosted malware detector, limiting each game to ten mutations and comparing RL with random actions.

4.1 Target Model: The target is a gradient-boosted decision tree trained on 100,000 malicious and benign samples with ROC-AUC 0.993.The detector threshold is set to 0.9, approximately corresponding to a 1% false positive rate at a 90% true positive rate.
4.5 Assumptions: Training the agent on the environment’s state features is expected to produce more generous results than would generally be expected in practice.The authors describe this choice as convenient for the proof-of-concept toolkit rather than strictly necessary.
4.2 RL Agent: The agent uses a Boltzmann exploration strategy that selects mutations proportionally to their expected Q-values.The preliminary experiments use a chainer-rl actor-critic model agent in the OpenAI Gym environment.
4.3 Training Procedure: Each game permits up to ten mutations, terminates early after successful evasion, and uses a combined training budget of 50,000 mutations per model.The ten-mutation limit also reduces reinforcement-learning credit-assignment complications from long action sequences.
4.4 Baseline: The black-box attack assigns R = 10 for evasion and R = 0 for failed evasion, and is compared with randomized actions without an RL agent.This comparison isolates the attack setting and the contribution of learned action selection within the same environment.

5 RESULTS

The experiments evaluate RL evasion on training and holdout malware samples, compare the agent with random mutations, examine cross-evasion, and test adversarial hardening. Results also identify functional-execution failures and overfitting risks from binary-manipulation fingerprints.

Experimental setup: The study trains four RL agents on VirusShare, ransomware, Virut, and BrowseFox malware datasets, reserving 200 samples per dataset for holdout evaluation.The remaining samples train each agent, while evasive variants are saved under a fixed 50K-mutation budget.
Generalization: Holdout evaluation applies each trained agent to 200 unseen samples with up to 10 mutations, excluding samples initially misclassified as benign.The same exclusion criterion is used during training and evaluation.
Mutation analysis: The RL agent generally exploits the model’s upx_pack blind spot in fewer mutations than random exploration on successful holdout evasions.Table 3 reports the median number of mutations required for successful evasion for both policies.
Cross-evasion: VirusTotal testing found that the median detection ratio dropped after RL-generated mutations, while random mutations were similarly effective against commercial antivirus products.The authors caution that VirusTotal does not represent vendors’ full detection platforms.
Model hardening: 33%: adversarial training reduced subsequent evasion from 12% to 8% on 200 ransomware holdout samples.The efficacy calculation includes only malware initially labeled malicious by the classifier.
Shortcomings and open challenges: Only eight of ten sampled VirusShare evasive variants executed properly, challenging the assumption that PE mutations preserve function across uncommon formats and obfuscation tricks.The authors suggest starting with malware samples known to parse correctly.
Shortcomings and open challenges: Adversarial training may overfit to unique LIEF fingerprints, such as uncommon section names, rather than learning distinctions between malicious and benign files.The paper recommends investigating this issue further.

6 DISCUSSION

The paper presents RL as a generic black-box attack on Windows PE machine-learning models, requiring no knowledge of the target model and capable of modifying new malware samples. Its effectiveness and hardening results depend on the dataset and model, while PE-format failures and adversarial-training poisoning remain important concerns.

6 DISCUSSION: The attacker requires no knowledge about the model under attack, and the RL agent can modify new malware samples to bypass it.The authors report that the agent can generalize to new samples after training against the model.
6 DISCUSSION: The approach is presented as the first to automatically create novel Windows PE evasive malware variants by modifying binary files.This capability is distinguished from attacks that cannot directly generate functioning PE malware under ordinary assumptions.
6 DISCUSSION: Evasion and model-hardening results are dataset and model dependent, so the authors leave a thorough study of hardening to future work.The paper also reports susceptibility among many VirusTotal engines to static malware changes, including random mutations.
6 DISCUSSION: Functionality-preserving mutations can still break PE files when uncommon practices or intentional obfuscation complicate parsing.These corner cases represent points of failure for generating evasive variants.
6 DISCUSSION: Adversarial retraining can introduce idiosyncratic samples unrelated to maliciousness, risking poisoning of the training dataset.The authors caution that these issues require care and further study.
6 DISCUSSION: The released code supports reproduction and future improvements through richer mutations and improved reinforcement-learning agents.The authors specifically encourage researchers and industry practitioners to extend the environment and agents.