Source-linked AI summary
REPLICANT: Learning Policies for Evading and Hardening Malware Detectors
Shae McFadden, Ilias Tsingenopoulos, Mario D'Onghia, Alexander Herzog, Myles Foley, Chris Hicks, Lorenzo Cavallaro, Fabio Pierazzi
TL;DR
Malware-detector robustness must be assessed against realistic adversaries because existing attacks often assume privileged target information. REPLICANT learns hierarchical, transferable evasion policies under a label-only black-box model, achieving stronger attacks and producing more generalizable adversarially trained detectors. Its reported effectiveness remains relative to the capability set and dataset used, rather than an absolute upper bound on evasion.
Problem
Existing malware-evasion attacks often assume privileged information and optimize per sample, leaving realistic, general, and efficient black-box effectiveness insufficiently characterized.
Method
REPLICANT formulates problem-space evasion as hierarchical deep-reinforcement-learning policy learning that selects behavior-preserving modifications and target queries.
Results
78.8% mean Attack Success Rate within twenty queries is achieved across seven classifiers, three feature spaces, and two training periods, versus 43.3% for sample transfer and 65.2%–56.6% for prior Android attacks.
Takeaways & Limitations
Learned evasion policies transfer across samples, detectors, and feature spaces and provide a signal for adversarial training that yields more generalizable robustness.
Takeaways & Limitations
Reported ASRs are relative to the evaluated capability set and dataset, while identifying the strongest capability set remains open.
Abstract
from arXiv · showhide
To determine the real-world effectiveness of machine learning based malware detection, it is vital to evaluate its robustness against highly capable adversaries. However, state-of-the-art attacks do not effectively model realistic adversaries, as they often assume access to privileged information such as the training data, feature space, or confidence scores of the target. In this work, we present Replicant, a deep reinforcement learning framework that learns the realistic task of evasion under a strict label-only black-box threat model. Replicant learns a reusable policy on how to modify a malware sample and when to query the target, which transfers across samples, detectors, and feature spaces. Across seven Android malware detectors and three feature spaces, Replicant is the strongest and most query-efficient approach achieving a mean attack success rate of 78.8%, a relative improvement of 20.9%-39.2% over the state-of-the-art. Furthermore, when used for adversarial training, Replicant also outperforms the state-of-the art by producing detectors with more generalizable robustness. With Replicant we demonstrate that learning the task of evasion not only results in stronger attack performance but, crucially, provides a better signal for hardening malware detectors.
1 Introduction
Android malware evasion research often relies on privileged information and per-sample optimization, limiting realism, query efficiency, and transferability. REPLICANT learns transferable problem-space evasion policies and improves both attack performance and adversarial-training outcomes.
- Feature-space attacks are unsuitable for malware when feature changes lack realizable, function-preserving application modifications, motivating problem-space attacks.Problem-space attacks modify applications directly while enforcing syntactical and semantic constraints.
- Existing Android evasion attacks optimize individual samples against single targets, depend on privileged information or heuristics, and do not retain knowledge across evasions.The literature also conflates attack capability sets with strategies, obscuring which component produces gains.
- REPLICANT uses deep reinforcement learning to learn reusable problem-space evasion policies over a capability set.The framework models evasion as sequential decision-making in which the agent modifies samples or queries the target.
- 20.9%–39.2% higher Attack Success Rate is reported for REPLICANT than state-of-the-art Android attacks.
- 82.0% relative ASR increase is achieved when transferring a learned policy rather than samples from the same surrogate.
- AT-REPLICANT reduces attacker success below 17% against both REPLICANTWB and APGWB, while robustness declines under expanded attacker capabilities.The evaluation comprises 379,680 attack evaluations isolating evasion-strategy gains.
2 Background
The background motivates realistic problem-space malware attacks and introduces the learning concepts underlying REPLICANT. It also frames adversarial training as optimization against admissible, functionality-preserving perturbations.
- Android malware detection commonly represents applications as high-dimensional binary vectors extracted from manifests and bytecode.The evaluation uses three Android feature spaces and a large, timestamped AndroZoo corpus.
- Temporal drift is intertwined with evasion because malware authors continually adapt samples to avoid detection while preserving functionality.
- Evasion attacks modify malware at test time to induce goodware classification while retaining malicious behavior.Transferability and querying are practical strategies when access to the target model is limited.
- Problem-space attacks use realizable modifications, such as benign-code gadget transplantation, that preserve semantics while changing malware features.Gadgets are harvested from real applications and transplanted into malware.
- Adversarial training minimizes classifier loss against admissible, functionality-preserving perturbations through an inner attack maximization and outer model minimization.
- MDPs formalize reinforcement learning with states, actions, rewards, transitions, and discounting, while PPO learns actor and critic networks using a combined objective.Action masking excludes inapplicable actions, and hierarchical policies separate decisions across abstraction levels.
3 REPLICANT Framework
REPLICANT formulates malware evasion as a hierarchical MDP that chooses both when to query and which behavior-preserving modification to apply. Its reward design and training pipeline emphasize successful, query- and modification-efficient attacks.
- REPLICANT models problem-space evasion as an MDP over abstract capabilities and uses hierarchical PPO to learn the attack policy.Android gadget transplantation instantiates the capability set in the experiments.
- The attacker seeks goodware classification while preserving malicious behavior and minimizing malware modifications and target queries.Capabilities are behavior-preserving modifications with observable effects on the attacker's feature representation.
- REPLICANT's state is the current feature representation, and its hierarchical actions first choose SUBMIT or MODIFY, then select an applicable capability when modifying.SUBMIT queries the classifier; MODIFY applies a capability without querying, and action masking removes inapplicable modifications.
- The episode terminates after successful submission, detection, or the horizon, with rewards assigned separately to success, detection, and modification actions.
- Reward magnitudes encode the trade-off between modification cost and query efficiency, while chosen values guarantee positive return for successful episodes and encourage balanced behavior.
- Periodic zero-cost submits encourage exploration early in training and voluntary early submission once the agent becomes more capable.
- A shared MLP feeds query, masked-modify, and critic heads trained with PPO, while adversarial-training updates increase as fixed-budget AE production collapses.Simultaneous AE collection exposes classifiers to diverse malware variants rather than a single converged evasive strategy.
4 Experimental Setup
The experiments use temporally structured Android malware data, multiple feature spaces, repeated random seeds, and established problem-space attack baselines to evaluate evasion and robustness.
- Dataset: 224,965 Android malware samples from 2021–2024 form the Hypercube dataset, sampled from Google Play with a 10% malware distribution and VTT2.
- Testing Sets: Black-box attacks use a one-year post-training test period, while white-box adversarial-training evaluations use two years to assess deployment-period robustness.
- Testing Sets: Each evasion test set contains malware correctly detected by every classifier, isolating attack effects from differences in classifier coverage.
- Variance: Experiments repeat across ten random seeds and report 95% confidence intervals, resampling network initialization, training splits, and test sets.
- Baselines: Comparisons include APG, AdvDroidZero, EvadeDroid, random-action REPLICANT, and five alternative reinforcement-learning agent designs.
5 Evaluation
REPLICANT is evaluated as a query-efficient black-box evasion attack, a transferable policy, and an adversarial-training strategy across broad Android malware settings. It achieves strong evasion and improves robustness, while robustness declines when attacker capabilities expand beyond those used during training.
- Black-Box Evasion: 96.6% mean ASR with 3.6 average queries per sample in the matched setting demonstrates strong query efficiency.The matched setting shares classifier architecture and feature space between surrogate and target but uses separate training sets.
- Black-Box Evasion: 78.8% ASR across 1,764 surrogate/target combinations, beating every Android baseline by 13.6–22.2 ASR points within twenty queries.The evaluation spans differing classifiers, feature spaces, and training periods.
- Policy Transfer: 78.8% PTE versus 43.3% STE, an 82.0% relative improvement, shows that learned policies transfer better than precomputed evasive samples.PTE overtakes STE within three queries on average, and REPLICANT ASR degrades gradually from 96.6% to 73.0% as surrogate-target gaps grow.
- Adversarial Training: AT-REPLICANT reduces mean ASR to 9.9% against REPLICANTWB and 16.3% against APGWB after 15 rounds, including against an unseen attacker.The results indicate that adversarial training using policy-generated evasions produces robustness beyond the attack used for training.
- Adversarial Training: AT-REPLICANT improves mean A-AUT(F1) by 2.8 versus 1.1 for AT-APG, despite a larger initial mean F1 drop of −5.6 versus −3.1.Both pipelines improve mean A-AUT(F1) in four of seven classifiers.
- Capability Drift: 91.3% and 96.1% full-capability-set ASR for REPLICANTWB against AT-REPLICANT and AT-APG shows that robustness decays under capability drift.AT-REPLICANT remains more robust than AT-APG to REPLICANTWB at every drift stage, but both defenses remain bound to their training capabilities.
6 Limitations
REPLICANT addresses problem-space evasion strategy under a fixed capability set, not capability-set design. Its reported ASRs are therefore relative to the chosen capability set and dataset, while identifying the strongest capability set remains open.
- REPLICANT evaluates evasion strategy given a fixed capability set rather than designing the capability set itself.The same capability set and threat model are used across attackers to preserve direct comparability and gain attribution.
- Reported ASRs are relative to the capability set and dataset used, not an absolute upper bound on problem-space evasion.
- The strongest capability set remains an open problem, although REPLICANT can be applied to new capability sets as they are developed.
- The study uses only the Android malware dataset of Hypercube to avoid experimental bias associated with other datasets.
7 Related Work
Related Android malware-evasion work includes feature-space attacks and transferability attacks, but REPLICANT combines transfer and query-based strategies while operating in problem space. Feature-space perturbations face an inverse-mapping problem because modified vectors may not correspond to realizable, behavior-preserving applications.
- Feature-space attacks directly perturb classifier inputs, but malware perturbations may not map to realizable programs.This inverse-mapping problem limits whether feature changes correspond to syntactical and semantic malware modifications.
- Problem-space attacks modify applications directly while enforcing syntactical and semantic constraints.
- Transferability attacks optimize adversarial examples on surrogate models and rely on decision-boundary alignment to evade targets.
- REPLICANT combines transfer and query-based attacks into one threat model and learns policies that transfer across classifiers and feature spaces.
8 Conclusion
REPLICANT reframes malware evasion as hierarchical policy learning, using PPO to learn which modification to apply and when to query. Its formulation emphasizes realizable constraints, stable policy optimization, and transfer across classifiers and feature spaces.
- REPLICANT reframes problem-space malware evasion as policy learning rather than per-sample optimization.
- PPO stabilizes policy updates by clipping probability ratios, limiting destructive drift from the rollout policy.
- Action masking excludes inapplicable capabilities, including conflicting gadgets and modifications exceeding per-gadget budgets.
- The architecture uses a shared MLP encoder with query, modify, and critic heads, while training samples actions for exploration and evaluation selects the highest-probability action.
C.4 RL-Based Attacks
The evaluated RL-based attacks use different learning algorithms and reward designs to optimize malware evasion. MEME uses PPO, whereas PSP-Mal uses Dueling Double-DQN with confidence-reduction rewards and prioritized replay.
- MEME applies PPO directly to a shared surrogate for model-extraction and evasion, with rewards for evasion, capability distance, similarity, and repeated-capability penalties.
- PSP-Mal uses Dueling Double-DQN with soft target updates, replay, exploration annealing, and a reward based on evasion or confidence reduction.
D Scale of Experiments
The study evaluates attacks and hardened classifiers at substantial scale across classifier, feature-space, training-period, dataset, and random-seed combinations.
- 379,680 attack evaluations and 560 hardened classifiers were produced across the paper’s main experiments.
- Sections 5.1–5.2 contain 352,800 evaluations across 1,764 surrogate-target combinations, two dataset variants, and ten seeds.
- Sections 5.3–5.4 contain 10,080 attack evaluations and 8,400 adversarial-training rounds.
E Extended Evaluations
The extended evaluations examine false-negative rates, attacker performance across test periods, and transferability across temporal, architectural, representational, and combined gaps. The associated figures report policy-transfer ASR over queries against sample-transfer baselines.
- Extended evaluations: Table 6 reports malware FNR by period, averaged across seven classifiers, three feature spaces, and ten seeds.
- Extended evaluations: Table 7 compares attackers on HypercubeVTT4 using mean ASR and average queries in matched and all settings.
- Extended evaluations: Table 8 reports ASR by test period relative to target training year, averaging results over targets trained in 2021b and 2022b.
- Transferability gaps: The temporal-gap curves show that cross-period performance follows matched-period performance and that the temporal gap is negligible.
- Transferability gaps: The architecture-gap, representation-gap, and combined-gap figures plot policy-transfer ASR over queries against sample-transfer ASR baselines.
F RL Design Ablation
The study compares REPLICANT with five prior RL-based malware-evasion designs adapted from Windows PE to the Android setting. REPLICANT remains the strongest attacker, with the highest evasion rate per query across the evaluated combinations.
- F RL Design Ablation: Five prior RL-based attacks—MEME, MAB, MRLN, AIM, and PSP—were adapted from Windows PE malware evasion to the Android environment, threat model, and capability set.The comparison isolates gains from REPLICANT’s agent architecture and reward function.
- F RL Design Ablation: REPLICANT remains the strongest attacker with statistically significant gains across all evaluated settings.The comparison follows the methodology of Table 1.
- F RL Design Ablation: REPLICANT achieves the highest evasion rate per query among all evaluated RL agent designs across 1,764 surrogate/target combinations and ten seeds.The result measures attack efficiency across the full combination set.
G Motivating Ablations
The appendix examines training-budget scaling, capability-set size, and surrogate/test overlap. These ablations motivate the reduced capability set and increasing training budget while showing negligible impact from surrogate overlap.
- G Motivating Ablations: The appendix motivates scaling the AT-REPLICANT training budget, reducing the ADZ capability set, and assessing surrogate/test overlap.These decisions are evaluated in Tables 12, 10, and 11, respectively.
- G Motivating Ablations: ADZ’s mean ASR decreases with the full capability set, while REPLICANT remains significantly stronger using the reduced 662-set.The experiments use the 2021a→2021b training period, Drebin features, five seeds, and seven classifiers.
- G Motivating Ablations: ∆ below 0.2 point in both settings measures the change in mean ASR when overlapping 2022a surrogate-training samples are excluded.The comparison uses the full 2022 test set versus a test set excluding overlap.
- G Motivating Ablations: The constant training budget collapses monotonically as the target hardens, whereas the increasing budget continues finding adversarial examples through 15 rounds.The two schedules are near-identical over the first five rounds.