REPLICANT: Learning Policies for Evading and Hardening Malware Detectors

• REPLICANT formulates evasion as a sequential decision process, learning when to query a target and which available capability to apply to a modified sample.The framework is capability-set agnostic; the Android implementation uses gadget transplantation.Source: §1 · p. 2
• REPLICANT improves Attack Success Rate (ASR) by 20.9%–39.2% over state-of-the-art Android attacks.The paper evaluates it as a black-box attack against prior methods.Source: §1 · p. 2
• Policy transfer yields an 82.0% relative increase in ASR over transferring samples that evade the same surrogate.The evaluation considers all 1,764 surrogate and target combinations.Source: §1 · p. 2
• AT-REPLICANT reduces attacker success below 17% for both REPLICANTWB and APGWB, while AT-APG leaves the model over five times more vulnerable to REPLICANTWB.The comparison evaluates white-box adversarial-training variants.Source: §1 · p. 2
• Robustness gained through adversarial training diminishes when attacker capabilities expand beyond the defender’s capability set.Source: §1 · p. 2
• Applying active learning to adversarially trained classifiers recovers drift-induced performance but degrades robustness, whereas Robust AL retains AL’s performance over time and AT’s robustness.Source: §1 · p. 2
• The study comprises 379,680 attack evaluations and isolates the contribution of evasion strategies to attacker success.The paper describes it as the largest problem-space Android malware evasion evaluation to date.Source: §1 · p. 2

2.1 Android Malware Detection

• The Android evaluation spans Drebin, APIGraph, and RAMDA features and diverse detector families, using time-aware splits and realistic ≈10% malware prevalence.The feature spaces vary in size and overlap, allowing evaluation across different levels of attacker knowledge; the detector spectrum supports assessment of attack transferability.Source: §2.1 · pp. 2–3

2.2 Evasion Attacks

• Evasion modifies malware to evade classification while retaining behavior; Replicant combines surrogate-trained transferability with target queries to choose modifications and query timing.Transferability attacks optimize against a surrogate, whereas querying attacks use target feedback to guide successive versions while minimizing queries.Source: §2.2 · p. 3
• Direct feature perturbations face an inverse-mapping problem, while problem-space attacks use realizable, behavior-preserving, plausible, preprocessing-robust code changes.Gadget transplantation adds benign code harvested from real applications, satisfying these constraints by construction and reducing evasion to strategy selection over available gadgets.Source: §2.2 · p. 3
• Adversarial training hardens detectors by retraining on evasion examples, maximizing loss over admissible functionality-preserving perturbations before minimizing expected adversarial loss.The outer minimization updates model parameters, typically with stochastic gradient descent.Source: §2.2 · p. 3

2.3 Deep Reinforcement Learning

• DRL action masking excludes inapplicable capabilities, while hierarchical policies separate high-level choices from low-level execution; prior security uses and pitfalls motivate best practices.Masking gives inapplicable actions near-zero probability, improving learning efficiency in large, conditionally applicable action spaces while retaining valid policy-gradient updates.Source: §2.3 · p. 4

3.1 Threat Model

• The attacker seeks to make malware appear benign while preserving malicious behavior, using behavior-preserving modifications and minimizing both modifications and target queries.The capability set is instantiated with benign-app code gadgets transplanted into malware [10] [78]. Black-box attacks receive only target labels and use a surrogate trained on different data; white-box AT attacks access target confidence scores. Baselines use the same capability set and threat model for fair comparison.Source: §3.1 · p. 4

3.2 REPLICANT MDP

• REPLICANT’s MDP represents the current sample in the attacker’s feature space and separates query-versus-modify decisions from capability selection.Action masking excludes no-op or conflicting modifications, forces periodic queries during training, and prevents querying before a modification; transitions apply known capability effects and end at the horizon or successful evasion.Source: §3.2 · pp. 4–5
• The reward trades off evasion success against detection and modification costs, while zero-cost forced training queries encourage exploration and later voluntary early submission.Successful submissions receive positive reward, detected submissions and each modification are penalized; the relative magnitudes determine modification and query efficiency.Source: §3.2 · p. 5

3.3 REPLICANT Agent

• A shared-encoder PPO agent learns a reusable policy that transfers to unseen malware, classifiers, and feature spaces; its white-box variant submits after every modification and rewards confidence reduction.The white-box variant removes the query head and scales target-confidence changes by λWB = 5.Source: §3.3 · p. 5
• AT-REPLICANT retrains a fresh white-box policy across 15 rounds and adds unique generated evasive examples to the classifier’s training set before each next round.It doubles policy updates every five rounds to sustain example production; collecting examples throughout training exposes the classifier to diverse perturbations. RAL adds one AT-REPLICANT round per active-learning period and includes its examples in the update set.Source: §3.3 · pp. 5–6

Dataset and training splits

• Experiments use 224,965 Hypercube Android malware samples from 2021–2024, with a 10% malware distribution and VTT2; a VTT4 variant tests the effect of VirusTotal threshold.The dataset is sampled from Google Play using publication timestamps and is designed to follow unbiased dataset-creation practices. Experiments use ten random seeds and training periods 2021 and 2022 [18] [50].Source: §4 · p. 6
• Stratified splits train surrogates and REPLICANT policies separately from targets, with additional cross-year surrogate–target pairs for black-box transfer evaluations.For each seed and training year, equal-sized splits create pairs such as 2021a-vs-2021b; cross-year pairs include 2021a-vs-2022b and 2022a-vs-2021b.Source: §4 · p. 6

Testing protocol

• Tests use malware from the year after target training for black-box attacks and two years for white-box adversarial-training evaluations, on samples detected by every classifier.This limits temporal drift in black-box tests; in the 2022a-vs-2021b setting, excluding overlapping surrogate/test samples changes ASR by less than 0.2 percentage points.Source: §4 · p. 6

Evaluation scope

• Evaluation spans Drebin, APIGraph, and RAMDA feature spaces, with 10,000, 11,912, and 379 features, respectively, and seven diverse Android malware classifiers.Drebin uses its top 10k features [18] [21] [41] [51]; APIGraph combines Drebin non-API features with its API clustering pipeline; RAMDA uses its original 379 features. The classifiers are Linear SVM, SecSVM [21], Random Forest, DeepDrebin, RAMDA, HCC [16], and DRMD [51]. The differing feature-space sizes and overlap support transferability evaluation.Source: §4 · p. 6

Attack capabilities

• Attacks use a common threat model and capability set per setting, comprising applicable gadgets from prior problem-space work.The feature-space subsets contain 3,143 Drebin, 4,804 APIGraph, and 1,739 RAMDA gadgets; in cross-feature-space attacks, gadgets are selected for the attacker’s feature space and their effects are observed in the target’s space [10] [78].Source: §4 · p. 6

RL design ablation

• Five adapted Windows PE RL attacks provide design ablations: MEME, MAB-Malware, MERLIN [60], AIMED-RL, and PSP-Mal.They were adapted to the Android environment, threat model, and capability set; their designs include PPO, Thompson sampling, DQN, distributional double DQN, and dueling double DQN approaches.Source: §4 · p. 7

Metrics and statistical testing

• Primary attack ASR is measured at 20 queries per sample, with AvgQ as a secondary efficiency metric; clean performance uses F1 and AUT(F1).ASR and A-AUT(F1) are averaged across 2021b and 2022b unless otherwise specified [18] [41] [58]. Experiments are repeated over ten seeds, reporting means and 95% confidence intervals; tables use Wilcoxon signed-ranks tests, with significance notation at p < 0.01 and p < 0.001.Source: §4 · pp. 6–7

5.1 Black-Box Evasion

• 78.8% ASR across 1,764 settings beats every Android baseline by 13.6–22.2 points, while REPLICANT averages 7.4 queries per sample versus APG’s 9.3.In the matched setting, REPLICANT reaches 96.6% mean ASR with 3.6 queries per sample. It learns from binary decisions rather than target confidence scores.Source: Table 1 · p. 8, §5.1 · pp. 8–9
• REPLICANT’s mean ASR is 94.1% under VTT4, 12.6% better than the best baseline; it reaches 70.0% in-distribution versus 78.7% on temporally withheld malware, then 89.4% and 95.4% two and three years after training.VTT2 is the more conservative setting; the authors report it as following current best practice [18].Source: §5.1 · p. 9

5.2 Transferring Policies

• 78.8% ASR for policy transfer versus 43.3% for sample transfer across all settings shows that learned policies transfer better than adversarial samples.As gaps widen, policy-transfer ASR falls from 96.6% to 73.0%; adaptive re-querying helps the policy cross a target boundary even when surrogate and target boundaries are misaligned.Source: Table 2 · p. 9, §5.2 · p. 10

5.3 Adversarial Training

• 9.9% and 16.3% mean ASR against REPLICANTWB and APGWB after 15 rounds show AT-REPLICANT’s robustness to both attackers, compared with 62.4% and 35.6% under AT-APG.AT-REPLICANT collects diverse adversarial examples during stochastic policy training, improving coverage of attack paths and generalization to an unseen attacker; it also reduces initial F1 more than AT-APG (−5.6 versus −3.1).Source: Table 3 · p. 10, §5.3 · p. 11

5.4 Capability Drift

• 91.3% and 96.1% REPLICANTWB ASR at the full capability set under AT-REPLICANT and AT-APG show that robustness decays as capabilities drift, though AT-REPLICANT is more robust at every stage.The authors conclude that adversarial training remains bound to the capabilities it trains against and must adapt as those capabilities evolve.Source: Figure 4 · p. 12, §5.4 · p. 12

5.5 Temporal Drift

• 49.0 A-AUT(F1) and 0.6% WB A-AUT(ASR) show that RAL improves performance over time while maintaining robustness, matching AL’s clean performance and AT’s robustness.The evaluation finds clean-performance adaptation and adversarial robustness are largely orthogonal: AL alone recovers clean performance but offers little robustness, while AT alone suppresses attacks but provides only minor drift adaptation.Source: Table 5 · p. 12, §5.5 · p. 12

6 Limitations

• REPLICANT isolates attacker strategy by using the same capability set and threat model across attackers, rather than studying how capability sets should be designed [50].The framework can be applied to new capability sets, but identifying the strongest capability set remains an open problem; [65] explores that direction.Source: §6 · p. 13
• Reported ASRs are relative to the capability set and dataset used, not an absolute upper bound on problem-space evasion.Determining the strongest capability set remains an important open problem.Source: §6 · p. 13
• The study uses only Hypercube’s Android malware dataset [18], chosen to avoid experimental bias because it is the current standard for unbiased dataset construction.It comprises 379,680 attack evaluations and is described as the largest and most comprehensive malware-evasion study to date.Source: §6 · p. 13

Malware Evasion

• Problem-space attacks modify applications to avoid the realizability and semantic problems of feature-space perturbations, while Android malware DRL evasion remains nascent.[59] introduced realizability, plausibility, preprocessing-robustness, and semantic-preservation constraints; [60] and focus on Windows PE approaches.Source: §7 · p. 13

Black-Box Attacks

• Decision-based attacks were popularized in vision [12] [15], with later work reducing query counts or improving query efficiency using transfer-based and Bayesian priors [20] [26] [17].Other work targets sequence classifiers [24], proposes a stateful defense [62], or uses DRL to optimize offensive and defensive sides.Source: §7 · p. 13

Malware Evasion; Black-Box Attacks

• REPLICANT learns reusable evasion strategies and combines transfer with query-based attacks, transferring across classifiers and feature spaces while minimizing queries.The work positions this as a novel learned approach to evasion rather than a hand-designed capability set.Source: §7 · p. 13

Adversarial Training

• Adversarial training followed earlier feature-manipulation defenses [21], and problem-space attack-and-retrain pipelines were later introduced for Android and Windows PE detectors.[48] studies transformations and non-problem-space-compliant adversarial examples for raw-bytes classifiers.Source: §7 · p. 13
• The work draws on temporal-drift and active-learning literature for RQ5, arguing that neither adversarial training nor active learning alone preserves both properties, but their interleaving does.Tesseract [41] [58] and establish the adopted evaluation methodology, while study active learning and adversarial data poisoning.Source: §7 · p. 13

8 Conclusion

• 78.8% of samples were evaded within twenty queries on average across seven classifiers, three feature spaces, and two training periods; sample transfer succeeded in 43.3% of cases.Prior Android attacks achieved 65.2%–56.6%. REPLICANTWB reduced residual attack success below 17% against itself and APG, was more robust than AT-APG to unseen capabilities, and interleaving active learning (AL) with adversarial training (AT) maintained robustness and clean performance across three years of drifting samples.Source: §8 · p. 13

A Common Acronyms

• DRL means Deep Reinforcement Learning; MDP, Markov Decision Process; AML, Adversarial Machine Learning; PTE, Policy Transfer Evasion; and STE, Sample Transfer Evasion.Source: A Common Acronyms · p. 18

B Proximal Policy Optimization; B.1 Clipped Surrogate Objective

• PPO [66] collects on-policy rollouts and updates the policy over minibatches; its clipped surrogate limits policy drift and discourages destructively large updates.The probability ratio is clipped to [1 − ε, 1 + ε], forming a pessimistic lower bound on policy improvement.Source: B Proximal Policy Optimization · p. 18, §B.1 · p. 18

B.2 Generalized Advantage Estimation

• GAE estimates advantages as exponentially weighted TD residuals, with λ controlling the bias-variance trade-off and episode termination truncating the sum.λ = 0 gives a one-step, low-variance/high-bias estimate; λ = 1 approaches the low-bias/high-variance Monte Carlo return. Advantages are normalized within each minibatch.Source: §B.2 · pp. 18–19

B.3 Clipped Value Loss; B.4 Entropy Bonus; B.5 Combined Objective

• Clipped value loss bounds value-prediction changes; an entropy bonus preserves exploration, and PPO combines policy, value, and entropy terms into one loss.Entropy coefficient ce controls exploration, while clipped value loss acts as a pessimistic bound that improves stability when rewards are large or the value function has not converged.Source: §B.3 · p. 19, §B.4 · p. 19, §B.5 · p. 19

C Hyperparameters; C.1 REPLICANT

• REPLICANT uses PPO with CleanRL defaults and masks conflicting gadgets or those exceeding per-gadget budgets of 3 permissions and 60 classes.Its shared two-layer encoder feeds query, modification, and critic heads; agents train for 50 updates and evaluate by choosing the highest-probability action.Source: C Hyperparameters · p. 19, §C.1 · pp. 19–20

C.2 Classifiers

• The classifier baselines are RandomForest, LinearSVC, SecSVM, DeepDrebin, DRMD, HCC, and RAMDA, with model-specific training and architecture settings.Configurations range from classical classifiers to neural and contrastive-learning models; the passage specifies each model’s implementation and training parameters.Source: §C.2 · p. 20

C.3 Baselines

• Attacker baselines include APG, ADZ [35], EvadeDroid, and random action selection; all use REPLICANT’s problem-space constraints except ADZ’s reconstructed 662-action space.APG greedily selects capabilities, ADZ samples hierarchical perturbations and updates its tree from confidence changes, and EvadeDroid samples untried capabilities at random; EvadeDroid and Rnd [50] average five trials per seed.Source: §C.3 · p. 20

C.4 RL-Based Attacks

• MEME is PPO trained directly against the shared surrogate for 6,144 steps, with +10 for evasion and otherwise step-wise surrogate malicious-score reduction.The extraction stage is collapsed; the method uses hyperparameters tuned in the original paper and otherwise Stable Baselines 3 defaults.Source: §C.4 · p. 20
• MAB uses Thompson-sampled capability arms and leave-one-out pruning to credit useful capabilities, with results averaged over 5 trials per seed.Each arm begins with a Beta(α,β) posterior at α=β=1; capabilities whose removal breaks evasion receive α+1, while redundant ones receive β+1.Source: §C.4 · p. 20
• MERLIN [60], AIMED-RL, and PSP-Mal use DQN variants with evasion bonuses and confidence-, distance-, similarity-, or repetition-based reward shaping.MERLIN and AIMED-RL use 1,000-episode budgets; AIMED-RL rewards distance and file similarity while penalizing repeated capabilities, and PSP-Mal uses a surrogate-computed Shapley-style prior in prioritized replay.Source: §C.4 · pp. 20–21

D Scale of Experiments

• 379,680 attack evaluations make this, to the authors’ knowledge, the largest malware-evasion evaluation to date, designed to distinguish attack-strategy performance from capability-set or threat-model effects.Source: D Scale of Experiments · p. 21
• 352,800 attack evaluations covered Sections 5.1–5.2 across ten attacks, 1,764 surrogate/target combinations, two dataset variants, and ten seeds.The combinations span 7×7 classifiers, 3×3 feature spaces, and 2×2 training periods.Source: D Scale of Experiments · p. 21
• 10,080 attack evaluations, 560 hardened classifiers, and 8,400 adversarial-training rounds were produced for Sections 5.3–5.4.These experiments span seven classifiers, ten seeds, two training periods, two pipelines, two adversarial-training capability sets, and six attacker capability sets.Source: D Scale of Experiments · p. 21
• 16,800 attack evaluations tracked monthly performance in Section 5.5 across five detection pipelines, seven classifiers, two training periods, and ten seeds from 2022–2024.Source: D Scale of Experiments · p. 21

E Extended Evaluations

• Table 6 reports target-classifier FNRs on malware by period, averaged across seven classifiers, three feature spaces, and ten seeds.It uses 2021b targets for full dataset coverage, 2021a as the surrogate held-out set, and 2022–2024 as temporally split test years; HypercubeVTT4 lacks 2024 malware, so its mean covers 2021–2023.Source: Table 6 · p. 21, E Extended Evaluations · p. 21
• Table 7 compares attackers on HypercubeVTT4 using mean ASR and AvgQ in matched and all settings.It also reports per-target breakdowns for the matched setting.Source: Table 7 · p. 21, E Extended Evaluations · p. 21
• Table 8 reports attack ASR across test periods relative to target training year, averaged over targets trained in 2021b and 2022b.The +0 split is the surrogate training-year split, +1 is the main test setting, and +2/+3 test further into the future; +3 includes only 2021b-trained targets because no 2025 samples are available.Source: Table 8 · p. 21, E Extended Evaluations · p. 21
• The temporal gap is negligible, although the cross-period curves also show temporal luck.Figure 5 plots PTE ASR over target queries and compares it with STE across training-period pairs.Source: Figure 5 · p. 22
• Figures 6–8 break down PTE-versus-STE transferability curves by target classifier, feature-space pair, and target classifier under combined gaps.These breakdowns correspond respectively to the architecture, representation, and combined gap results aggregated in Table 2 and Figure 2.Source: E Extended Evaluations · p. 21, Figure 6 · p. 22, Figure 7 · p. 22

F RL Design Ablation

• The comparison adapts MEME, MAB, MRLN [60], AIM, and PSP from Windows PE to Android.The designs were modified to use Replicant’s environment, threat model, and capability set.Source: F RL Design Ablation · pp. 22–23
• Replicant remains the strongest attacker, with statistically significant gains across all settings.The comparison evaluated prior RL-based attacks adapted to the shared environment, threat model, and capability set.Source: F RL Design Ablation · p. 23
• Replicant achieves the highest evasion rate per query among the compared RL agent designs.Figure 9 reports attack efficiency across 1,764 surrogate/target combinations and ten seeds.Source: Figure 9 · p. 23

G Motivating Ablations

• ADZ’s mean ASR decreases with the full capability set, indicating it does not scale; even with the reduced 662-set, REPLICANT remains significantly stronger than ADZ [35].The comparison uses matched settings across seven classifiers, five seeds, and the 2021a→2021b training period in Drebin.Source: Table 10 · p. 23
• The surrogate/test overlap change is below 0.2 point in both 2022a→2021b settings.The ablation compares the full 2022 test set with a version excluding test samples present in the 2022a surrogate training split.Source: Table 11 · p. 23
• As targets harden, a constant AT-REPLICANT budget collapses monotonically, while doubling training length every five rounds continues finding AEs throughout 15 rounds.The two schedules are near-identical over the first five rounds.Source: Table 12 · p. 23