PaperlayerView exact revision on arXiv

Source-linked AI summary

PhantomCall: Evading ML Malware Detectors via Function Call Graph Perturbation

Md Ajwad Akil, Adrian Shuai Li, Imtiaz Karim, Arun Iyengar, Ashish Kundu, Elisa Bertino

arXiv:2609.00705v1cs.CR

TL;DR

Windows PE malware detectors have been attacked through bytes, headers, and intra-function graphs, but their function call graphs remain an underexplored surface. PhantomCall injects fully executable dummy functions at internal call sites and combines this perturbation with classifier-guided search. Across three detector architectures, it achieves 85-100% attack success while preserving malicious functionality in 86-97% of evaluated evasive samples.

  • Problem

    Prior Windows PE malware attacks target raw bytes, PE headers, overlays, or intra-function control-flow graphs, leaving the function call graph underexplored despite its use in graph-based detectors.

  • Method

    PhantomCall uses black-box classifier-guided search and tunable parameters to inject fully executable dummy functions at internal call sites, adding CFG and FCG structure while preserving semantics.

  • Results

    85-100% attack success rate is achieved across three architecturally distinct classifiers, with evasive variants generated up to 2.9× faster and 86-97% retaining malicious functionality.

  • Takeaways & Limitations

    FCG topology is a viable attack surface across raw-byte, CFG+FCG, and pure-FCG malware detectors, although effective injection parameters differ by architecture.

  • Takeaways & Limitations

    Injected functions use parameterless, structurally simple self-canceling or minimal bodies, motivating richer function bodies and attention-aware defenses as future work.

Abstract

from arXiv · show

Prior adversarial attacks on Windows PE malware detectors target raw bytes, PE headers, or intra-function control-flow graphs, leaving the function call graph (FCG) unexplored as an attack surface. Yet the FCG structure is an important feature in graph-based malware detectors. We present Phan- tomCall, a black-box attack that perturbs the FCG of Windows PE malware by injecting fully executable dummy functions at targeted call sites, adding new nodes and edges to both the CFG and FCG while preserving program semantics. We pair this structural perturbation with classifier-guided search and tunable injection parameters, effective across three archi- tecturally distinct classifiers. Evaluated on a 2025-collected Windows malware corpus against MalConv (raw-byte CNN), MalGraph (graph-based GNN), and SAFE+GNN (pure FCG GNN trained from scratch on a 2024 corpus) at two FPR thresholds, the best PhantomCall variant achieves 85-100% attack success rate across all configurations, exceeding prior state-of-the-art by up to 14.78 percentage points on MalGraph and 95.5 percentage points on SAFE+GNN, and generating evasive variants up to 2.9x faster on average across all targets. For MalConv and MalGraph, the majority of evasions require only a single call site modification, and 86-97% of evaluated evasive variants preserve the original malicious behavior in sandbox-based semantic testing across all configurations.

1 Introduction

PhantomCall addresses an underexplored FCG attack surface in Windows PE malware detectors by combining executable graph perturbations with classifier-guided search. Across three detector architectures, it achieves high evasion while preserving malicious functionality.

  • Prior attacks target raw bytes, PE headers, overlays, or intra-function control-flow graphs, leaving function call graphs comparatively unexplored.
  • 85-100% attack success rate is achieved across MalConv, MalGraph, and SAFE+GNN configurations at 0.1% and 1% FPR thresholds.The best variant exceeds prior state-of-the-art by up to 14.78 percentage points on MalGraph and 95.5 percentage points on SAFE+GNN.
  • Up to 2.9× faster generation is obtained while evaluating evasion rate and throughput across the three target classifiers.
  • 86-97% of evasive samples retain their original malicious functionality in semantic testing.
  • PhantomCall injects fully executable, reachable dummy functions and uses classifier-guided search with tunable injection parameters across architecturally distinct classifiers.

2 Background

This background defines ML malware detection and adversarial malware generation in the executable problem space. It distinguishes feature-space perturbations from runnable executable modifications that preserve malware behavior.

  • An ML malware detector maps a Windows PE executable to numerical features and outputs a maliciousness probability used for threshold-based classification.
  • Adversarial malware generation seeks a modified executable that is misclassified as goodware while preserving the original payload and runtime behavior.
  • Feature-space attacks perturb representations directly and may not produce valid executables, whereas problem-space attacks modify raw executables into runnable PE files.

3 Overview

PhantomCall is framed as a black-box, semantics-preserving problem-space attack against detectors whose feature extraction recovers FCGs from internal direct calls. Its design combines graph perturbation with classifier-guided search.

  • The threat model gives the attacker only classifier labels and maliciousness probabilities, while permitting semantics-preserving PE transformations.
  • The attack objective minimizes maliciousness probability subject to evasion below the classifier threshold and semantic equivalence between original and modified executables.
  • PhantomCall injects fully executable dummy functions through call-site displacement and a trampoline that preserves stack semantics while altering both CFG and FCG.

4 Detailed Design

PhantomCall patches internal call sites with trampolines that preserve original execution while adding executable dummy functions and graph structure. Search procedures then use classifier feedback to select injection configurations, with simulated annealing providing exploration beyond greedy local optimization.

  • FCG Perturbation: The trampoline calls the original target first, invokes k dummy leaf functions, and jumps back to the instruction following the displaced call.
  • FCG Perturbation: Internal direct call sites are selected because their uniform 5-byte PC-relative encoding can be safely replaced with an unconditional jump.
  • Dummy Leaf Functions: Dummy functions are register- and memory-neutral, with prologue/NOP/epilogue bodies for MalConv and MalGraph and minimal bodies for SAFE+GNN.
  • Graph Perturbation Effects: Each displacement adds k FCG nodes and k directed caller-to-dummy edges, while also introducing trampoline and dummy CFG basic blocks.
  • Call Site Selection: Random call-site selection remains the default because no structural ranking strategy consistently outperforms it across classifiers.
  • Classifier-Guided Search: Greedy search accepts only immediate classifier-score improvements, whereas simulated annealing can accept worse moves probabilistically to escape local minima.

5 Evaluation

PhantomCall consistently evades all three evaluated malware classifiers, with graph-based models benefiting most from FCG topology perturbation. Ablations show that search strategy, injection volume, and isolated topology govern the trade-off between evasion, footprint, throughput, and semantic preservation.

  • RQ1: 85-100% attack success rate is achieved across all classifier and FPR configurations, with PhantomCall matching or exceeding MalGuise in ASR and throughput.The largest gains occur on graph-based classifiers, while MalConv shows comparable ASR but PhantomCall is faster.
  • RQ2: 97.4% to 0.9% ASR at 0.1% FPR and 85.0% to 0.1% at 1% FPR when FCG edges are removed from SAFE+GNN attacks.MalGraph gains 7.37-10.15 percentage points, whereas MalConv gains at most 1 percentage point, confirming topology as the effective mechanism for the FCG-only classifier.
  • RQ2: ∼26-62 percentage points of ASR are gained from iterative NOP search across MalConv and MalGraph configurations.Without iterative search, fewer than half of samples evade at 1% FPR; for SAFE+GNN, Greedy reaches 85% overall ASR versus 64.5% for SA at 1% FPR.
  • RQ2: 65-80% of MalConv and MalGraph evasions use a single call-site modification, with median perturbed call sites of 1 and median byte overhead of 0.21-0.41%.SAFE+GNN requires median 2-4 sites and much larger injection volumes, producing 5.0% and 16.5% median byte overhead at 0.1% and 1% FPR.
  • RQ2: 3.8% to 84.2% ASR is observed for SAFE+GNN as k changes from 1/3 to 2,500/3,500, exposing a footprint-evasion trade-off.The larger volume follows SAFE+GNN’s global mean pooling, which requires enough injected nodes to shift the averaged representation.
  • RQ3: Isolated leaf injection outperforms DAG-constrained injection across all classifiers, while SA temperature is robust but MalGraph remains sensitive to NOP diversity.DAG chaining forms a recognizable connected subgraph, and semantic NOP variety is classifier-dependent.
  • RQ4: 86-97% of evaluated evasive variants preserve original malicious functionality in sandbox testing.Preservation rates are 93-97% for SAFE+GNN, 90-93% for MalGraph, and 86-87% for MalConv.

6 Related Works

Prior malware-evasion research spans API manipulation, CFG modification, binary-level changes, and component diversification, while FCG attacks have mainly targeted non-PE settings.

  • API-call attacks inject irrelevant or benign-looking calls using optimization, generative models, greedy strategies, or graph rewiring.
  • CFG-focused studies modify malware assembly-level control flow as an alternative attack surface.
  • Binary-level approaches inject or append bytes while maintaining functionality, and other methods modify headers, overlays, or diversify binaries semantically.
  • FCG attacks have been explored for Android, WebAssembly, and IoT malware through dead calls, function insertion, call-site redirection, or branch-function injection.
  • Managed runtimes and intermediate representations simplify calling conventions and stack management, whereas FCG-based detectors increasingly use call-graph structure as a primary discriminative feature.

7 Conclusion, Limitations, and Future Work

PhantomCall establishes executable FCG injection as a viable attack surface across diverse classifiers, while showing that current defenses reduce but do not eliminate its effectiveness. The paper identifies architecture-dependent injection effects and several directions for strengthening the attack and defenses.

  • Conclusion: 85-100% ASR is achieved across three architecturally distinct classifiers while 86-97% of evaluated samples preserve malicious functionality.The attack’s effective injection strategy differs by classifier architecture.
  • Conclusion: Instruction-level classifiers respond primarily to NOP content diversity, whereas less content-sensitive architectures are governed more by injection volume.This contrast identifies architecture-dependent factors shaping evasion effectiveness.
  • Conclusion: Adversarial finetuning and structural heuristic defenses reduce but fail to eliminate PhantomCall’s attack effectiveness.The paper therefore treats the FCG as a persistent attack surface requiring further defensive work.
  • Limitations and Future Work: Injected functions currently use parameterless, self-canceling or minimal bodies, making them structurally simpler than benign code.Future work includes enriching bodies with benign API calls, conditional branches, and local variables.
  • Limitations and Future Work: Attention-based graph classifiers and detectors distinguishing semantically inert injected functions from legitimate code are identified as promising defensive directions.The paper also proposes targeting learned attention weights as a future perturbation strategy.

Ethics Considerations

The evaluation used known MalwareBazaar malware samples in a static-only pipeline, and generated variants were not executed outside isolated environments, distributed, or publicly released.

  • Ethics Considerations: All experiments used known malware samples from MalwareBazaar under a static-only pipeline.The generated adversarial variants were not executed outside isolated environments, distributed, or publicly released.

A.1 SAFE+GNN: Architecture and Training Details

The SAFE+GNN adaptation converts a family-classification architecture into binary malware detection using x86-oriented preprocessing, IDA-based FCG extraction, and a 2024 corpus. Training retains the original model hyperparameters and evaluates a held-out test set.

  • Architecture and Training Details: SAFE+GNN originally classified malware families using FCG structure and learned function embeddings across MalwareBazaar and BIG-2015 datasets.Its reported F1-scores were 98.27% and 98.31%, respectively.
  • Architecture and Training Details: The adaptation changes the task from multi-class family classification to binary malware-versus-benign classification.The modified model has two output classes.
  • Architecture and Training Details: The original pretrained SAFE encoder targets AMD64 instructions, whereas the evaluation corpus contains x86 32-bit PE binaries with differing instruction representations.This ISA mismatch motivates the stated feature-extraction adaptation.
  • Architecture and Training Details: FCGs are extracted with IDA Home, and the model is trained and evaluated on 20,000 samples from a 2024-collected corpus.IDA Home is used for consistency with the MalConv and MalGraph evaluation pipeline.
  • Architecture and Training Details: The architecture retains three SAGEConv layers, global mean pooling, and a 64→2 classifier head with 29,506 parameters.Training uses AdamW, cross-entropy loss, batch size 128, and early stopping on validation F1-macro.
  • Architecture and Training Details: Table 7 reports SAFE+GNN test-set performance at epoch 100 on 2,000 samples.The held-out set contains 1,000 malware and 1,000 benign samples.
  • Architecture and Training Details: Assembly instructions are represented as Word2Vec tokens after normalization of comments, delimiters, and numeric constants.The CBOW embeddings use dimension 100, context window 2, and 100 training epochs.

A.2 PATCHATCALL Implementation Details

PATCHATCALL patches a selected call site by allocating a contiguous injected PE-section region containing a trampoline and sequential dummy function bodies. It computes target addresses from the image base, RVA, and accumulated body sizes, then encodes the resulting call and jump operands.

  • PATCHATCALL Implementation Details: PATCHATCALL patches call site c with semantic NOPs by allocating a contiguous region in a newly injected PE section.The region is specified by a Relative Virtual Address (RVA).
  • PATCHATCALL Implementation Details: The trampoline’s Virtual Address is computed as imagebase+RVA, with k dummy function bodies placed sequentially after it.Each dummy function address is obtained by accumulating the sizes of preceding bodies from the trampoline’s end.

A.3 Call-Site Ranking Strategies

Call-site ranking strategies prioritize candidate locations for perturbation using signals extracted from malware binaries before NOP search.

  • Two ranking strategies are considered: assembly-level instruction density and structural CFG centrality.

A.3.1 Instruction-Density Ranking

Instruction-density ranking scores assembly windows around call sites, filters inadequate or overlapping windows, and returns a ranked set of candidates for perturbation.

  • Assembly-level instruction-density signals prioritize call sites in classifier-sensitive regions.
  • Each call-site window uses a fixed-size instruction context centered on the call site.
  • Arithmetic, logic, transfer, and data-movement instructions contribute density features to each window’s score.
  • Windows below min_inst are discarded, while remaining windows are sorted by score and filtered with one-dimensional IoU-based non-maximum suppression.
  • The top-Kw non-overlapping windows determine the ranked candidate call-site set Casm.

A.3.2 Structural CFG Centrality Ranking

Structural and semantic design choices affect attack effectiveness differently across classifiers, while evaluation shows strong evasion, semantic preservation, and important defensive boundaries.

  • 72-74% of test families overlap with training, representing 95-97% of labeled test samples, so unseen-family novelty is unlikely to explain ASR.
  • Structural CFG Centrality Ranking: For MalGraph, Betweenness↑ gives the largest ASR gain (+3.2 pp at 1% FPR), while Degree↑ gives the best throughput gain (+6 byp/hr).
  • Structural CFG Centrality Ranking: For MalConv, ASR is largely insensitive to ranking strategy (±0.25 pp), but Eigenvector↓ increases throughput by +29 byp/hr at 1% FPR.
  • MalGraph ASR varies by less than 0.6 pp across the tested hyperparameter range, while MalConv remains unchanged; temperature mainly affects throughput.
  • Semantic NOP Variety Sensitivity: NOP variety is classifier-dependent: MalGraph loses 18.2 pp ASR at s=1, whereas MalConv reaches 100% ASR at its single-type default.
  • For MalGraph, k=1/1 reduces ASR by 11.5 pp, while k≥3 stabilizes performance; MalConv remains flat across k values.
  • Commercial engines achieve 14-18% ASR against PhantomCall, whereas ClamAV reaches 74-75% because modified regions disrupt byte-pattern signatures.
Loading 2609.00705v1…