PaperlayerView exact revision on arXiv

Source-linked AI summary

Windows Malware Detector as a Compound AI System: Trade-Offs in Accuracy, Efficiency, and Adversarial Robustness

Andrea Ponte, Luca Demetrio, Luca Oneto, Battista Biggio, Fabio Roli

arXiv:2609.08394v1cs.CR

TL;DR

Industrial Windows malware detectors combine multiple components, but their secrecy and the focus of academic work on isolated models leave their system-wide trade-offs difficult to evaluate. This paper proposes a methodology for evaluating performance, computational requirements, and robustness together, with threat models spanning attacker knowledge. On real-world data, filtering reduces resource use with only a marginal performance decrease, while the threat models expose a trade-off between efficiency and robustness.

  • Problem

    Industrial secrecy limits systematic evaluation of Windows malware systems, while academic studies typically assess individual components rather than integrated systems across performance, cost, and robustness.

    Source: §I · p. 1

  • Method

    The paper evaluates a layered Compound AI System using filtering to balance performance and computational requirements, alongside system-level threat models for attackers with varying knowledge.

    Source: §I · p. 2, §III · p. 3, B. Evaluating Robustness of Compound… · pp. 4–5

  • Results

    Filtering significantly reduces resource consumption with a marginal performance decrease, while more knowledgeable attackers mount more effective evasion attacks that degrade responsiveness.

    Source: §I · p. 2

  • Takeaways & Limitations

    Efficiency gains and robustness losses stem from the same filtering mechanism, and the paper translates the trade-offs into deployment guidelines for operational constraints.

    Source: §VII · p. 12

    The reported limits of dynamic analysis reflect the current state of the art, not artifacts of the chosen model.

    Source: §VI · p. 11

Abstract

from arXiv · show

Industrial Windows malware detectors are commonly described as Compound AI Systems composed of multiple heterogeneous components, including rule-based mechanisms as well as machine-learning-based static and dynamic analyses. However, due to industrial secrecy and limited public disclosure, the internal architectures of these systems can only be inferred, rendering systematic evaluations of detection accuracy, computational costs, and adversarial robustness largely infeasible. In contrast, academic research provides reproducible and transparent evaluation methodologies, but typically investigates individual detection components in isolation. To bridge the gap between academic research and industrial practice, and inspired by state-of-the-art industrial architectures for Windows malware detection, we propose a novel methodology that (i) explicitly balances the trade-off among detection performance, computational requirements, and robustness, and introduces (ii) system-level threat models that capture how attackers exploit different degrees of knowledge to evade the entire Compound AI System rather than isolated detectors. Experiments conducted on real-world data demonstrate that the Compound AI System training time can be reduced and responsiveness improved while incurring only a marginal loss in detection performance. Leveraging our threat modeling, we show that increasingly knowledgeable attackers craft more effective adversarial examples, revealing the system's strengths and weaknesses, degrading its responsiveness, and exposing a direct trade-off between efficiency and robustness. Finally, we translate these trade-offs into take-home messages and deployment guidelines, helping practitioners to select the system that best matches their operational constraints.

I. INTRODUCTION

Industrial Windows malware detection combines layered signatures and machine-learning analyses, but secrecy and component-focused academic evaluations leave whole-system trade-offs in performance, cost, and robustness insufficiently studied. The paper proposes an integrated evaluation methodology and system-level threat models to address this gap.

I. INTRODUCTION

  • Industrial Windows malware pipelines are commonly assumed to sequence signatures, static ML, and dynamic analysis, but their internal structures can only be partially inferred.Signatures identify known samples; static analysis uses code and metadata without execution, while dynamic analysis assesses runtime behavior.Source: §I · p. 1
  • The paper identifies a gap: industrial systems are secretive, while academic evaluations largely study isolated detectors rather than integrated systems and their joint trade-offs.Existing evaluations also overlook computational requirements and system-wide evasion by adversarial EXEmples.Source: §I · p. 1
  • The proposed methodology jointly considers detection performance, computational requirements at design and deployment, and robustness, including attackers with varying system knowledge.It focuses on feasible attacks against pipeline components for which practical attack techniques are available.Source: §I · p. 2
  • Earlier filtering can reduce downstream training data and deployment costs, but may also reduce accuracy or robustness; optimized forwarding thresholds offer a way to balance performance and computation.The paper reports that filtering reduces resource consumption with a marginal performance decrease, while also exposing a trade-off with robustness.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

  • Prior work on Windows Compound AI Systems was an initial step, but had not explicitly addressed joint trade-offs or system-level threat models spanning attacker knowledge.Windows programs use the Portable Executable format, whose headers and sections contain metadata, loading information, and program content.Source: §II · p. 2

III. COMPOUND AI SYSTEMS FOR WINDOWS MALWARE DETECTION

The paper frames Windows malware detection as a sequential system in which signatures, static analysis, and dynamic analysis filter samples for later stages. This architecture can save resources, but its interdependent components create design and failure trade-offs.

A. Compound AI System Paradigm

  • Real-world Compound AI Systems typically sequence signatures, static metadata analysis, and dynamic behavioral analysis, filtering samples before more costly stages.The levels depend on one another because each filters and refines the data passed downstream.Source: A. Compound AI System Paradigm · p. 3
  • Filtering may benefit shallow models but can disadvantage deep neural networks that need large datasets; system designers must also define labels for upstream failures.The system must return a decision even when preprocessing fails or a sandbox crashes, accepting associated false-alarm or detection-rate trade-offs.Source: A. Compound AI System Paradigm · pp. 3–4
  • Signatures provide fast pattern matches for known samples, but do not natively tune detection and false-alarm trade-offs; they can also constrain downstream training data.Blocklists flag matches as malicious, while allowlists of known benign hashes help mitigate false alarms.Source: A. Compound AI System Paradigm · p. 3
  • Static ML models require feature extraction and expensive training, while probabilistic thresholds determine whether uncertain samples proceed to dynamic analysis.Training on samples remaining after signatures can reduce data volume and cost while better matching the filtered deployment distribution.Source: A. Compound AI System Paradigm · p. 4
  • Dynamic analysis runs samples in a sandbox and models runtime traces, but expensive neural processing and large data needs make upstream filtering attractive.Behavioral analysis is challenging because traces are noisy, behaviors overlap, and malware can obfuscate or evade sandboxes.Source: A. Compound AI System Paradigm · p. 4

B. Evaluating Robustness of Compound AI Systems.

The section argues that isolated detector evaluations can misstate the robustness of a Compound AI System, motivating threat models that represent different attacker knowledge. The paper evaluates a practical subset of those models using OBELISK.

B. Evaluating Robustness of Compound AI Systems.

  • Isolated evaluations can understate or overstate whole-system robustness because upstream components may alter attacks, while attackers can target samples that evade the complete pipeline.The paper therefore models attackers with progressively greater knowledge while keeping evasion goals and test-time functionality-preserving manipulations consistent.Source: B. Evaluating Robustness of Compound… · pp. 4–5
  • The four threat models range from TM1, where the attacker lacks system knowledge, to TM4, where the attacker knows the signatures and static model.TM2 assumes knowledge of signatures; TM3 assumes knowledge of the static model; TM4 combines both, without knowledge of deployed thresholds.Source: Figure 3 · p. 5, B. Evaluating Robustness of Compound… · p. 5
  • The evaluation focuses on transfer-based attacks because querying the full pipeline is costly, and treats obfuscation and packing as outside its scope.The section describes transfer, query-based, and obfuscation or packing attacks, but excludes the latter and selects transfer-based evaluation.Source: B. Evaluating Robustness of Compound… · p. 5
  • The study restricts threat models to four feasible cases because stable tools for attacks on dynamic analysis are lacking, deferring those attacks to future work.The authors cite unavailable, deprecated, or feature-space-only implementations as constraints on evaluating dynamic-analysis attacks.Source: B. Evaluating Robustness of Compound… · p. 5

IV. IMPLEMENTATION DETAILS

  • OBELISK implements the system and attacks used to quantify trade-offs among performance, computational requirements, and robustness.The implementation and attacks are publicly available.Source: §IV · p. 5

A. OBELISK Implementation

OBELISK implements signature, static, and dynamic detection levels, then varies training and inference filtering. Its attacks adapt to the four threat models by changing what the attacker knows about signatures and the static model.

A. OBELISK Implementation

  • OBELISK combines YARA signature matching, an EMBER-feature GBDT static model, and a Nebula transformer trained on Speakeasy-generated runtime reports.Signature rules producing any false alarm on training data are discarded.Source: A. OBELISK Implementation · pp. 5–6
  • Static and dynamic models are each trained either on the full dataset or on samples remaining after earlier filtering levels.The filtered dynamic setting retains samples in the static model's uncertainty interval, defined by the two cutoffs δ and 1−δ.Source: A. OBELISK Implementation · p. 6
  • At inference, signature matches receive immediate labels, static scores outside the uncertainty interval receive labels, and the remaining samples go to dynamic analysis.The cutoffs are p ≤ δ for benign and p ≥ 1−δ for malicious.Source: A. OBELISK Implementation · p. 6

B. Attacks Implementation

  • GAMMA injects non-executable content from benign programs, while PAD appends bytes to the overlay; both are used to generate attacks for each threat model.TM1 uses a surrogate; TM2 avoids signature artifacts; TM3 uses the deployed static model; TM4 combines signature-aware attacks with that model.Source: B. Attacks Implementation · p. 6
  • Attacks against TM3 and TM4 are generated for both full-data and YARA-filtered GBDT variants, as well as for the applicable attack techniques.The variants distinguish GBDT-ALL from GBDT-YARA.Source: B. Attacks Implementation · pp. 6–7

V. EXPERIMENTAL ANALYSIS

The experimental analysis reports the setup, examines performance and computational requirements, and evaluates robustness and computational requirements. It also includes an analysis of detection levels and summarizes findings as take-home messages and deployment guidelines.

V. EXPERIMENTAL ANALYSIS

  • The analysis first details its experimental setup.It then reports findings on performance versus computational requirements, robustness versus computational requirements, and the contribution of individual detection levels.Source: §V · p. 7
  • The paper summarizes its findings and analysis in take-home messages and practical deployment guidelines.Source: §V · p. 7
  • The reported trade-offs cover performance and computational requirements, as well as robustness and computational requirements.Source: §V · p. 7

A. Experimental Setup

The setup uses a malware-and-goodware PE dataset, defines 37 system configurations and their evaluation metrics, and specifies model training and adversarial attack procedures.

A. Experimental Setup

  • The final PE dataset contains 71,505 malware and 28,707 goodware training samples, and 17,495 malware and 19,809 goodware test samples.Windows system files are included only in training, while Chocolatey goodware samples collected in 2024 are reserved for testing.Source: A. Experimental Setup · p. 7
  • The static level uses a 1,000-tree GBDT, while Nebula is trained on up to 512-token sequences for 60 epochs.After removing failed and duplicate emulation reports, the training data contain 67,611 unique reports; Nebula uses a 50,000-token BPE vocabulary.Source: A. Experimental Setup · p. 7
  • The attacks use GAMMA variants and PAD, with up to 500 queries and a population of 10 against a shared pool of 700 malware samples.The pool contains 100 test samples from each of seven malware families; A1–A4 vary the attacker’s target model and whether YARA is included.Source: A. Experimental Setup · pp. 7–8
  • The 37 evaluated systems comprise STND and twelve threshold configurations each for OBV1, OBV2, and OBV3.STND uses YARA, GBDT-ALL, and NEBULA-ALL; OBV1 adds two inference thresholds, OBV2 uses GBDT-YARA, and OBV3 also trains Nebula on filtered reports.Source: A. Experimental Setup · p. 8
  • Evaluation measures TPR, FPR, F1, transfer rate, training time, and mean inference time.Training time includes feature and report preparation as well as model training; inference time sums the pipeline stages used for a prediction.Source: A. Experimental Setup · p. 8

B. Performance vs Computational Requirements

The performance–cost evaluation finds that OBV3 with δ11 offers a strong training-time trade-off, while OBV1 with δ11 is best on inference performance. ROC and Pareto comparisons evaluate configurations at an operating point near 1% FPR.

B. Performance vs Computational Requirements

  • The systems are compared at an operating point near 1% FPR, where increasing δ improves TPR as filtering becomes more aggressive.The ROC curves show configurations of OBV1, OBV2, and OBV3, with STND as the black dashed-line reference.Source: B. Performance vs Computational… · p. 8, Figure 4 · p. 9
  • OBV3 is the fastest family of systems to train; Fig. 5 presents Pareto fronts comparing performance with training time (Tt) and inference time (Ti) across all 37 Compound AI Systems.Source: B. Performance vs Computational… · p. 8, Figure 5 · p. 9
  • STND, OBV1, and OBV2 require at least 100 hours of training, dominated by data preparation rather than model fitting.Feature extraction takes about 0.26 seconds per sample, and emulation about 8.1 seconds per sample.Source: B. Performance vs Computational… · p. 8
  • OBV1 with δ11 is the best system for inference-time performance, while OBV3 with δ11 nearly matches it at comparable inference cost.Source: B. Performance vs Computational… · p. 8, Figure 5 · p. 9
  • OBV3 with δ11 is identified as the best balance of performance and computational cost when only those two aspects are considered.Goodware traverses the full pipeline and incurs feature extraction and emulation, making its inference time an order of magnitude slower than competitors.Source: B. Performance vs Computational… · p. 9

C. Adversarial Robustness vs Computational Requirements

Under attack, STND is the most robust system across almost all threat models, while greater attacker knowledge increases transfer rate. Attacks also increase inference time, coupling robustness with responsiveness.

C. Adversarial Robustness vs Computational Requirements

  • STND is most robust across almost all threat models, whereas OBV3 is the least robust system family.The robustness comparison uses worst-case TPR and transfer rate across 37 systems.Source: C. Adversarial Robustness vs… · p. 9, Figure 6 · p. 10
  • OBV2 attains its best robustness at intermediate values δ6 and δ7; OBV3 is most robust at δ6, with an average TPR drop of less than 10% relative to STND.Source: C. Adversarial Robustness vs… · p. 9
  • Transfer rate grows as attacker knowledge increases from TM1 to TM4, as attackers can tailor attacks to deployed components rather than a surrogate.Source: C. Adversarial Robustness vs… · p. 9
  • Attacks inflate inference time by enlarging inputs and routing some evasive samples to the costly dynamic-analysis stage.Responsiveness can degrade even when attacks do not fully defeat robustness.Source: C. Adversarial Robustness vs… · pp. 9–10
  • STND is reported as the best robustness–computational-requirements choice under attack because it avoids a filtering threshold and trains its levels on more data.Source: C. Adversarial Robustness vs… · p. 10

D. Level-wise Analysis

Level-wise results show that filtering shifts decisions toward static analysis, while the strongest system delegates many benign samples to dynamic analysis. Under attack, signatures, filtering thresholds, and dynamic-model training determine whether adversarial samples are stopped across the pipeline.

D. Level-wise Analysis

  • ≈80% of the test set is resolved by the static level under filtering, while STND delegates most benign samples to dynamic analysis, which accounts for ≈47% of correctly classified samples.Fig. 7 separates correct and misclassified test-set predictions by the level making the final decision.Source: D. Level-wise Analysis · p. 10, Figure 7 · p. 11
  • For STND, signatures stop ≈75% of attacks under A1 and A3 but ≈25% under A2 and A4, where attackers avoid rule-triggering artifacts.Fig. 8 shows the level-wise prediction breakdown under attacks A1–A4.Source: D. Level-wise Analysis · p. 10, Figure 8 · p. 11
  • An overly wide benign region lets attacks bypass the static level, while OBV3 with δ11's minimally trained dynamic model stops virtually no attacks after the first two levels are bypassed.The failed residual defense is associated with the model trained on the smallest number of reports.Source: D. Level-wise Analysis · p. 10
  • Robustness is not additive across levels: aggressive filtering removes decisions from levels best placed to catch unknown samples and gives attackers a shortcut through the pipeline.The attack breakdown shows distinct vulnerabilities in signature coverage, the benign-region threshold, and dynamic models trained on less data.Source: D. Level-wise Analysis · p. 10

E. Final Recommendations

The recommendations match system configurations to operational constraints across training cost, inference speed, performance, robustness, and latency. The recommended options range from a balanced compromise to the most accurate and robust system when seconds-per-sample decisions are tolerable.

G1

  • Choose the system with all levels trained on filtered data and inference-time filtering for a balanced compromise that excels at none of the metrics but avoids collapse on any..Source: E. Final Recommendations · p. 11

G2

  • Deploy OBV3 with δ11 when training and inference speed are constrained: it is fastest to train and among the fastest at inference, reaching ≈96% TPR at ≈1% FPR, but sacrifices robustness.Its wide benign region enables static-level evasion, and its dynamic level provides no residual defense after training on the fewest reports.Source: E. Final Recommendations · p. 11

G3

  • Deploy OBV1 with δ6 when training resources are abundant but deployment resources are constrained; it combines strong performance, robustness close to STND, and fast inference at the price of training cost..Source: E. Final Recommendations · p. 11

G4

  • Deploy STND when performance and robustness outweigh latency: it is the most accurate and robust system evaluated, but benign programs traverse the full pipeline and decisions take seconds per sample..Source: E. Final Recommendations · p. 11

VI. LIMITATIONS

The limitations bound the evaluation's dynamic-analysis, adversarial-attack, packing and obfuscation, data-source, and temporal-drift coverage. In particular, attacks target static models, while transfer attacks provide a lower bound on attack effectiveness and an upper bound on measured robustness.

VI. LIMITATIONS

  • Dynamic-analysis limits are framed as limits of the current state of the art, not artifacts of choosing Nebula.The authors describe Nebula as one of the best-performing deployable dynamic malware architectures, while noting its large training-set and emulation-time requirements.Source: §VI · p. 11
  • Robustness was evaluated only against static-model attacks, and transfer attacks imply an upper bound on measured system robustness rather than an end-to-end attack assessment.Runtime-behavior manipulations and query-based attacks against the full system were not evaluated; the authors expect runtime attacks to make robustness figures optimistic.Source: §VI · p. 11
  • Packing and obfuscation were not evaluated separately or controlled for, and their prevalence in the vendor-collected Speakeasy data was not quantified.The evaluation relies on the dataset authors' report that packed samples are included.Source: §VI · p. 11
  • The methodology needs raw executables for attacks and emulation, while access to some large datasets requires paid VirusTotal retrieval.The authors say the methodology can be replicated on any dataset that provides raw executables.Source: §VI · pp. 11–12
  • Temporal drift was not explicitly evaluated: the ordered train/test splits do not substitute for a longer observation window, and first-observed timing data were unavailable.Establishing when samples first appeared in the wild would require vendor telemetry or VirusTotal access, neither viable in this study.Source: §VI · p. 12

VII. FUTURE WORK AND CONCLUSIONS

The paper evaluates OBELISK as a Compound AI System and uses system-level threat models to examine trade-offs among detection performance, computational requirements, and robustness. Filtering reduces training costs and improves responsiveness with little loss in detection performance, while the conclusions identify where robustness is retained or lost and translate those trade-offs into deployment guidance.

VII. FUTURE WORK AND CONCLUSIONS

  • Future work targets richer behavioral representations, alternative emulators and virtualization, malware-family classification, direct dynamic-detection evasion, and training-data poisoning.The authors also plan to investigate whether evading signature and static levels can systematically inflate inference time enough to overload dynamic analysis as a denial-of-service attack.Source: §VII · p. 12
  • Filtering policies substantially reduce training cost and improve responsiveness while sacrificing little detection performance.The experiments compare OBELISK configurations using different filtering policies at training and/or test time against a standard version without filtering.Source: §VII · p. 12
  • Better-informed attackers craft adversarial examples that transfer more effectively to the complete system and degrade responsiveness as a side effect.The paper examines four system-level threat models.Source: §VII · p. 12
  • Signature defenses collapse when disclosed, aggressive filtering enables bypass by evading the static level, and dynamic analysis helps only marginally when trained on the full dataset.The level-wise analysis identifies where robustness comes from and where it is lost.Source: §VII · p. 12
  • The authors translate the efficiency–robustness trade-offs into four deployment guidelines that map operational constraints to a suitable configuration.They describe the work as a first step toward more realistic Compound AI System evaluation that narrows the gap between industrial practice and academic research.Source: §VII · p. 12
Loading 2609.08394v1…