PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the Black Box: Byte-Level Boundary Inference of Real-World Antivirus Systems

Jieshuai Yang, Zhi Wang, Yan Jia, Zhenhua Wu, Jianfei Tang, Chenbin Su, Jingwei Ye, Jianwen Tian, Wanpeng Li

arXiv:2609.31012v1cs.CR

TL;DR

Existing black-box AV surrogates reproduce binary decisions but offer limited insight into the byte-level regions governing detection. AVHunter probes AVs to construct boundary-labeled data and train AV-specific models; it achieves high boundary recall and detection agreement, while its scope is constrained by static detection and the dataset’s coverage.

  • Problem

    Black-box AV approaches infer binary malware/benign decisions, providing limited insight into fine-grained byte-level boundaries, while coarse features may not recover those regions.

    Source: §I · p. 1

  • Method

    AVHunter probes malware to identify decision-critical bytes, expands them into contextual boundary regions, and trains AV-specific AV-DBNet models to predict decisions and boundary probabilities.

    Source: §I · p. 2, §III · p. 3

  • Results

    85.07% average boundary prediction recall and 97.43% detection agreement were achieved across 11 real-world AV products.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results show that fine-grained AV boundary knowledge can be recovered from black-box interactions and validated through boundary-guided evasion, false-positive induction, and temporal evaluation.

    Source: §VII · p. 11

    AVHunter focuses on static detection boundaries and samples with fewer than 20 boundary points, which may bias evaluation toward simpler detection logic.

    Source: A. Limitation and Future Work · p. 10

Abstract

from arXiv · show

Existing approaches for understanding the detection logic of real-world antivirus (AV) software infer only binary malware/benign decisions from black-box queries, providing limited insight into the fine-grained decision-critical regions that govern AV detection. In this paper, we present \textbf{AVHunter}, the first framework for inferring byte-level decision-critical regions of real-world AV products under a black-box threat model. AVHunter constructs the first large-scale Byte-Level AV Boundary Dataset (BABD) by systematically probing 11 real-world AV products, revealing that modern AV detections are largely associated with a small number of compact decision-critical byte regions. Leveraging BABD, AVHunter trains AV-specific models that not only reproduce binary AV decisions, but also localize the decision-critical byte regions underlying these decisions, achieving an average boundary prediction recall of 85.07% while maintaining 97.43% detection agreement with the target AVs. We further validate that the predicted regions capture genuine AV decision knowledge through boundary-guided malware evasion, false-positive induction on benign executables, and a seven-month longitudinal study demonstrating that the inferred regions remain largely stable as AV products evolve. Overall, AVHunter moves beyond conventional binary-label AV modeling by enabling fine-grained boundary-region localization and revealing a new form of AV knowledge leakage with important implications for malware analysis, AV security, and boundary-aware defenses.

I. INTRODUCTION

Black-box AV surrogates can reproduce malware/benign decisions without recovering the byte-level regions that drive them. AVHunter combines boundary probing with AV-specific modeling to localize those regions while reproducing AV decisions.

I. INTRODUCTION

  • Prior surrogate approaches learn from binary feedback or coarse-grained features, so high agreement or successful evasion does not establish recovery of byte-level AV boundaries.Real-world AVs combine detection sources, while many decisions remain tied to discrete static patterns; existing approaches observe only binary outcomes and miss fine-grained boundary information.Source: §I · p. 1
  • BABD addresses the challenge of turning sparse, critical byte patterns into learnable supervision without access to AV internals.The second challenge is isolating weak boundary signals within large binary inputs rather than producing noisy or overly coarse estimates.Source: §I · pp. 1–2
  • AVHunter perturbs malware to identify decision-critical bytes, expands them into contextual boundary regions, and trains AV-DBNet to predict AV decisions and boundary probabilities.AV-DBNet uses a multi-scale convolutional encoder, attention fusion, and a decoder; its boundary predictions support tasks including evasion analysis and AV detection-logic analysis.Source: §I · p. 2
  • 85.07% average boundary prediction recall and 97.43% detection agreement show AVHunter localizes decision-critical regions while reproducing target AV decisions.The evaluation spans 11 real-world AV products; boundary-guided evasion achieves an average evasion rate of 83.78%, and detection agreement decreases by only 1.59% over seven months.Source: §I · p. 2
  • BABD is the first large-scale byte-level AV boundary dataset, spanning 11 real-world products and revealing that detections often depend on a few compact critical regions.The dataset supplies a benchmark for AV boundary-region localization.Source: §I · p. 2
  • AVHunter’s evaluation validates predicted regions through malware evasion, false-positive induction, temporal robustness, and comparison with binary-feedback-only approaches.The supplied passages identify these validation dimensions but do not report the comparative results for each one.Source: §I · p. 2

II. PRELIMINARIES AND THREAT MODEL

The paper defines AV decision boundaries through discrete patterns and their critical byte positions, then explains why coarse surrogate behavior can diverge from those boundaries. Its threat model assumes black-box access to Windows AVs’ static detection interfaces and limited-boundary malware samples.

1) Decision boundary:

  • 93.26% of evaluated samples across 11 AV engines have fewer than 20 boundary points, although hybrid detection components can make precise inference harder.The paper defines a decision boundary as discrete patterns such as signatures or rules, with boundary points as critical byte positions and boundary regions as their surrounding context.Source: 1) Decision boundary · p. 2

2) Boundary Inconsistency:

  • Coarse surrogates can miss the true decision-critical regions: byte-sensitive modifications may disrupt AV detection even when the surrogate still labels the sample malicious.Conversely, injection-based attacks may mislead a surrogate while the AV continues detecting the sample if its original detection patterns remain intact.Source: 2) Boundary Inconsistency · pp. 2–3

B. Threat Model

  • Boundary-guided modifications use inferred boundary regions to obtain higher evasion rates than generic methods without localization, while supporting high agreement and consistency under boundary-sensitive perturbations.The threat model assumes Windows-based AVs that return binary hard labels for static detection, with no access to signatures, models, or confidence scores; it focuses on samples with a limited number of boundary points.Source: 2) Boundary Inconsistency · p. 3, B. Threat Model · p. 3

III. APPROACH

AVHunter probes real-world AVs to identify byte-level decision boundaries, then trains AV-specific AV-DBNet models using the resulting annotations. Its probing procedure locates boundary points and expands them into context-preserving regions.

III. APPROACH

  • AVHunter probes malware against 11 target AV products to build BABD, then trains an AV-specific AV-DBNet model for each product.Each model produces a binary classification score and a boundary probability distribution.Source: §III · p. 3

1) Probing Procedure:

  • Iterative binary search masks file suffixes with 0x90; when the AV classifies a masked sample as benign, the search narrows to a decision-critical byte.Tail-oriented masking preserves the PE header and critical metadata in the prefix, reducing false benign outcomes caused by structural corruption.Source: 1) Probing Procedure · pp. 3–4
  • After finding a boundary point, AVHunter masks it and repeats probing until no further point is found, then merges adjacent points into compact annotations.Each point is expanded to a boundary region spanning 30 preceding and 10 following bytes, retaining context for surrogate learning.Source: 1) Probing Procedure · p. 4

2) Sample Collection:

BABD combines malware and benign executables annotated for 11 AV products. Most evaluated samples have fewer than 20 boundary points, and probing points overlap recovered ClamAV signature regions in most samples where ground truth is available.

2) Sample Collection:

  • BABD contains 241,070 malware and benign samples and 2,651,770 byte-level boundary annotations from 11 AV products.Samples come from BODMAS, SOREL-20M, an internal VirusShare/VirusTotal-derived corpus, commercial PE software, and Windows system executables; files are capped at 1 MB.Source: 2) Sample Collection · p. 4

3) Boundary-Point Cardinality Analysis:

4) Ground-Truth Validation:

  • 97.39% of samples with recoverable ClamAV signature regions show overlap with AVHunter’s probing bytes.Ground-truth regions were recovered for 96.64% of samples; the remaining non-overlap may reflect multiple signatures, compound logic, or contextual bytes rather than exact signature regions.Source: 4) Ground-Truth Validation · pp. 4–5

B. Training AV-DBNet

AV-DBNet combines multi-scale feature extraction, attention, and dual-output decoding to predict AV-specific byte boundaries and malware classifications. Training uses perturbation augmentation and a composite loss to address robustness and sparse boundary labels.

1) Data Augmentation and Boundary Alignment:

  • Training augmentation simulates byte changes, layout shifts, and benign-feature injection, with boundary labels aligned to each transformation.Random byte perturbation and benign section injection preserve boundary labels; random sequence injection shifts offsets and requires displacement-based alignment.Source: 1) Data Augmentation and Boundary… · p. 5

2) Model Architecture:

  • AV-DBNet uses a multi-scale convolutional encoder, attention fusion, and a decoder with boundary-prediction and classification heads.The classification head fuses global raw-byte features with features from predicted boundary regions.Source: 2) Model Architecture · p. 5

3) Loss Function:

  • A composite loss combines focal loss and an adaptive spatial penalty for boundary prediction with classification loss for malware decisions.Focal loss addresses boundary-positive bytes below 1%; the spatial penalty discourages false positives outside boundary regions.Source: 3) Loss Function · p. 5

4) Model Outputs:

AV-DBNet outputs a binary classification score and byte-level boundary probabilities, which are thresholded and grouped into predicted regions. The evaluation is organized around boundary prediction, binary-feedback baseline comparisons, temporal robustness, and ablations.

4) Model Outputs:

  • AV-DBNet produces a binary classification score and a byte-level boundary probability distribution for each executable.Thresholding the boundary probabilities and merging contiguous positive bytes yields predicted boundary regions.Source: 4) Model Outputs · p. 5

IV. EVALUATION

  • The evaluation asks how well AVHunter predicts boundary points, compares with binary-feedback-only baselines, remains stable over time, and benefits from boundary signals.Figure 4 depicts probability distributions; peaks above τp = 0.5 that overlap ground-truth regions indicate precise predictions.Source: §IV · p. 6, Figure 4 · p. 6

A. Evaluation Setup

The evaluation uses AV-specific surrogate training and testing across 11 real-world AV products, comparing AVHunter with binary-feedback surrogates and coarse-grained evasion baselines. It assesses detection agreement, ground-truth accuracy, and several measures of boundary coverage and compactness.

A. Evaluation Setup

  • BABD is split 5:1 by malware type, with a dedicated AV-DBNet trained and evaluated for each AV product.The 11 targets comprise ClamAV and 10 commercial products; evaluations use default configurations and no query-rate limits during the subscription period.Source: A. Evaluation Setup · p. 6, 1) Target Antivirus · p. 6

1) Target Antivirus:

  • The evaluation covers 11 Windows-compatible AV products with command-line interfaces selected for reliable detection behavior consistent with their GUIs.Offline command-line tools are used for IkarusT3 and McAfee, whose desktop clients lack command-line interfaces.Source: 1) Target Antivirus · p. 6

2) Baseline Methods:

  • FFNN-TL and dualFFNN serve as binary-feedback surrogate baselines, while GAMMA and MalGuise serve as coarse-grained modification baselines.The surrogate baselines use EMBER features; all methods operate without access to internal detection logic or byte-level boundaries.Source: 2) Baseline Methods · p. 6

3) Evaluation Metrics:

  • Detection Agreement measures surrogate-target consistency, while Accuracy measures surrogate correctness against ground-truth labels.Boundary Prediction Recall and Precision quantify boundary-point coverage and overlap, while Boundary Region Coverage Ratio measures predicted-region byte compactness.Source: 3) Evaluation Metrics · p. 6

4) Experimental Setup:

Experiments train one AV-DBNet surrogate per target AV and evaluate its byte-level boundary predictions. AVHunter achieves high recall across the 11 products while keeping predicted regions compact.

4) Experimental Setup:

  • Each target AV receives one AV-DBNet surrogate trained for 50 epochs, with the checkpoint selected by lowest validation loss.Experiments use Windows 11, an NVIDIA GeForce RTX 4080 SUPER GPU, and an Intel Core i7-14700K CPU.Source: 4) Experimental Setup · p. 6

1) Boundary Prediction:

  • 85.07% average Boundary Prediction Recall across 11 AV products, with recall above 75% for every target.Recall exceeds 85% for six products; the paper treats BPR as primary because it measures how much of the true decision boundary is captured.Source: 1) Boundary Prediction · p. 7, Table II · p. 7
  • 0.05% average Boundary Region Coverage Ratio indicates that predicted boundary regions occupy a very small fraction of input bytes.Average Boundary Prediction Precision is 64.29% and average F1 is 72.70%; the authors attribute the lower precision mainly to additional predicted bytes that may belong to genuine but unprobed regions.Source: 1) Boundary Prediction · p. 7

2) Validating Boundary Predictions via AV Evasion:

Boundary-guided BR Encryption substantially outperforms generic feedback-driven modification in evasion testing, while using smaller perturbations and preserving functionality. Injection tests also indicate that predicted regions contain discriminative detection patterns for most evaluated AV products.

2) Validating Boundary Predictions via AV Evasion:

  • 83.78% average evasion rate for BR Encryption across 11 AV products, versus 20.87% for MalGuise and 9.43% for GAMMA.BR Encryption consistently outperforms both generic methods; the results support that inferred regions closely match decision-critical AV content.Source: 2) Validating Boundary Predictions via… · p. 7
  • 3.22% average perturbation rate for BR Encryption, compared with 11.31% for MalGuise and 345.37% for GAMMA.BR Encryption encrypts only predicted boundary-region bytes, with most overhead coming from the inserted decryption routine; its functionality preservation rate on 48 known samples is comparable to or higher than the generic baselines.Source: 2) Validating Boundary Predictions via… · pp. 7–8

3) Validating Boundary Predictions via BR Injection:

  • 81.33% average false positive rate across nine AV products followed injection of predicted regions into benign executables, at an average perturbation rate of 0.26%.For several targets, one predicted region at 0.07% perturbation triggered false positive rates between 93.41% and 99.95%.Source: 3) Validating Boundary Predictions via… · p. 8
  • Some AV products require expanded context around a region, while no effective single-region injections were found for ESET and FProtect.DrWeb reached a 99.94% false positive rate after one region was expanded to 10 times its original context, at a 0.7% perturbation rate.Source: 3) Validating Boundary Predictions via… · p. 8

1) Detection Agreement:

AVHunter achieves higher detection agreement and adversarial resilience than binary-feedback-only baselines. Over time, surrogate agreement declines modestly, and boundary-guided evasion and injection results show moderate degradation across later AV versions.

1) Detection Agreement:

  • 97.43% average detection agreement for AV-DBNet across 11 AVs, exceeding dualFFNN at 94.50% and FFNN-TL at 94.67%.The authors attribute this advantage to explicit byte-level boundary supervision.Source: 1) Detection Agreement · p. 8

2) Detection Resilience Against Adversarial Attacks:

  • 89.86% average detection rate for AVHunter under adversarial modifications, versus 62.51% for FFNN-TL and 59.46% for dualFFNN.AVHunter remains above 80% on all targets; tested modifications include benign-byte injection, control-flow-preserving rewriting, and PE metadata changes.Source: 2) Detection Resilience Against… · p. 9, Figure 6 · p. 9

1) Surrogate Agreement Over Time:

  • 95.94% average agreement after seven months, compared with 97.53% initially and 96.39% after three months across seven AV products.Average degradation is 1.59%; Windows Defender has the largest reported drop, from 97.29% to 95.00%, while ESET declines by 0.21%.Source: 1) Surrogate Agreement Over Time · p. 9

2) Temporal Stability of BR Encryption and BR Injection Results:

  • After seven months, average BR Encryption evasion declines by 7.26%, while average BR Injection false positive rate falls from 91.00% to 83.79%.The largest reported declines are 16.08% for Windows Defender and 14.20% for ClamAV in evasion, and 16.98% for Windows Defender and 18.27% for Dr.Web in injection.Source: 2) Temporal Stability of BR Encryption… · p. 9

E. RQ4: Ablation Analysis

The inference-time ablation tests whether predicted boundary signals contribute useful regional information beyond global raw-byte features. Across 11 AV models, the full model outperforms zero-filled and random-region variants, especially on adversarially modified samples.

E. RQ4: Ablation Analysis:

1) Ablation Design:

  • The three settings use zero-filled regional inputs, random Bernoulli(0.5) masks over valid bytes, or predicted boundary regions alongside global raw-byte features.Random regional inputs select original file bytes and follow the model’s region-count and region-length limits.Source: 1) Ablation Design · pp. 9–10

2) Impact of Boundary Knowledge:

  • 97.43% average agreement and 89.86% adversarial detection: Full Model exceeds No Boundary Signal (89.79% agreement; 55.95% detection) and Random Boundary Signal (91.33%; 58.49%) across all 11 AV models.The metrics are surrogate agreement on unmodified samples and detection rate on adversarially modified samples.Source: Table VI · p. 10, 2) Impact of Boundary Knowledge · p. 10

1) Ablation Design:

  • Agreement on unmodified samples and detection on adversarially modified samples are averaged equally across 11 AV models; the test isolates inference-time boundary use, not boundary-supervision benefits during training.Source: 1) Ablation Design · p. 10

2) Impact of Boundary Knowledge:

  • Predicted boundary signals provide regional information complementary to global raw-byte features, particularly for recognizing malicious evidence in adversarially modified samples.Random regions perform slightly better than zero-filled inputs on average, but neither matches predicted boundary signals, indicating that relevant regional information matters more than merely adding regional bytes.Source: 2) Impact of Boundary Knowledge · p. 10

V. RELATED WORK

Prior black-box surrogates and malware-modification methods generally approximate coarse or implicit detector boundaries rather than localizing decision-critical bytes. AVHunter targets that gap, while its scope, proposed mitigations, and experimental safeguards qualify the paper’s conclusions.

V. RELATED WORK

  • Prior AV surrogates approximate global behavior from coarse binary feedback, while malware-modification methods infer implicit, coarse boundaries rather than explicitly locating decision-critical bytes.AVHunter probes boundary points, constructs AV-specific boundary labels, and jointly learns classification behavior and boundary regions.Source: §V · p. 10

A. Limitation and Future Work

  • AVHunter infers static boundaries and evaluates samples with fewer than 20 boundary points, potentially biasing results toward simpler detection logic.Dynamic execution, sandbox, or cloud analysis may still detect samples; probing averages 19.20 queries per boundary point and 66.53 queries per malware sample.Source: A. Limitation and Future Work · p. 10

B. Mitigations

  • Proposed vendor defenses include pre-scan integrity checks, randomized selection among independent pattern sets, and faster signature updates, with possible accuracy and efficiency tradeoffs.Source: B. Mitigations · p. 11

C. Ethical Consideration

  • Experiments used isolated offline or sandboxed AV engines; the study did not test malware in the wild or publicly distribute surrogate models or boundary annotations in a misuse-enabling form.Source: C. Ethical Consideration · p. 11

VII. CONCLUSION

  • Across 11 real-world AV products, AVHunter localizes byte-level decision-critical regions and outperforms binary-label surrogate baselines in AV agreement and adversarial resilience.Boundary-guided modifications also achieve stronger evasion than generic adversarial baselines.Source: §VII · p. 11

Related papers

Loading 2609.31012v1…