PaperlayerView exact revision on arXiv

Source-linked AI summary

AVScan2Vec: Feature Learning on Antivirus Scan Data for Production-Scale Malware Corpora

Robert J. Joyce, Tirth Patel, Charles Nicholas, Edward Raff

arXiv:2306.06228v1cs.CRcs.LG

TL;DR

Malware analysts need scalable features and similarity search for enormous corpora, but existing approaches face computational and representation limits. AVScan2Vec learns compact vectors from antivirus scan data and, with Dynamic Continuous Indexing, outperforms prior representations across evaluated malware tasks.

  • Problem

    Scalable malware analysis is needed for large repositories, while many feature extraction approaches are computationally infeasible or limited in scope.

    Source: §1 · p. 1, §4.2.4 · p. 13

  • Method

    AVScan2Vec uses self-supervised feature learning to embed antivirus scan data into compact malware vectors, with Dynamic Continuous Indexing for nearest-neighbor lookup.

    Source: §5 · p. 15

  • Results

    AVScan2Vec outperforms prior approaches across evaluated classification, clustering, and nearest-neighbor tasks, including over 78% accuracy when predicting the correct token out of 10 million.

    Source: §5 · p. 15

  • Takeaways & Limitations

    AV scan data contains informative malware features that can support compact representations and large-scale downstream machine-learning tasks.

    Source: §5 · p. 15

    AVScan2Vec is effective for all-malware datasets but less so for mixed malware/benign datasets, and model capacity was limited by hardware constraints.

    Source: §5 · p. 14

Abstract

from arXiv · show

When investigating a malicious file, searching for related files is a common task that malware analysts must perform. Given that production malware corpora may contain over a billion files and consume petabytes of storage, many feature extraction and similarity search approaches are computationally infeasible. Our work explores the potential of antivirus (AV) scan data as a scalable source of features for malware. This is possible because AV scan reports are widely available through services such as VirusTotal and are ~100x smaller than the average malware sample. The information within an AV scan report is abundant with information and can indicate a malicious file's family, behavior, target operating system, and many other characteristics. We introduce AVScan2Vec, a language model trained to comprehend the semantics of AV scan data. AVScan2Vec ingests AV scan data for a malicious file and outputs a meaningful vector representation. AVScan2Vec vectors are ~3 to 85x smaller than popular alternatives in use today, enabling faster vector comparisons and lower memory usage. By incorporating Dynamic Continuous Indexing, we show that nearest-neighbor queries on AVScan2Vec vectors can scale to even the largest malware production datasets. We also demonstrate that AVScan2Vec vectors are superior to other leading malware feature vector representations across nearly all classification, clustering, and nearest-neighbor lookup algorithms that we evaluated.

1 INTRODUCTION

The introduction presents AVScan2Vec as a scalable approach to malware search and machine learning that learns compact vector representations from antivirus scan data. It motivates this approach by examining the latent information in inconsistent and imperfect antivirus labels and outlines its applicability and advantages over existing malware feature representations.

1 INTRODUCTION

  • Production malware corpora can exceed a billion files and require petabytes of storage, while feature extraction must remain robust to obfuscation and scalable.Disassembly and dynamic analysis counter adversarial techniques but are too time-consuming for production-scale datasets; simpler methods may miss higher-level features.Source: §1 · pp. 1–2
  • AVScan2Vec represents an entire AV scan report as one vector, placing malware from the same family or with similar behavior nearby for search, clustering, and classification.It uses self-supervised feature learning rather than manually selected features and can embed any sample detected by one or more AV products.Source: §1 · p. 2
  • AVScan2Vec outperforms prior approaches for malware-family and behavior classification, dominates clustering metrics, and enables 16 millisecond searches over 7 million samples.The vectors can be computed quickly using a single GPU and work with metric acceleration structures.Source: §1 · p. 3

1.1 Related Work

  • Existing malware representations trade off file-format coverage, robustness, scalability, or vector size, motivating AVScan2Vec as a self-supervised alternative based on AV scan data.For example, BWMD uses a 65536-dimensional vector, 85× larger than AVScan2Vec’s, while obfuscation and polymorphism can separate behaviorally identical files.Source: §1.1 · p. 3

1.2 Antivirus Scan Data

  • AV labels are token sequences encoding attributes such as platform, behavior, family, campaign, vulnerability, programming language, and packer.A scan report aggregates labels from multiple AV products because individual products may produce noisy, inconsistent, or ambiguous results.Source: §1.2.1 · p. 3, §1.2.2 · p. 4

1.2.2 Antivirus Scan Reports

  • AV scan reports are difficult to interpret because products use inconsistent naming conventions, differing label granularity, and potentially erroneous information.Each label originates from a product’s signature, heuristic, or other detection technology.Source: §1.2.2 · p. 4
  • A fictitious report illustrates that AV labels can jointly identify a malware family, exploit, vulnerability, ransomware behavior, target operating system, and an incorrect family classification.The example includes WannaCry, EternalBlue, CVE-2017-0147, ransomware targeting 32-bit Windows, and an incorrect Rbot label.Source: Figure 1 · p. 4
  • Similar AV scan reports indicate that the corresponding malware samples likely share common latent attributes.This relationship forms the foundation of the work.Source: §1.2.2 · p. 4

2 AVSCAN2VEC MODEL ARCHITECTURE

This section describes the AVScan2Vec architecture, covering preprocessing, pre-training, and fine-tuning to transform heterogeneous antivirus labels into report-level representations. It also details Siamese fine-tuning on malware-family scan-report pairs using Multiple Negatives Ranking loss.

2 AVSCAN2VEC MODEL ARCHITECTURE

  • AVScan2Vec processes heterogeneous antivirus labels through preprocessing, pre-training for token semantics, and fine-tuning that embeds related malware reports nearby.The model converts scan-report contents into token embeddings, learns semantic meaning, and produces report-level vectors.Source: §2 · p. 4

2.1 Data Preprocessing

  • The preprocessing design supports several-hundred-token reports and over 40 million distinct tokens, while representing absent and benign detections explicitly.The pre-training dataset vocabulary exceeds BERT’s by over 1000×.Source: §2.1 · p. 4

2.1.1 Antivirus Label Preprocessing.

  • Labels are lowercased, split on non-alphanumeric characters, truncated to five tokens, and wrapped with product-specific start, end, padding, absence, and benign markers.The five-token limit covers 95.17% of labels, while behavior, architecture, and family indicators usually appear near the beginning.Source: §2.1.1 · pp. 4–5

2.1.2 Token Embedding.

  • AVScan2Vec employs an embedding method to address traditional embeddings’ infeasibility caused by the task’s extreme vocabulary size and tokens outside a pre-established vocabulary.Source: §2.1.2 · p. 5

2.2 Pre-training Tasks

  • Pre-training encodes each scan report with a <CLS> token and fixed-position product-label embeddings, then applies a Transformer for masked token and label prediction.The default Transformer uses four blocks, and the <CLS> state supports label-level prediction while token states support token-level prediction.Source: §2.2 · pp. 5–6

2.2.1 Masked Token Prediction.

  • Masked Token Prediction randomly selects and holds out approximately 5% of tokens in each sequence, requiring AVScan2Vec to predict them from surrounding AV scan report context.Source: §2.2.1 · p. 6

2.2.2 Masked Label Prediction.

  • Masked Label Prediction replaces one product’s label and trains an LSTM decoder conditioned on the Transformer representation to autoregressively reconstruct it.Adaptive softmax handles the 10 million most common tokens, and the model performs best by selecting the highest-likelihood token without beam search.Source: §2.2.2 · pp. 6–7

2.3 Fine-tuning Tasks

  • After pre-training learns general token-label semantics, AVScan2Vec can be fine-tuned for high performance on other tasks.Fine-tuning follows the pre-training stage and adapts the model beyond its semantic pre-training objective.Source: §2.3 · p. 7
  • AVScan2Vec fine-tuning uses a Siamese network that encodes complete scan reports and receives batches of anchor-positive pairs.The anchor is sampled from the fine-tuning dataset, while the positive is another scan report from the same malware family.Source: Figure 4 · p. 7, §2.3 · p. 7
  • Multiple Negatives Ranking loss reduces distance between each anchor-positive pair while treating every nonmatching positive in the batch as a negative candidate.Representations are obtained from the <CLS> token, and candidate scores are computed using dot products.Source: §2.3 · p. 7
  • MNR loss suits malware tasks with scarce labels because identifying related samples is easier than identifying unrelated samples, which requires family labels.Related pairs can be identified with robust file-similarity metrics, whereas unrelated pairs require explicit family information.Source: §2.3 · p. 7
  • MNR avoids explicit negative candidates by exploiting batch diversity, making same-family matches among nonmatching batch examples unlikely across the many malware families.Thus, unrelated candidates are supplied implicitly by the batch rather than through additional family annotations.Source: §2.3 · pp. 7–8

3 PRE-TRAINING AVSCAN2VEC

AVScan2Vec was pre-trained on a large VirusTotal corpus to learn AV-scan token semantics and label formats, achieving high pre-training accuracy. Validation indicates robust performance across AV products, sparse reports, and temporal shifts, with degradation where training data is limited.

3.1 Pre-Training Implementation

  • AVScan2Vec was pre-trained on more than 33.4 million AV scan reports collected from May 2006 to April 2023, supporting token semantics, AV-specific label formats, and scan-content inferences across 89 distinct AVs.The 89 AVs were present in at least 1% of queried scans.Source: §3 · p. 8, §3.1.2 · p. 8, §3.1.1 · p. 8

3.1.2 Technical Details.

  • AVScan2Vec achieved 91.02% Masked Token Prediction accuracy and 83.97% Masked Label Prediction accuracy after pre-training on more than 33.4 million AV scan reports.The resulting pre-training dataset contained 37,132,493 total scan reports.Source: §3.1.2 · p. 8, §3 · p. 8, §3.1.1 · p. 8

3.2.1 Accuracy Per Antivirus Product.

  • Across 89 AV products, Masked Token Prediction ranged from 79.64% to 100.00%, while Masked Label Prediction ranged from 66.91% to 99.35%.Accuracy was highest for products with simple label formats or correlated labeling decisions, but these metrics do not measure an AV product’s malware-detection performance.Source: §3.2.1 · p. 9

3.2.2 Accuracy Per Number of Positive Detections in Scan Report.

  • Above 78% token-prediction and above 77.5% label-prediction accuracy remained achievable with only two positive detections in a report.The model therefore performed both pre-training tasks under the restrictive condition of observing a single AV label while another label was masked.Source: §3.2.2 · p. 9, Figure 5 · p. 9

3.2.3 Accuracy Per Month of Scan Report.

  • Monthly accuracy generally stayed high but declined during periods with limited training data, especially before 2009 when only about 5,000–10,000 reports were available monthly.The model still performed both tasks on temporally shifted data nearly two years into the future, although accuracy was lower in data-sparse periods.Source: §3.2.3 · pp. 9–10

4 FINE-TUNING AVSCAN2VEC

The section presents validation views of AVScan2Vec accuracy across report months and temporal generalization. Accuracy generally remains high, including for data nearly two years beyond the temporal training period.

4 FINE-TUNING AVSCAN2VEC

  • Monthly accuracy generally stays high, although it degrades in months with limited training data.The figure organizes AVScan2Vec accuracy by the month in which each scan report was created.Source: Figure 6 · p. 10
  • Temporal-split training remains successful on both pre-training tasks for data nearly two years into the future.This evaluates performance on scan reports created after the temporal training period.Source: Figure 7 · p. 10
  • Fine-tuning used 10 million anchor-positive AV-scan-report pairs and compared the resulting vector representation with other malware vector representations.The section describes technical fine-tuning details followed by comparative evaluation.Source: §4 · p. 10

4.1 Fine-Tuning Implementation

Fine-tuning seeks to place related malware samples near one another while remaining resilient to changes in AV products and versions. It uses TLSH-derived pairs and a large anchor-positive training set.

4.1.1 Anchor-Positive Selection.

  • TLSH identified related malware files using a distance below 30, producing 26,274,859 candidate pairs before downsampling.The threshold was associated with a reported false-positive rate of 0.00181%, while large edits, packing, polymorphism, or obfuscation may hide related files.Source: §4.1.1 · p. 11
  • 10,010,000 anchor-positive pairs were selected for fine-tuning after duplicate removal and random downsampling.The pairing strategy was intended to provide high-fidelity related files despite possible AV-label changes caused by small malware alterations.Source: §4.1.1 · pp. 10–11

4.1.2 Technical Details.

  • Fine-tuning used 10,000,000 training pairs plus 10,000 validation pairs, and training stopped after one epoch when overfitting began.The model was trained with AdamW and a learning-rate reduction when validation loss plateaued.Source: §4.1.2 · p. 11

4.2 Fine-Tuning Evaluation

This section evaluates AVScan2Vec against alternative malware representations across classification, clustering, and nearest-neighbor tasks on MOTIF and SOREL benchmarks. It also examines Dynamic Continuous Indexing for scalable nearest-neighbor search over large AVScan2Vec vector collections.

4.2 Fine-Tuning Evaluation.

  • The study compared AVScan2Vec with BWMD and EMBER, including normalized EMBER-N, across nearest-neighbor lookup, classification, and clustering using public benchmarks when possible.Dynamic analysis and disassembly methods were excluded because they could not scale to production-size malware corpora.Source: §4.2 · p. 11

4.2.1 Evaluation Datasets.

  • MOTIF samples were labeled manually and filtering removed invalid VirusTotal reports and singleton families, leaving 2,962 samples; SOREL contains 9,919,065 malicious PE files labeled with 11 behavior tags.Source: §4.2.1 · p. 11

4.2.2 Malware Representation Sizes.

  • AVScan2Vec vectors were 3× smaller than EMBER vectors and 85× smaller than BWMD vectors, facilitating faster comparisons and lower memory usage.For 7,292,023 SOREL files, AVScan2Vec vectorization took six hours and 16 minutes on one GPU, compared with estimated 153 and 161 hours for BWMD and EMBER using 72 CPU cores.Source: Table 2 · p. 12, §4.2.2 · p. 12

4.2.3 Behavioral Tag Classification Results.

  • AVScan2Vec achieved the highest ROC-AUC for all 11 behavioral tags with FFNN and for nine tags with LightGBM OvR.The evaluation used five-fold cross validation on SOREL-110000, with FFNN and LightGBM benchmark architectures or hyperparameters.Source: §4.2.3 · pp. 12–13, Table 3 · p. 12, Table 4 · p. 12

4.2.4 Family Classification Results. Next, we tested each

  • AVScan2Vec had the highest overall family-classification accuracy by over six percentage points and performed best or second-best with each classifier.FFNN, LightGBM, and 1-NN were evaluated on MOTIF; EMBER performed better with LightGBM, while AVScan2Vec remained second-highest there.Source: Table 5 · p. 13, §4.2.4 · p. 13

4.2.5 Clustering Results.

  • AVScan2Vec scored highest in homogeneity, completeness, and V-measure for both K-Means and hierarchical agglomerative clustering.The algorithms used Euclidean distance, complete linkage for HAC, and 323 clusters corresponding to MOTIF’s malware families.Source: §4.2.5 · p. 13, Table 6 · p. 13, Table 7 · p. 13

4.2.6 Nearest-Neighbor Lookup. Our final experiments

  • AVScan2Vec vectors for the 10-nearest lookup produced the highest F1 score for nine of 11 behavioral tags, significantly improving over BWMD and marginally over EMBER and EMBER-N.Source: §4.2.6 · p. 14

4.2.7 Dynamic Continuous Indexing.

  • Dynamic Continuous Indexing was employed to evaluate nearest-neighbor lookup over massive datasets, with continuous indices constructed for the experiment.DCI is described as a highly efficient nearest-neighbor lookup algorithm [17].Source: §4.2.7 · p. 14
  • 75 seconds was sufficient to construct continuous indices for the same dataset with 64 CPU cores, reducing average lookup time to 0.029 seconds per vector.Using a single Intel Xeon Gold 6240R CPU instead yielded 0.501 seconds per vector for 100,000 queries.Source: §4.2.7 · p. 14
  • 16.41 ms average query time was achieved for 10-nearest-neighbor lookup over 7,192,023 AVScan2Vec vectors using DCI.The queries returned relevant malware, with precision, recall, and F1 reported per SOREL behavioral tag.Source: Table 9 · p. 14, §4.2.7 · p. 14
  • Repeating the prior experiment with 64 CPU cores over 1 billion AVScan2Vec vectors would still allow an average 10-nearest-neighbor lookup time of ≈4 seconds per vector.Source: §4.2.7 · p. 14

5 CONCLUSION

The conclusion presents AVScan2Vec as a compact representation learned from AV scan data, with broad utility for malware analysis and machine-learning tasks. It also identifies dataset composition, hardware, and concept drift as practical boundaries on performance.

5 CONCLUSION

  • AVScan2Vec is effective on all-malware datasets but less effective on mixed malware/benign datasets.Source: §5 · p. 14
  • Performance is constrained by hardware-limited model size and concept drift, making monthly retraining likely necessary for optimal performance.The authors state that adding Transformer encoder layers would almost certainly be beneficial.Source: §5 · pp. 14–15
  • AVScan2Vec learns informative malware features from AV scan data and embeds samples into compact vectors for downstream machine-learning tasks.The vectors are intended for related-malware queries over enormous corpora and various ML tasks.Source: §5 · p. 15
  • AVScan2Vec predicted the correct token out of 10 million with over 78% accuracy for reports with just a single positive detection.Source: §5 · p. 15
  • Family-label tasks show significant improvement, whereas behavioral-tag tasks show only marginal accuracy improvement.Source: §5 · p. 15
  • Combining AVScan2Vec with Dynamic Continuous Indexing enables nearest-neighbor lookup over massive malware datasets through sub-linear query runtime complexity.The combination is presented as promising for threat hunting.Source: §5 · p. 15

A APPENDIX

The appendix lists the AV products included in AVScan2Vec’s pre-training dataset and reports per-product accuracies for its two pre-training tasks.

A APPENDIX

  • Table 10 lists the included AV products and their per-product accuracy for Masked Token Prediction and Masked Label Prediction.Source: Table 10 · p. 17

Related papers

Loading 2306.06228v1…