Source-linked AI summary
EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models
Hyrum S. Anderson, Phil Roth
TL;DR
Machine-learning malware detection lacked a large, open benchmark containing both benign and malicious PE files. EMBER addresses this gap with a labeled dataset, feature-extraction code, and a baseline model, and reports that its out-of-the-box LightGBM model outperforms recently published end-to-end deep learning for malware detection.
Problem
Open research has lacked rich benchmark datasets for machine-learning malware detection, including an agreed-upon public set of malicious and benign files.
Method
EMBER constructs a PE malware benchmark with raw features, labels, hashes, chronology, unlabeled training samples, feature-extraction code, and a baseline classifier.
Results
The out-of-the-box LightGBM model trained on EMBER features outperforms recently published end-to-end deep learning for malware detection.
Takeaways & Limitations
EMBER provides a public benchmark and baseline for comparing malware-detection models, including novel architectures and end-to-end deep-learning systems.
Takeaways & Limitations
The dataset is relatively “easy” because benign files had no vendor detections and malicious files had more than 40 vendor reports.
Abstract
from arXiv · showhide
This paper describes EMBER: a labeled benchmark dataset for training machine learning models to statically detect malicious Windows portable executable files. The dataset includes features extracted from 1.1M binary files: 900K training samples (300K malicious, 300K benign, 300K unlabeled) and 200K test samples (100K malicious, 100K benign). To accompany the dataset, we also release open source code for extracting features from additional binaries so that additional sample features can be appended to the dataset. This dataset fills a void in the information security machine learning community: a benign/malicious dataset that is large, open and general enough to cover several interesting use cases. We enumerate several use cases that we considered when structuring the dataset. Additionally, we demonstrate one use case wherein we compare a baseline gradient boosted decision tree model trained using LightGBM with default settings to MalConv, a recently published end-to-end (featureless) deep learning model for malware detection. Results show that even without hyper-parameter optimization, the baseline EMBER model outperforms MalConv. The authors hope that the dataset, code and baseline model provided by EMBER will help invigorate machine learning research for malware detection, in much the same way that benchmark datasets have advanced computer vision research.
1 INTRODUCTION
EMBER addresses the lack of large, open benchmark datasets for machine-learning malware detection by releasing a dataset, feature-extraction code, and a baseline classifier for static PE malware detection.
- Malware detection has lacked the rich, open benchmark datasets available for tasks such as image labeling and speech recognition.
- EMBER releases a dataset extracted from a large corpus of malicious and benign Windows PE files.The release is intended to address dissemination concerns surrounding malicious and benign files.
- The release includes open-source code for computing PE features from additional binaries and a baseline classifier based on a subset of the features.The pre-selected features limit direct feature-set comparisons, while the extraction code supports feature comparison studies.
- EMBER is positioned as a relevant baseline for machine-learning malware detection research and comparison with featureless deep-learning studies.The paper demonstrates such a comparison later in the paper.
2 BACKGROUND
The paper reviews PE structure, static malware detection, prior machine-learning approaches, and existing datasets to motivate a public benchmark for malicious-versus-benign classification.
- PE is the predominant executable format for Microsoft Windows and includes executables, DLLs, and FON font files.
- PE files contain standard headers and sections holding code, data, imports, exports, resources, and loader-related metadata.Headers describe properties such as machine type, file type, section count, entry point, and code or data sizes.
- Static malware detection classifies files as malicious or benign without execution, enabling detection before execution when successful.Dynamic detection instead relies on runtime behavior, including time-dependent system-call sequences.
- Earlier datasets and tools were often small, unpublished, sensitive, family-specific, or potentially biased toward distinguishing Windows from non-Windows files.The reviewed examples include PE-Miner, the Adobe Malware Classifier, and the Microsoft Malware Classification Challenge.
- An agreed-upon public set of malicious and benign files for machine-learning benchmarking had not existed.Malware-sharing services supplied samples, but large-scale access to some sources required paid subscriptions.
3 DATA DESCRIPTION
EMBER organizes PE malware data as extensible, human-readable raw features with labels, hashes, chronology, and parsed or format-agnostic representations that can be vectorized for modeling.
- EMBER was structured to support model comparison, concept-drift analysis, interpretability, feature comparison, and featureless deep-learning comparisons.
- Each JSON-lines record contains a sha256 hash, month-resolution chronology, a label, and eight groups of raw features.Labels are 0 for benign, 1 for malicious, and -1 for unlabeled.
- Raw features are human readable, while supplied code converts them into fixed-size numeric model features and provides a default baseline vectorization.Keeping raw features separate from vectorization supports explainable-machine-learning and feature-importance studies.
- The training set includes unlabeled samples to encourage semi-supervised malware-classification research.
- EMBER links feature records to original binaries through hashes and uses VirusTotal detection thresholds to label benign and malicious files.The authors describe the resulting dataset as relatively “easy.”
- Parsed features cover general file, header, imported-function, exported-function, and section information, while other groups use byte, byte-entropy, and printable-string statistics.Feature hashing converts several symbolic fields and lists into model vectors with specified bin counts.
4 EXPERIMENTS
EMBER provides a feature-vector pipeline and baseline LightGBM detector, then evaluates its test-score distributions and compares performance with MalConv and an older J48 model. The baseline achieves very high ROC performance, while the J48 model performs poorly on the EMBER test set.
- Baseline model: 2351-dimensional raw-feature objects are vectorized, then used to train a LightGBM gradient-boosted decision tree with default parameters.The baseline uses 100 trees and 31 leaves per tree, with fewer than 10K tunable parameters.
- Comparative evaluation: The authors evaluated model-score distributions on the test set and compared the baseline with MalConv trained on the underlying raw binaries.MalConv used the prescribed architecture and training setup except for a reduced batch size of 100 because of GPU memory constraints.
- Baseline performance: The baseline model’s ROC AUC exceeds 0.9991 on the EMBER test set.
- Baseline performance: At less than 0.1% FP rate, a model-score threshold of 0.871 yields a detection rate exceeding 92.99%.
- Baseline performance: At less than 1% FP rate, the baseline exceeds 98.2% detection rate.
- Comparative evaluation: The pre-trained J48 model produces a 53% false positive rate and an 8% false negative rate on the EMBER test set.The paper states that whether this reflects stale training data, dataset bias, or both is outside its scope, but calls J48 an inappropriate baseline.
5 DISCUSSION
EMBER is presented as a large public malware-detection benchmark with released code and a nonoptimized LightGBM model. The authors position it as infrastructure for comparing models and supporting multiple malware machine-learning research use cases.
- Dataset contribution: EMBER is presented as the first large public dataset for machine-learning malware detection that includes benign files.
- Research use cases: The benchmark supports use cases including model-performance comparison, adversarial machine learning, and semi-supervised malware detection.
- Dataset contribution: The dataset is accompanied by source code and a simple nonoptimized LightGBM benchmark model.The dataset and source code are available through the project repository.
- Benchmarking implication: The out-of-the-box LightGBM model outperforms recently published end-to-end deep-learning work for malware detection.The authors suggest that EMBER can benchmark novel architectures, including end-to-end deep-learning models.