Source-linked AI summary
SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection
Richard Harang, Ethan M. Rudd
TL;DR
SOREL-20M addresses limited comparability and accessibility in malware-detection research by releasing a large benchmark with rich features, metadata, labels, disarmed binaries, baselines, and reproducible code. Its scale supports model comparisons at relevant false-positive rates, while its malware reference set permits evaluation with researcher-developed features.
Problem
Malware-detection models are often evaluated on private or proprietary datasets, while existing public benchmarks have limited size, features, or labels, hindering direct comparison and further exploration.
Method
The paper releases nearly 20 million labeled malware and benign files with pre-extracted features, metadata, behavioral tags, disarmed malware binaries, baseline models, and supporting code.
Results
SOREL-20M provides data volumes intended to establish stable model rankings and assess performance at lower false-positive rates, with baseline FFNN and LightGBM models and behavioral-tag targets.
Takeaways & Limitations
The dataset supports fairer comparisons between malware-detection approaches and evaluation using novel researcher-developed features on a reference set of recent malware.
Takeaways & Limitations
Disarmed binaries are provided for malware only because benign files are excluded over potential intellectual-property concerns.
Abstract
from arXiv · showhide
In this paper we describe the SOREL-20M (Sophos/ReversingLabs-20 Million) dataset: a large-scale dataset consisting of nearly 20 million files with pre-extracted features and metadata, high-quality labels derived from multiple sources, information about vendor detections of the malware samples at the time of collection, and additional ``tags'' related to each malware sample to serve as additional targets. In addition to features and metadata, we also provide approximately 10 million ``disarmed'' malware samples -- samples with both the optional\_headers.subsystem and file\_header.machine flags set to zero -- that may be used for further exploration of features and detection strategies. We also provide Python code to interact with the data and features, as well as baseline neural network and gradient boosted decision tree models and their results, with full training and evaluation code, to serve as a starting point for further experimentation.
1 Introduction: Why Another Dataset?
SOREL-20M addresses the difficulty of comparing malware-detection models by providing a substantially larger, more accessible benchmark with richer data and reproducible baselines.
- Private datasets, commercial data sources, and small evaluation sets make progress in machine-learning malware detection difficult to measure and research harder to access.
- SOREL-20M provides 12,699,013 training, 2,495,822 validation, and 4,195,042 test samples under recommended time splits.The authors report that validation sets of roughly 3–4 million examples can establish stable model rankings and assess low false-positive rates.
- The dataset partially supports feature exploration with 9,919,251 disarmed malware binaries and complete PE metadata, while enabling threshold-level detection-rate comparisons.Disarming sets optional_headers.subsystem and file_header.machine to zero to prevent execution.
- Baseline LightGBM and PyTorch FFNN models support comparisons, while multi-target behavioral tags provide additional prediction targets.The authors note substantial room for improvement, particularly at lower false-positive rates.
- The paper provides dataset statistics, access procedures, baseline models, and associated code for training and evaluation.
2 Dataset description
The dataset combines large-scale malware and benign samples with time-based splits, behavioral tags, metadata, and accessible storage formats for model development and evaluation.
- The release includes disarmed malware binaries, SQLite and LMDB databases, pre-extracted features, PE metadata, and pretrained baseline results.
- Python code interacts with the SQLite and LMDB databases and can train the provided baseline models.
- Samples are keyed by SHA-256, while LMDB entries store serialized and compressed EMBER feature vectors or PE metadata.
- The corpus uses first-seen telemetry to define training, validation, and testing periods from January 2017 through April 2019.The recommended boundaries are November 29, 2018 and January 12, 2019.
- Behavioral-tag corpus statistics are reported for the training set.
3 Baseline Models
The paper establishes reproducible FFNN and LightGBM baselines on EMBER-v2 features and evaluates malware and behavioral-tag outputs with ROC plots.
- Both baseline models use EMBER-v2 features and five random seeds, while LightGBM applies bounded leaf complexity, subsampling, feature subselection, and early stopping.
- The FFNN comprises three Linear–LayerNorm–ELU–Dropout blocks followed by output heads for tags, counts, and malware classification.
- ROC plots cover FFNN malware output, LightGBM malware output, and individual FFNN behavioral-tag outputs.
- The FFNN uses multi-target learning, whereas LightGBM is trained only on malware classification, which may partly explain lower LightGBM performance.
4 GitHub Repository
The GitHub repository supplies reproducibility infrastructure for accessing the dataset, training and evaluating baselines, plotting results, and adapting data loaders to other frameworks.
- The repository contains baseline-training code, database interaction utilities, environment files, and supporting dataset metadata.
- The train.py and evaluate.py scripts train and evaluate models, while plot.py generates ROC-style plots from selected runs.
- dataset.py links SQLite metadata to feature LMDBs, and generators.py wraps dataset instances in PyTorch DataLoaders.
- A utility converts generated feature data into NumPy .npz files suitable for LightGBM training.
5 Conclusion
SOREL-20M combines large-scale malware data, disarmed binaries, extracted features, metadata, labels, tags, baseline models, and reproducibility code. Its scale and recent in-the-wild reference set support fairer model comparisons at relevant false positive rates.
- Nearly 10 million disarmed malware files are included, while benign files are excluded because of potential intellectual property concerns.
- SOREL-20M provides extracted features and metadata for 20 million malicious and benign portable executable files, with labels, behavior-like tags, detection counts, and ten pre-trained baseline models.
- Complete source code required to reproduce the results and explore further developments using the data is provided.
- The dataset is described as the largest released malware benchmark training set and the first to contain a comparatively recent reference set of malware observed in the wild.
- SOREL-20M enables fair comparisons between models using sufficient data for relevant false positive rates and supports evaluation with novel researcher-developed features.