PaperlayerView exact revision on arXiv

Source-linked AI summary

EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers

Robert J. Joyce, Gideon Miller, Phil Roth, Richard Zak, Elliott Zaresky-Williams, Hyrum Anderson, Edward Raff, James Holt

arXiv:2506.05074v1cs.CRcs.LG

TL;DR

Researchers have limited access to large, recent, well-labeled malware corpora, while many public benchmarks cover only one platform or classification task. EMBER2024 identifies files first submitted to VirusTotal each day from September 24th, 2023 to December 14th, 2024, and retrieves their VirusTotal analysis results. Despite a ROC AUC score of 0.9969, the authors identified populations of malicious files that can reliably bypass detection.

  • Problem

    Researchers have limited access to large, recent, well-labeled malware corpora, while many public benchmarks cover only one platform or classification task.

    Source: §1 · pp. 1–2

  • Method

    EMBER2024 identifies files first submitted to VirusTotal each day from September 24th, 2023 to December 14th, 2024, and retrieves their VirusTotal analysis results.

    Source: §2 · p. 2

  • Results

    Despite a ROC AUC score of 0.9969, the authors identified populations of malicious files that can reliably bypass detection.

    Source: §4.7 · p. 7

  • Takeaways & Limitations

    The EMBER2024 test and challenge sets support further research on evasive malware, and the authors advocate further study of robust malware classifiers.

    Source: §7 · p. 8

    The LightGBM benchmark classifiers are untuned and intended as baselines; hyperparameter selection, model choice, and training strategy may improve performance.

    Source: §4.7 · p. 7

Abstract

from arXiv · show

A lack of accessible data has historically restricted malware analysis research, and practitioners have relied heavily on datasets provided by industry sources to advance. Existing public datasets are limited by narrow scope - most include files targeting a single platform, have labels supporting just one type of malware classification task, and make no effort to capture the evasive files that make malware detection difficult in practice. We present EMBER2024, a new dataset that enables holistic evaluation of malware classifiers. Created in collaboration with the authors of EMBER2017 and EMBER2018, the EMBER2024 dataset includes hashes, metadata, feature vectors, and labels for more than 3.2 million files from six file formats. Our dataset supports the training and evaluation of machine learning models on seven malware classification tasks, including malware detection, malware family classification, and malware behavior identification. EMBER2024 is the first to include a collection of malicious files that initially went undetected by a set of antivirus products, creating a "challenge" set to assess classifier performance against evasive malware. This work also introduces EMBER feature version 3, with added support for several new feature types. We are releasing the EMBER2024 dataset to promote reproducibility and empower researchers in the pursuit of new malware research topics.

1 Introduction

Malware classifier research is constrained by limited access to recent, well-labeled data and public benchmarks with narrow platform and task coverage. EMBER2024 addresses these gaps with a recent, multi-format dataset supporting multiple tasks and a challenge set.

1 Introduction

  • Public malware benchmarks are often stale or lack benign files, while many support only one platform or classification task.These limitations make it difficult to compare classifier performance across research.Source: §1 · pp. 1–2

1.1 The EMBER2024 Dataset

  • Among notable 1M+ file malware datasets, EMBER2024 is the only multi-platform dataset supporting malware detection, family classification, and behavior identification.It is also the first to include a challenge set and code to replicate dataset construction.Source: Table 1 · p. 2
  • EMBER2024 contains 3,238,315 files collected from September 2023 to December 2024, spanning six formats and seven types of labels and tags.The labels and tags support malware detection, family identification, and other multi-label tasks.Source: §1.1 · p. 2
  • EMBER2024 introduces a challenge set of malicious files initially undetected by antivirus products, for evaluating classifiers on evasive malware.The dataset also provides a feature format for unified representation across formats.Source: §1.1 · p. 2

2 Data Collection and Labeling

EMBER2024 collects and labels files using VirusTotal reports gathered over 64 weeks, with training and test periods separated in time. Its selection, challenge-set, and labeling procedures support malware detection and further analyses of families and behaviors.

2 Data Collection and Labeling

  • Files first submitted on each collection day were queried within 24 hours and again at least 90 days later; recent AV detections were used for labeling.Files suspected to be benign received a rescan at least 30 days after first submission.Source: §2 · p. 2

2.1 File Selection

  • Files were collected across 64 weeks with equal numbers per week; per-file-type selection depended on availability, with ELF the rarest format.Files were bucketed by week, file type, and malicious-benign label; uncertain labels were discarded, and TLSH distance of 30 or less marked near-duplicates for exclusion.Source: §2.1 · pp. 2–3

2.2 Training and Test Sets

  • The training set uses the first 52 collection weeks and contains 2,626,000 files; the final 12 weeks form a 606,000-file test set.Both sets contain files collected at 50,500 per week.Source: §2.2 · p. 3

2.3 Challenge Set

  • The 6,315-file challenge set contains files initially undetected by VirusTotal AV products but later detected by enough products to be labeled malicious.It spans all 64 collection weeks, is separate from training and test sets, and excludes same-week near-duplicates of challenge files from those sets.Source: §2.3 · p. 3

2.4 Labeling Methodology

  • Files undetected by any AV products after a rescan at least 30 days after first submission are labeled benign; files detected by at least five unrelated products are labeled malicious.ClarAVy supplies family labels and tags for behaviors, file properties, exploits, packers, and threat-group attribution when the AV detections provide sufficient information.Source: §2.4 · p. 3

2.5 Family and Tag Demographics

  • 1,356,182 of 1,616,000 malicious training and test files have family labels; the 6,787 families are approximately Zipfian in size.Twelve families exceed 10,000 instances and account for 34.04% of malicious files; 2,709 files belong to 75 test families absent from training.Source: §2.5 · p. 3
  • Tags cover malware behaviors, file properties, packers, exploited vulnerabilities, and threat groups; tags applied to at least 10 files are used for benchmark classifiers.Few files have vulnerability-exploitation or known-threat-group tags.Source: §2.5 · p. 4

3 Dataset Contents

EMBER2024 provides file metadata, feature vectors, labels, and code for dataset use and reproducibility. EMBER feature version 3 expands feature coverage and supports limited extraction from non-PE and unparsable files.

3 Dataset Contents

  • EMBER2024 releases metadata, feature vectors, labels, and tags, but original files require a VirusTotal API key to download.The authors also publish code for dataset construction, feature extraction, and model training.Source: §3 · p. 4

3.1 File Metadata and Feature Vectors

  • Each file’s JSON metadata includes hashes, upload and scan timestamps, antivirus detection ratios, file type, family labels, tags, and raw features.MD5, SHA-1, and SHA-256 identify files; TLSH supports approximate file comparison.Source: §3.1 · p. 4

3.2 EMBER Feature Version 3

  • EMBER feature version 3 updates earlier categories, adds feature types, and supports limited extraction from non-PE and unparsable files.Its 2,568-dimensional vectors can be truncated to 696 dimensions for non-PE files because later entries are zero.Source: §3.2 · pp. 4–5
  • Version 3 adds PE COFF, Optional, Section, and DOS header features, including section counts, memory-region sizes, and section-size ratios.It reduces features allocated to PE section names.Source: §3.2 · p. 4
  • EMBER feature version 3 adds DOS-header, data-directory, Rich-header, Authenticode-signature, and PE-file parse-warning features; the parse-warning category has 88 features tracking errors and warnings during file parsing.General file features include file size, entropy, and the first four bytes.Source: §3.2 · pp. 4–5
  • The limited feature set extractable from non-PE and unparsable files remains effective for classifying Linux, Android, and PDF malware.The extractable categories are general file features, string patterns and statistics, byte histograms, and byte-entropy histograms.Source: §3.2 · p. 5

3.3 Source Code

  • Released code supports dataset replication, feature extraction and vectorization, LightGBM training, and custom splits by set, file format, and label or tag.Researchers can combine the training, test, or challenge sets with selected file formats and label or tag types.Source: §3.3 · p. 5

4 Benchmark Model Results

This section benchmarks 14 LightGBM classifiers across malware detection, family identification, and attribute prediction, finding that strong benchmark scores coexist with difficulty detecting evasive and novel malware.

4 Benchmark Model Results

  • The section releases 14 LightGBM classifiers for malware detection, family identification, and malware attribute prediction.Source: §4 · p. 5

4.1 Evaluating a Malware Detection Classifier

  • 0.9949 ROC AUC and 94.48% true positive rate at 1% false positive rate were achieved by the detection classifier on the EMBER2024 test set.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 Evaluating EMBER2024 Subset Classifiers

  • Subset classifiers accurately detected test-set malware appearing 1–12 weeks after the latest training file; ELF and PDF ROC and PR AUC exceeded 0.99 with reduced features.The APK classifier’s ROC AUC and PR AUC were 0.9868 and 0.9877, respectively.Source: §4.2 · p. 6

4.3 Evaluations Using the Challenge Set

  • Challenge-set results varied sharply: .NET achieved 0.8539 PR AUC, while APK and Win64 reached 0.2374 and 0.4651, respectively.The authors caution that class imbalance makes ROC AUC misleading for this evaluation.Source: §4.3 · p. 6, Table 6 · p. 6

4.4 Detecting Newly Emerging Families

  • 0.8992 PR AUC was achieved on 2,709 malware from 75 unseen families combined with 303,000 benign files, while performance degraded for novel families.The authors contrast this with resistance to multiple weeks of concept drift and encourage further study of novel-family detection.Source: §4.4 · pp. 6–7

4.5 Evaluating Malware Family Classification

  • 67.97% accuracy was achieved in family identification, but weaker macro-averaged metrics suggest more difficulty classifying smaller families.The authors believe insufficient training data for those families is the primary reason.Source: §4.5 · p. 7

4.6 Multi-Label Malware Classification Tasks

  • LightGBM classifiers struggled across all five multi-label tasks, with especially low precision for tasks containing many tags.The authors attribute poor macro-averaged results likely to limited training data for many tags and task difficulty.Source: §4.6 · p. 7

4.7 Discussion of Benchmark Results

  • 0.9969 ROC AUC for EMBER2024 is similar to EMBER2018’s 0.996 and SOREL-20M’s 0.998, yet evasive malware can still bypass detection.The authors identify populations of malicious files that reliably bypass detection despite the benchmark classifier’s score.Source: §4.7 · p. 7

5 EMBER Dataset Retrospective

The retrospective recounts EMBER’s goal of standardizing malware-classifier benchmarking and describes its broad research uptake. It presents EMBER2024 as an update that expands features and file-format support.

  • The original EMBER aimed to provide a standardized benchmark for malware detection research, against a backdrop of proprietary or very small datasets.Source: §5 · p. 7
  • EMBER was cited over 600 times by more than 350 institutions across six continents, and citing work included dataset releases and ML-security innovations.A sample of citing papers had an 82% academic and 18% industry affiliation split.Source: §5 · pp. 7–8
  • Educators reported using EMBER at institutions ranging from high school to graduate school; the authors state they cannot legally provide benign files.Source: §5 · p. 8
  • EMBER2024 updates the feature format to ease feature calculation, add more capable Windows PE features, and broaden support beyond Windows PE files.Source: §5 · p. 8

6 Related Work

The related-work discussion compares large malware datasets and describes differences in their file coverage, labels, and inclusion of benign files.

  • SOREL-20M releases disarmed malicious executables and behavioral tags; MalDICT enables less-common task benchmarks, while VirusShare has a large malware-only corpus.VirusShare had 41,680,896 files available at the time of writing, with AVClass labels for approximately 79.5% of files from April 2019 and earlier.Source: §6 · p. 8

7 Conclusion

EMBER2024 provides a large, multi-format, multi-label malware benchmark and a challenge set for evasive malware, alongside resources for replication and further classifier research.

  • EMBER2024 provides metadata, feature vectors, and labels for more than 3.2 million malicious and benign files across six formats and seven label and tag types.Source: §7 · p. 8
  • The EMBER2024 challenge set is described as the first of its kind and supports new studies on evasive malware.Source: §7 · p. 8
  • The authors advocate further study of robust malware classifiers and provide dataset-building code, EMBER version 3 features, and 14 trained benchmark classifiers.Source: §7 · p. 8
  • The authors hope EMBER2024 will become a valuable research resource and catalyst for investigating malware analysis topics.Source: §7 · p. 8

A EMBER Feature Version 3 Raw Features

Figure 5 presents examples of EMBER feature version 3 raw features.

  • Figure 5 presents examples of EMBER feature version 3 raw features.Source: Figure 5 · p. 10

B Top Families and Tags in EMBER2024

Figure 6 identifies the most common families and tags in EMBER2024.

  • Figure 6 presents the most common families and tags in EMBER2024.Source: Figure 6 · p. 11
Loading 2506.05074v1…