Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features

Joshua Saxe; Konstantin Berlin

Source-linked AI summary

Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features

Joshua Saxe, Konstantin Berlin

arXiv:1508.03096v2Aug 13, 2015cs.CR

TL;DR

Malware detection needs high detection rates with very low false positives, while machine-learning approaches have lagged behind signature-based methods on that requirement. The paper introduces a deep neural network using static binary features and reports 95% detection at a 0.1% false positive rate over more than 400,000 binaries.

Problem
Machine-learning malware detectors have struggled to achieve the very low false positive rates needed for deployment, despite avoiding manual signature generation.
Method
The system combines static binary features, a deep neural network classifier, and score calibration, including two-dimensional byte-entropy histograms.
Results
95% detection at a 0.1% false positive rate was achieved over an experimental dataset of more than 400,000 software binaries.
Takeaways & Limitations
The detector was deployed in a cloud security analytics platform processing files streamed from thousands of customer endpoints.
Takeaways & Limitations
Standard cross-validation does not distinguish detection of slightly modified malware from detection of new malware toolkits or toolkit versions.

Abstract

from arXiv · show

Malware remains a serious problem for corporations, government agencies, and individuals, as attackers continue to use it as a tool to effect frequent and costly network intrusions. Machine learning holds the promise of automating the work required to detect newly discovered malware families, and could potentially learn generalizations about malware and benign software that support the detection of entirely new, unknown malware families. Unfortunately, few proposed machine learning based malware detection methods have achieved the low false positive rates required to deliver deployable detectors. In this paper we a deep neural network malware classifier that achieves a usable detection rate at an extremely low false positive rate and scales to real world training example volumes on commodity hardware. Specifically, we show that our system achieves a 95% detection rate at 0.1% false positive rate (FPR), based on more than 400,000 software binaries sourced directly from our customers and internal malware databases. We achieve these results by directly learning on all binaries, without any filtering, unpacking, or manually separating binary files into categories. Further, we confirm our false positive rates directly on a live stream of files coming in from Invincea's deployed endpoint solution, provide an estimate of how many new binary files we expected to see a day on an enterprise network, and describe how that relates to the false positive rate and translates into an intuitive threat score. Our results demonstrate that it is now feasible to quickly train and deploy a low resource, highly accurate machine learning classification model, with false positive rates that approach traditional labor intensive signature based methods, while also detecting previously unseen malware.

1. Introduction

Malware enables crime, espionage, and other unwanted network activity, motivating detection methods that can stop or remove it. The paper introduces a deployable deep neural network detector using static features, enabled by advances in data availability, computing power, and machine learning.

Problem: Malware facilitates crime, espionage, and other unwanted network activities, creating a need to detect it before or after it achieves its objectives.Detection can prevent malware from achieving its objectives or enable its removal after discovery.
Existing Approaches: Detection approaches include hand-crafted rule or signature methods, machine learning models, and automatic signature generation as an intermediate approach.Rule-based methods require analysts to craft rules, whereas machine learning automatically fits model parameters using malicious and benign data.
Motivation: Recent developments increased the prospect that machine-learning detectors could achieve high detection rates at low false positive rates without manual signature generation.The paper attributes this opportunity to advances in data, computing, and machine learning.
Motivation: Threat intelligence feeds provide timely labeled malware data, cheaper computing enables larger-model iteration, and machine learning has advanced as a discipline.Together, these three trends support more capable malware-detection research.
Contribution: The paper introduces a deployable deep neural network malware detector using static features and presents it as having the best reported accuracy among previously published detection engines.The approach is explicitly designed to take advantage of all three enabling trends.

2. Method

The classification framework has three components: complementary static binary feature extraction, a deep neural network classifier, and score calibration for realistic score interpretation.

Framework components: The framework extracts four complementary feature types from static benign and malicious binaries, classifies them with a network containing two hidden layers, and calibrates outputs into interpretable scores.The classifier includes input, hidden, and output layers.

2.1. Feature Engineering

The section describes a file-format-agnostic feature pipeline that converts binaries into a compact 1024-dimensional vector using byte entropy, imports, and PE metadata, with labels assigned through VirusTotal voting.

Byte entropy features: The byte-entropy feature uses a 1024-byte window, 256-byte step, and 16×16 histogram flattened into a 256-value vector.Each window’s base-2 entropy is paired with its 1024 byte occurrences; entropy spans [0,8] and byte values span [0,255].
Byte entropy features: Entropy-context representation separates byte values associated with x86 instruction data from those occurring in compressed data without relying on file format.The representation models the contents of input files in a file-format-agnostic way.
Import features: Import features hash each DLL-name/function tuple into a 256-element counter array, preserving fixed feature size for scalability.The hashed imports capture external-call semantics and combinations associated with suspicious files or known malware families.
PE metadata features: PE metadata features extract numerical fields with the pefile Python library and aggregate them into a 256-length array.These features expose suspicious packaging characteristics and potential malware-family signatures.
Feature construction: A 1024-dimensional feature vector concatenates four fixed-size 256-dimensional vectors, reducing memory and CPU requirements with only minor accuracy degradation.The four vectors represent the feature sets described in the section.
Labeling: Files are labeled malware when at least 30% of approximately 55 VirusTotal engines alarm, benignware when none alarm, and uncertain cases are discarded.The labeling procedure supports low-false-positive training and evaluation without filtering binaries by content.

2.2. Neural Network

The classifier is a four-layer deep feedforward neural network with dropout, PReLU activations in its first two layers, and a sigmoid output. Its design uses dropout and normalized Gaussian initialization to improve robustness and convergence, and trains with backpropagation and Adam.

Architecture: The model has four layers: three 1024-node layers with dropout, followed by a dense prediction layer.The first two hidden layers use PReLU activations, while the last hidden layer uses sigmoid activation.
Activation functions: PReLU activations dynamically adjust their behavior for negative inputs to improve convergence relative to standard ReLU.ReLU can perform poorly for inputs below 0, while sigmoidal activations can slow training through saturation.
Initialization and optimization: Weights are initialized from a Gaussian distribution normalized by each layer’s input and output sizes before training.The feature values are transformed by applying the base-10 logarithm before initialization.
Regularization: Dropout randomly retains a fraction h of nodes per batch update, with independent Bernoulli samples resampled for each update.The approach seeks complementary weights across nodes and is intended to reduce overfitting and improve resilience to imperfect data.
Initialization and optimization: The network is trained with backpropagation and Adam, which the authors observed converged significantly faster than standard stochastic gradient descent.The loss is defined using cross-entropy between model predictions and true labels, where 0 denotes benignware and 1 denotes malware.

2.3. Bayesian Calibration

The system calibrates classifier scores into malware probabilities by combining network malware base rates with the neural network’s empirical error profile. It computes these threat scores using class-conditional score densities estimated non-parametrically with KDE.

Bayesian calibration: The Bayesian calibration combines the customer network’s malware-to-benignware base rate with the neural network’s empirical error profile on test data.This extends the system beyond binary malware detection to provide calibrated probabilities.
Threat-score definition: Threat score is defined as P(C = m|S = s), the probability that a file is actually malware given classifier score s.Scores range from 0, completely benign, to 1, certainly malware.
Bayesian calibration: The threat score is derived from the benign and malware score densities, p(S = s|C = m) and p(S = s|C = b), together with the malware base rate.Bayes’ rule and the constraint that probabilities sum to 1 yield the final expression.
Density estimation: Because classifier outputs need not follow a standard distribution, the system estimates class-conditional densities non-parametrically using kernel density estimation.The paper contrasts this with parametric distribution fitting.
Density estimation: The implementation uses an Epanechnikov kernel, mirrors samples outside [0,1], and sets the window size empirically to 0.01.The Epanechnikov kernel had a better validation score than the standard Gaussian kernel, while mirroring supports density estimation near distribution tails.

3. Evaluation

Evaluation used cross-validation on a 431,926-binary dataset and live enterprise feeds to assess feature performance and deployment-relevant false positives. Time-split testing showed substantially lower detection for genuinely novel malware than standard validation.

Evaluation datasets: The final dataset contained 431,926 binaries: 81,910 benignware and 350,016 malware, sourced from Invincea systems, customer networks, Jotti, and Invincea’s private database.The dataset was filtered using VirusTotal.
Cross-validation: Five separate 4-fold cross-validation experiments trained on three equally sized partitions and tested on the fourth.Experiments assessed individual feature types and their combination.
Cross-validation: 95.2% of malicious binaries were detected on average using all feature types together, outperforming individual feature sets.PE metadata detected close to 87% of unseen malware and string features detected 69%, both at a 0.1% false positive rate; byte-entropy and import features further boosted combined accuracy.
Enterprise deployment: 0.1% FPR produced about five false positives per day, based on an average of around 5 previously unseen executed binaries per endpoint per day.The estimate was derived from observations across the entire customer set over a span of a few days.
Time-split evaluation: 67.7% of malware was detected at a 0.1% FPR in the time-split experiment, while detection approached 100% at a 1% FPR.The experiment was designed to better estimate detection of genuinely new toolkits and new versions, which standard cross-validation does not isolate from modified known malware.

4. Related Work

Related work distinguishes static from dynamic malware analysis and frames static analysis as especially suitable for machine learning because it supports inexpensive large-scale datasets with accurate labels. Prior research spans many binary features and learning methods, while scalability and inconsistent evaluation datasets remain challenges.

Analysis paradigms: Static analysis supports inexpensive aggregation of very large binary datasets and accurate labels from antivirus aggregator sites, despite vulnerability to obfuscation.Dynamic detection can directly observe malware action but faces sandbox evasion and an ongoing arms race with malware.
Machine learning foundations: Machine learning is framed as a two-part process: feature engineering transforms binaries into features, and a learning algorithm builds the classifier.Its suitability for static analysis follows from the availability of large datasets and the tendency of machine learning to improve as data size increases.
Feature representations: Proposed static binary features include printable strings, import tables, byte n-grams, opcodes, informational entropy, opcode images, decompiled assembly features, and aggregate statistics.The authors state that they are unaware of published methods breaking files into subsamples using sliding windows.
Scalability: High-dimensional feature spaces create scalability problems, including O(N^2) kernel multiplication for nonlinear SVMs and substantial computation and storage for k-NN prediction.Hashing and random projections have been used when feature spaces become large.
Learning algorithms: Neural networks have emerged as scalable alternatives, but comparing malware-detection results is difficult because datasets and pre-filtering procedures differ.Earlier malware-detection work also used neural networks and other machine-learning approaches.

5. Conclusion

The approach achieved 95% detection at 0.1% false positive rate over more than 400,000 binaries with modest computation and single-GPU training. Its layered features support training on all binaries, while neural-network properties enable continual updating and deployment at scale.

Results: 95% detection rate and 0.1% false positive rate were achieved over an experimental dataset of more than 400,000 software binaries.Feature extraction required modest computation, and the corpus could be trained on a single GPU within modest timeframes.
Methodological implications: Layered deep neural networks and two-dimensional histogram features provide implicit categorization, enabling direct training on all binaries without separating packer types or other internal categories.The authors attribute this capability to the combination of the layered architecture and two-dimensional histogram features.
Operational properties: Neural networks support incremental learning through efficient batch retraining as new training data is collected.The passage states that retraining can occur even on an hourly or daily basis.
Operational properties: Neural networks can combine labeled and unlabeled data through pretraining of individual layers, and their compact classifiers enable very fast prediction.These properties are identified as advantages for malware detection.
Deployment: The system became a crucial part of the company’s malware detection and prevention product and was deployed on a cloud security analytics platform serving files from thousands of customer endpoints.The platform performs detection on files streaming from deployed endpoints.