DeepSign: Deep Learning for Automatic Malware Signature Generation and Classification

Eli David; Nathan S. Netanyahu

Source-linked AI summary

DeepSign: Deep Learning for Automatic Malware Signature Generation and Classification

Eli David, Nathan S. Netanyahu

arXiv:1711.08336v2Nov 21, 2017cs.CRcs.LGcs.NEstat.ML

TL;DR

Conventional malware signatures often fail to recognize new variants, motivating a method that represents malware behavior more invariantly. DeepSign uses a deep belief network built from denoising autoencoders to generate compact signatures from sandbox logs, achieving 98.6% classification accuracy on test data. The signatures also cluster variants from the same malware family in signature space.

Problem
Conventional signature and token-based methods do not detect a majority of new variants for existing malware, while manual signature analysis is slow.
Method
DeepSign converts sandbox logs into binary strings and trains a deep belief network implemented with denoising autoencoders to generate compact behavioral signatures.
Results
98.6% classification accuracy was obtained on 600 test samples using the supervised network initialized with DBN weights.
Takeaways & Limitations
DBN-generated signatures support accurate classification of unseen malware variants and cluster variants from the same family in signature space.

Abstract

from arXiv · show

This paper presents a novel deep learning based method for automatic malware signature generation and classification. The method uses a deep belief network (DBN), implemented with a deep stack of denoising autoencoders, generating an invariant compact representation of the malware behavior. While conventional signature and token based methods for malware detection do not detect a majority of new variants for existing malware, the results presented in this paper show that signatures generated by the DBN allow for an accurate classification of new malware variants. Using a dataset containing hundreds of variants for several major malware families, our method achieves 98.6% classification accuracy using the signatures generated by the DBN. The presented method is completely agnostic to the type of malware behavior that is logged (e.g., API calls and their parameters, registry entries, websites and ports accessed, etc.), and can use any raw input from a sandbox to successfully train the deep neural network which is used to generate malware signatures.

I. INTRODUCTION

Existing malware defenses rely on slow, handcrafted signatures that variants can evade through small code changes. DeepSign proposes DBN-generated behavioral signatures and reports accurate classification on unseen malware data.

Motivation: 160,000 new malware programs appeared daily on average in 2013, while conventional defenses still depended on manually generated signature updates.Manual analysis delays detection, leaving malware able to infect additional computers before signatures are released.
Motivation: Small code modifications let malware variants evade conventional anti-virus signatures, sustaining a long-running cycle between malware developers and defenders.The paper describes this as a recurring “cat and mouse” game affecting major malware programs.
Contribution: DeepSign trains a DBN to create an invariant compact representation of general malware behavior rather than relying on any specific malware aspect.The approach is intended to remain invariant to many code modifications and detect new variants.
Contribution: The method runs malware in a sandbox, converts behavioral logs to binary strings, and trains a deep belief network implemented with denoising autoencoders.Training is described as unsupervised and proceeds layer by layer.
Results: 98.6% classification accuracy was achieved on unseen data using DBN-generated signatures from several hundred variants per malware category.The result is reported as evidence of the representation power of the generated signatures.

II. RELATED WORK

Prior automatic signature methods focus on specific vulnerabilities, traffic patterns, executable substrings, or statistical code properties. DeepSign instead targets broader program behavior to reduce sensitivity to small variant modifications.

Traffic-based signatures: Traffic-based methods assume malware traffic patterns remain substantially unchanged across variants, making altered traffic patterns a potential evasion route.The related-work discussion includes Autograph, Honeycomb, and PAYL as traffic-oriented approaches.
Traffic-based signatures: Autograph uses frequent byte sequences from scanner-generated network traffic as signatures, while Honeycomb uses largest common substrings in honeypot packet payloads.Both approaches derive signatures from network observations rather than complete program behavior.
Executable-based signatures: Executable-based approaches analyze code substrings, code distributions, or equal-sized executable segments to generate signatures.The cited methods include statistical analysis across variants and ranked segment-level signatures.
Limitations of prior methods: These approaches can be limited when malware changes the targeted behavior or encrypts its executable, and statistical analysis is less accurate for larger malware.The paper notes that many real-world malware samples contain multiple modules and sub-modules.
DeepSign’s positioning: DeepSign addresses this gap by generating signatures from program behavior without focusing on a specific executable feature or network-traffic pattern.The stated goal is to make signatures more invariant to small-scale changes.

III. PROPOSED SIGNATURE GENERATION METHOD

The proposed method applies deep-learning representation principles to malware behavior, aiming to produce fixed-size signatures that remain invariant under small changes.

Research question: The central question is whether a program signature can represent behavior while remaining invariant to small-scale changes.The paper frames this as the main question of the proposed method.
Representation learning: DeepSign models malware behavior with a deep neural network trained to produce invariant representations.The approach adapts representation-learning principles previously used for difficult recognition tasks.
Input representation: The method first converts sandbox-observed behavior into a fixed-sized vector suitable as neural-network input.This vectorization step precedes training the network to produce an invariant representation.
Objective: The resulting representation is intended to support detection of malware variants that differ through small-scale changes.The paper states this as the goal of applying invariant representation learning to program behavior.

A. Program Behavior as Binary Vector

DeepSign represents sandbox-recorded program behavior as a fixed-size binary vector derived from text logs. To remain domain agnostic, it extracts unigrams without preprocessing the log’s markup or tags.

Sandbox behavior logs: A sandbox records behavior such as API calls, parameters, filesystem changes, websites, and accessed ports in a text file.These records are normally examined to learn information for malware signature creation.
Binary vector construction: Unigram extraction represents each text sample by indicating which words from a fixed dictionary are present.The paper illustrates this with a 5,000-word dictionary producing a 5,000-bit string.
Representation scope: The resulting binary representation captures high-level program behavior rather than low-level executable features such as strings.The paper explicitly contrasts its content representation with low-level image or executable features.
Domain-agnostic parsing: The method treats the sandbox file as simple text and extracts unigrams without preprocessing its markup or tagged components.For example, both a tag and its associated API name are extracted as terms while their meaning is ignored.

B. Training a Deep Belief Network

The DBN is built by stacking denoising autoencoders to learn malware representations that are resilient to small behavioral changes. Layer-wise training freezes earlier encoder weights while progressively adding higher-level layers, producing a 30-value signature.

B. Training a Deep Belief Network: The method trains a deep belief network by stacking denoising autoencoders to obtain an invariant malware representation.The representation is intended to remain resilient to small changes in malware behavior.
B. Training a Deep Belief Network: Denoising autoencoders corrupt inputs but optimize reconstruction of the original uncorrupted input.The paper describes a typical corruption ratio of about 0.1 to 0.2 and links denoising to reduced overfitting and better generalization.
B. Training a Deep Belief Network: Layer-wise training discards each decoder, fixes the preceding encoder, and trains a new autoencoder on the resulting hidden representation.Repeating this process assembles a deep network with multiple hidden layers.
B. Training a Deep Belief Network: The eight-layer network maps a 20,000-dimensional input vector to 30 floating-point values treated as the program signature.The layer sizes are 20,000–5,000–2,500–1,000–500–250–100–30.

IV. IMPLEMENTATION AND EXPERIMENTAL RESULTS

This section introduces the dataset and sandbox, then describes the trained neural network and presents the experimental results.

IV. IMPLEMENTATION AND EXPERIMENTAL RESULTS: The section covers the malware dataset and sandbox, the trained neural network, and the experimental results.

A. Malware Dataset and Sandbox

The study uses six malware categories with 300 variants each, runs them in Cuckoo sandbox, and adopts Kaspersky classifications as ground truth. The dataset is split into separate training and testing sets.

A. Malware Dataset and Sandbox: The dataset contains 1,800 samples across six malware categories, with 300 variants in each category.The categories are Zeus, Carberp, SpyEye, Cidox, Andromeda, and DarkComet.
A. Malware Dataset and Sandbox: The malware families include widely used threats whose variants modify parts of the malware to evade antivirus detection.The paper describes these variants as remaining undetected until manual detection, analysis, and signature generation.
A. Malware Dataset and Sandbox: Kaspersky anti-virus categories serve as the ground truth, although the six classes are not completely separated.The stated labeling convention makes the prediction task difficult because malware families can overlap.
A. Malware Dataset and Sandbox: Each sample is run in Cuckoo sandbox, which records API traces, filesystem activity, network access, and registry changes.The resulting behavior is saved in a JSON-format text file.
A. Malware Dataset and Sandbox: The dataset is randomly split into 1,200 training samples and 600 testing samples.Each category contributes 200 training samples and 100 testing samples.

B. Training the DBN

The implementation uses an eight-layer denoising autoencoder network with dropout and ReLU activations. Sandbox behavior is converted into a bit-string, processed by the network, and returned as a 30-value signature.

B. Training the DBN: The network uses eight layers sized 20,000–5,000–2,500–1,000–500–250–100–30 and trains one layer at a time.Dropout omits each hidden unit with probability 0.5 to regularize the network.
B. Training the DBN: ReLU activations replace standard logistic or tanh nonlinearities in the network.The paper associates ReLU with faster convergence and reduced gradient vanishing in deep networks.
B. Training the DBN: Training uses a denoising noise ratio of 0.2, 1,000 epochs per autoencoder layer, a decaying learning rate, batch size 20, and L2 regularization.
B. Training the DBN: GPU execution reduces training time to under two days despite a layer containing more than 100,000,000 weights.
B. Training the DBN: The end-to-end pipeline runs malware in a sandbox, converts the sandbox file to a binary bit-string, and produces a 30-value signature.The 30-sized output vector is treated as the program signature.

C. Experimental Results

DeepSign compresses sandbox behavior into 30-dimensional signatures that preserve malware-family structure and support high-accuracy classification of unseen samples.

Variants from the same malware family mostly cluster together in signature space, indicating that the representations capture invariant malware behavior.The t-SNE visualization reduces the 30-dimensional signatures to two dimensions; labels color nodes but do not train the visualization.
98.6% test accuracy is achieved by a supervised deep network initialized with DBN weights, improving over the SVM result.The network uses the same layers as the DBN plus a six-neuron softmax output layer.

V. CONCLUDING REMARKS

The paper proposes deep-belief-network-based malware signatures to address the evasion weaknesses of specific-feature approaches. These signatures provide compact, invariant representations usable for unsupervised detection or supervised classification.

Existing signature-generation methods rely on specific malware aspects, allowing variants to evade detection through small code modifications.The paper frames this weakness as motivating a different signature-generation strategy.
DeepSign converts sandbox logs into binary strings and feeds them to an eight-layer neural network whose 30 outputs form each program's signature.The signatures are designed as invariant representations of malware behavior.
Unsupervised deep learning generates high-level invariant representations beyond vision, language, and speech, including malware signature generation.The conclusion identifies malware as a challenging domain where this approach is successfully applied.