Predicting Domain Generation Algorithms with Long Short-Term Memory Networks

Jonathan Woodbridge; Hyrum S. Anderson; Anjum Ahuja; Daniel Grant

Source-linked AI summary

Predicting Domain Generation Algorithms with Long Short-Term Memory Networks

Jonathan Woodbridge, Hyrum S. Anderson, Anjum Ahuja, Daniel Grant

arXiv:1611.00791v1Nov 2, 2016cs.CRcs.AI

TL;DR

Defending against DGA-based command-and-control traffic is difficult because dynamically generated domains make preemptive registration or blacklisting increasingly challenging. This paper uses featureless LSTM networks to classify DGA domains and families, reporting stronger performance than prior techniques and real-time deployment on commodity hardware.

Problem
Preemptively defeating DGA-based attacks becomes increasingly difficult as the rate of dynamically generated domains increases.
Method
The paper introduces a featureless LSTM classifier that operates on raw domain names and produces binary or family-specific predictions.
Results
The LSTM classifier performed significantly better than other real-time and retrospective techniques, classifying 90% of DGAs at a false positive rate of 10^-4.
Takeaways & Limitations
The classifier can run in real time on single domains using standard commodity hardware, supporting deployment across security settings.
Takeaways & Limitations
Class imbalance limits detection of families with little training support, and zero-support families with distinctive structures may not generalize well.

Abstract

from arXiv · show

Various families of malware use domain generation algorithms (DGAs) to generate a large number of pseudo-random domain names to connect to a command and control (C&C) server. In order to block DGA C&C traffic, security organizations must first discover the algorithm by reverse engineering malware samples, then generating a list of domains for a given seed. The domains are then either preregistered or published in a DNS blacklist. This process is not only tedious, but can be readily circumvented by malware authors using a large number of seeds in algorithms with multivariate recurrence properties (e.g., banjori) or by using a dynamic list of seeds (e.g., bedep). Another technique to stop malware from using DGAs is to intercept DNS queries on a network and predict whether domains are DGA generated. Such a technique will alert network administrators to the presence of malware on their networks. In addition, if the predictor can also accurately predict the family of DGAs, then network administrators can also be alerted to the type of malware that is on their networks. This paper presents a DGA classifier that leverages long short-term memory (LSTM) networks to predict DGAs and their respective families without the need for a priori feature extraction. Results are significantly better than state-of-the-art techniques, providing 0.9993 area under the receiver operating characteristic curve for binary classification and a micro-averaged F1 score of 0.9906. In other terms, the LSTM technique can provide a 90% detection rate with a 1:10000 false positive (FP) rate---a twenty times FP improvement over comparable methods. Experiments in this paper are run on open datasets and code snippets are provided to reproduce the results.

I. INTRODUCTION

DGA-based malware generates many domains, making algorithm-and-seed-dependent defenses difficult to maintain. The paper introduces a featureless, real-time LSTM classifier for binary detection and DGA-family identification.

DGAs generate hundreds to tens of thousands of pseudo-random domains daily, then try some domains to locate a C2 server.
Preemptive sinkholing, registration, or blacklisting becomes increasingly difficult as dynamically generated domains increase.
Public blacklists covered less than 1.2% of analyzed DGAs, while vendor coverage varied from 0% to 99.5% across malware families.
Real-time detection classifies individual domain names without contextual information, but prior methods often used time-consuming, circumvention-prone hand-crafted features.
The proposed LSTM classifier uses raw domain names, supports binary and multiclass classification, and classifies a domain in 20 ms on commodity hardware.
The paper reports an LSTM DGA predictor, experimental improvements over prior techniques, and source code using open datasets for reproducibility.

A. Domain Generation Algorithms

DGA classification includes retrospective, contextual approaches and harder real-time classification from individual domain names. Existing systems use clustering, statistical or linguistic features, HMMs, and sequential testing, with deployment limitations.

The paper evaluates DGA classification on 30 malware types, including ransomware, banking trojans, and information-stealing malware.
DGA algorithms range from uniform random generation to seed-based arithmetic or concatenated random dictionary words.
Suppobox domains are especially difficult to classify without context, yet this LSTM was the only real-time technique reported to classify them.
Classification approaches: Retrospective methods classify domain groups using distributions, clustering, shared hosts, DNS outcomes, and contextual statistics.
Classification approaches: Real-time approaches classify domains individually without additional contextual information, including HMMs and linguistic-feature classifiers.
Prior limitations: A sequential-testing system detected only 83% of domains in time to prevent a connection and could not perform multiclass classification.

C. LSTM Networks

The model uses an embedding layer, an LSTM for implicit character-pattern extraction, and logistic regression outputs for binary or multiclass DGA classification.

RNNs use recurrent state to capture sequence relationships, while LSTMs address vanishing gradients through gated state updates.
LSTM gates can retain, overwrite, reset, and expose state, allowing information to persist across long character sequences.
The model accepts variable-length character sequences without auxiliary feature extraction and uses a compact embedding–LSTM–logistic architecture.
Dropout is applied after the LSTM layer and before logistic regression to reduce overfitting during training.
The embedding layer maps valid domain characters into 128-dimensional vectors, with preprocessing that removes top-level domains and lowercases characters.
The LSTM learns implicit character patterns optimized for classification rather than explicitly representing domains as bigrams.
Binary classification outputs one value, whereas multiclass classification outputs one softmax-based score per known DGA family.

IV. EXPERIMENTAL SETUP

The experimental setup evaluates binary DGA-versus-non-DGA classification and multiclass identification using publicly available domain names and DGA data.

Experiments use publicly available domain names and DGA data for binary and multiclass classifier evaluation.

A. Evaluation Metrics

The paper evaluates classifiers with Precision, Recall, F1 score, and ROC, using both micro and macro averaging for class-based results.

Precision measures the purity of positively labeled instances.
Recall measures the completeness of instances that should have been labeled positive.
F1 score is the harmonic mean of Precision and Recall.
ROC measures the trade-off between true positive rate and false positive rate across classifier-score thresholds.AUC summarizes the ROC curve; an AUC of 1 is perfect, while 0.5 equals chance for binary classification.
Micro averaging weights classes by test-set size, whereas macro averaging weights all classes equally.The paper notes that macro averaging may better predict performance when class distributions are not representative.

B. Experimental Designs

The evaluation tests binary detection, detection of unseen DGA families, and multiclass identification of the generating family.

Three experimental designs measure binary detection, generalization to new DGA families, and multiclass family discrimination.
Binary classification: Binary classification uses ten-fold cross-validation to distinguish DGA from non-DGA domains.Evaluation includes ROC, class-level Precision, Recall, and F1, along with micro and macro averages.
New DGA families: The unseen-family experiment removes the ten smallest DGA families before training and evaluates Precision, Recall, and F1 on them.Both micro and macro averages are calculated across classes.
Multiclass classification: The multiclass experiment predicts which DGA family generated each domain.The random forest uses One-vs-Rest, while LSTM and Bigram classifiers perform direct multiclass classification.

C. Data

The experiments use open domain datasets and compare the featureless LSTM with HMM, bigram, and manually crafted-feature classifiers.

Data: The paper emphasizes open datasets and recommends expanded data for real-world systems to hinder adversarial reverse engineering.
Data: The dataset combines Alexa top 1 million non-DGA domains with approximately 750,000 DGA examples from thirty OSINT families.
Baselines: The comparison includes a featureless HMM, logistic regression on character bigrams, and a random forest using manually crafted domain features.
Baselines: Manual features include domain length, character-distribution entropy, and the vowel-to-consonant ratio.
Baselines: The random forest also uses Alexa n-gram cooccurrence, n-gram normality, and meaningful-character-ratio features.Normality uses n = 3, 4, and 5 as distinct features because larger n-grams performed better in preliminary experiments.
HMM baseline: Four HMMs model the non-DGA class and the three largest DGA classes, with classification based on a Neyman-Pearson likelihood ratio test.The improved HMM still performed worse than other techniques and was omitted from the multiclass experiment because the dataset contains thirty unevenly supported classes.
Comparison scope: Retrospective-technique comparisons are limited to binary classification because the dataset contains domain names without contextual information.

V. RESULTS

This section presents results from the paper’s three experiments and interprets model performance.

Results for the three experiments are presented in this section.
The section includes an interpretation of model performance.
The passage introduces the results section without reporting an experimental finding.

A. Binary Classification

Binary experiments evaluate whether classifiers distinguish DGA from non-DGA domains and how robustly they detect families under leave-one-out testing. The LSTM performs strongly overall, while performance varies with family representation and structure.

Binary Classification: 0.9993 AUC is achieved by the LSTM, compared with 0.9939 for the bigram model in binary DGA detection.At 90% detection, the LSTM has a 1 in 10,000 false positive rate, versus 1 in 550 for bigrams.
Binary Classification: Matsnu is undetectable by all algorithms, whereas the LSTM detects suppobox to some extent because its larger training set permits dictionary learning.The suppobox training set was about twenty times larger than matsnu’s, allowing repeated dictionary words to appear during training.
Binary Classification: The HMM performs worse here than previously reported because this evaluation uses over 30, more diverse DGA families.The earlier HMM results used only a small homogeneous set of families.
Binary Classification: In leave-one-out testing, classifiers are evaluated on DGA families absent from training, with Recall reported because the test set contains no non-DGA domains.The experiment removes the ten smallest families before training and evaluates the remaining classifiers on those held-out classes.
Binary Classification: The manual-feature random forest has the best micro and macro averages, but the LSTM performs best on more individual families.The LSTM leads on five families, compared with four for the manual-feature classifier.
Binary Classification: LSTM blind spots include beebone and symmi, whose rigid or patterned domains lack sufficient training examples.The authors suggest adversarial-network training as a way to address these blind spots.

C. Multiclass

Multiclass classification distinguishes among DGA families, but similar character distributions and limited class representation create confusion. Grouping related families into super families improves performance, with the LSTM remaining the strongest classifier.

C. Multiclass: The LSTM significantly outperforms manual-feature random forest and bigram classifiers on micro- and macro-averaged multiclass Precision, Recall, and F1.Cryptolocker remains undetectable by multiclass classifiers, although binary classifiers distinguish it from other families.
C. Multiclass: Cryptolocker is frequently misclassified as ramnit because both have uniform unigram distributions over the same range.Both families are generated through sequences of multiplies, divisions, and modulos based on a single seed.
C. Multiclass: Suppobox can resemble benign Alexa domains because it concatenates randomly selected English dictionary words.This distributional similarity contributes to confusion with the benign set.
C. Multiclass: Dyre is extremely dissimilar to other algorithms, with a nearly uniform distribution dominated by hexadecimal characters.Non-hexadecimal letters are rare in dyre domains.
C. Multiclass: Similar unigram distributions correlate with DGA families that are often misclassified, beyond the effect of limited training representation.Many DGAs produce domains that are nearly identical in character distributions, making family-level classification difficult.
C. Multiclass: All three classifiers perform much better on super families, and deployment is suggested to alert on DGA groups rather than individual families.The LSTM still performs significantly better than the other algorithms in the super-family experiment.

D. Model Interpretability

The paper analyzes how the LSTM classifier represents domain characters and updates internal cell states, while noting that most state functions remain difficult to interpret.

Each LSTM layer is jointly optimized for the binary classification task, so independent layer analysis provides intuition rather than isolated functional explanations.
The embedding layer learns a 128-dimensional vector for each valid domain character, unlike the orthonormal one-hot encoding used by the logistic regression bigram model.
Character embeddings cluster alphabetic characters, numeric digits, and underscore, reflecting similarity or interchangeability for DGA versus non-DGA discrimination.The learned placement suggests that replacing “9” with “5” affects the classifier less than replacing “9” with “w”.
Each LSTM cell updates its state from the current embedded character and prior emission, while its emission depends on the current state, input, and previous emission.
Some cells track interpretable patterns such as domain length, random alphanumeric sequences, or hexadecimal sequences, but most states lack a clearly identifiable function.This interpretability is weaker than that reported for some natural-language LSTM applications.

VI. CONCLUSION

The paper concludes that featureless LSTM networks can classify DGA-generated domains in real time and outperform prior techniques, while interpretability and class imbalance remain limitations.

The LSTM classifier uses raw domain names without manually engineered features and can run in real time on single domains using commodity hardware.The approach is intended to simplify maintenance and deployment in security settings.
90% detection with a 10^-4 false positive rate was achieved on publicly available datasets, with performance significantly better than other real-time and retrospective techniques.
Multiclass classification can provide context about the origin and intent of domain-generating malware.
DGA families modeled on Alexa’s character distributions are hardest to classify, but significant training data and super-family grouping enabled their distinction.
Most LSTM states lacked clear functional interpretability, although character embeddings and some cells tracked alphabetic, numeric, hexadecimal, or random-character patterns.
Class imbalance limits detection of families with little training support, including matsnu, symmi, and cryptowall; with zero support, generalization can fail.Manually engineered features detected some distinctive families that the featureless LSTM missed.