Source-linked AI summary
Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams
Aaron Tuor, Samuel Kaplan, Brian Hutchinson, Nicole Nichols, Sean Robinson
TL;DR
Insider-threat detection must filter large, heterogeneous system-log streams that can overwhelm human analysts. The paper presents an online unsupervised deep-learning system that models normal user behavior with DNN and RNN variants and decomposes anomaly scores into feature contributions. On CERT v6.2, the models outperformed PCA, SVM, and Isolation Forest baselines in threat-detection recall, while the best model’s threat events averaged in the 95.53 percentile of anomaly scores.
Problem
High-volume, heterogeneous real-time system logs create a need for automated filtering to support analysts reviewing potential insider threats.
Method
An online unsupervised deep-learning system models user-characteristic activity with DNN and RNN variants and decomposes anomaly scores into interpretable feature contributions.
Results
The DNN and LSTM models outperformed Isolation Forest, SVM, and PCA baselines on the CERT Insider Threat v6.2 dataset using threat-detection recall.
Takeaways & Limitations
The system is designed to filter high-volume streams to a manageable set of analyst-reviewed events while conveying why user-days were flagged.
Takeaways & Limitations
The evaluation found equivalent LSTM and DNN performance, and the authors suspect LSTM advantages may require real-world data with more complicated temporal patterns.
Abstract
from arXiv · showhide
Analysis of an organization's computer network activity is a key component of early detection and mitigation of insider threat, a growing concern for many organizations. Raw system logs are a prototypical example of streaming data that can quickly scale beyond the cognitive power of a human analyst. As a prospective filter for the human analyst, we present an online unsupervised deep learning approach to detect anomalous network activity from system logs in real time. Our models decompose anomaly scores into the contributions of individual user behavior features for increased interpretability to aid analysts reviewing potential cases of insider threat. Using the CERT Insider Threat Dataset v6.2 and threat detection recall as our performance metric, our novel deep and recurrent neural network models outperform Principal Component Analysis, Support Vector Machine and Isolation Forest based anomaly detection baselines. For our best model, the events labeled as insider threat activity in our dataset had an average anomaly score in the 95.53 percentile, demonstrating our approach's potential to greatly reduce analyst workloads.
Introduction
Insider threat is a complex, growing organizational challenge, while cyber defenders must assess high-volume, heterogeneous real-time activity. The paper proposes online unsupervised deep learning to filter logs and identify anomalous user behavior for analyst review.
- Insider threat includes potentially harmful employee actions ranging from unsanctioned data transfer to sabotage and multi-year intelligence campaigns.
- Cyber defenders face high-velocity, heterogeneous streams spanning many entities and activities, motivating automated filtering for analysts.
- The proposed online unsupervised system learns user-characteristic activity with DNN and RNN models while assessing normality or anomaly in real time.
- The models decompose anomaly scores into human-readable feature contributions to help analysts interpret flagged behavior.
Prior Work
Prior work addresses parts of the requirements for real-time, multivariate anomaly detection in system logs, but the paper positions its approach as addressing the full set of stated constraints and features.
- Online anomaly detection for multivariate sequences has been described as underdeveloped in prior work.
- A practical system must track multiple entities, analyze structured multivariate data, adapt to shifting activity distributions, and provide interpretable judgments.
- The paper states that surveyed prior approaches address subsets of these requirements, whereas its work addresses all of them.
- Isolation Forest is included as a baseline because prior insider-threat work used it for unsupervised network-log detection and feature-based explanations.
- Earlier neural-network approaches differed by using unstructured inputs or offline training rather than structured, online processing.
- Related recurrent-network applications in other domains did not process multivariate combinations of categorical and continuous features.
System Description
The system converts user log activity into time-indexed feature vectors and models each user with a parameter-sharing neural network, using either DNN or RNN variants.
- Raw system-log events are aggregated into count features, producing one vector for each user and day.
- Each user’s feature sequence is processed by a neural network, with separate DNN and RNN system variants.
- User models share parameters, while RNN variants maintain separate hidden states.
Feature Extraction
Feature extraction transforms heterogeneous system logs into categorical user attributes and continuous activity counts suitable for neural-network input. The count features enumerate combinations of activity characteristics over fixed time windows.
- The system extracts categorical user attributes and continuous count features from heterogeneous tracking sources.
- Categorical features include organizational attributes such as role, department, and supervisor.
- The system accumulates counts for 408 user activities over a fixed time window, such as removable-media file copies during specified hours.
- Figure 2 enumerates count features by combining one choice from each activity-characteristic set along a right-to-left path.
- Table 1 lists the categorical variables used in the experiments together with the number of distinct values in each category.
Structured Stream Neural Network
The system maps each user’s daily feature-vector sequence to probability distributions for subsequent observations using shared DNN or recurrent LSTM models. Negative log probability produces anomaly scores that can be decomposed by variable and feature for analyst interpretation.
- Model architecture: The model jointly trains shared-parameter neural networks across users to map daily feature-vector sequences to probability distributions over subsequent observations.The DNN processes current inputs, whereas the RNN incorporates sequence history and maintains user-specific hidden states.
- Model architecture: The DNN computes each hidden representation from the current input through stacked nonlinear layers with trainable weight matrices and bias vectors.The activation function may be ReLU, tanh, or logistic sigmoid.
- Model architecture: The LSTM computes hidden states from the input sequence and cell states, allowing the model to capture temporal patterns in user behavior.LSTM weights are shared across users, while recurrent states encode sequential information.
- Structured prediction: The predicted observation combines a 408-dimensional continuous count vector with six categorical variables: role, project, functional unit, department, team, and supervisor.Categorical probabilities use softmax outputs; count probabilities use multivariate normal variants with identity or diagonal covariance.
- Anomaly scoring: Anomaly scores are negative log probabilities of observations conditioned on hidden representations, with next-time-step and same-time-step prediction defining alternative targets.Next-time-step prediction conditions on information through t-1, while same-time-step prediction uses information through t.
- Anomaly scoring: The anomaly score decomposes into variable and individual-feature contributions, enabling analysts to identify which user behaviors drive a flagged user-day.The paper gives abnormal attachment emails to uncommon recipients during overnight hours as an example of an interpretable contribution pattern.
Online Training
Standard batch training is unsuitable for unbounded streams because data cannot feasibly be stored or repeatedly revisited while predictions must be made during learning. The paper therefore adapts DNN and RNN training to process observations online.
- Online Training: Online anomaly detection must handle an effectively unbounded streaming dataset while making predictions as the model learns.Repeatedly training on all data is infeasible, and retraining only on recent events may exclude important historical events.
- Online Training: For DNNs, online training restricts the model to observing each sample only once.
- Online Training: For RNNs, the method trains multiple user sequences concurrently and updates shared weights whenever a new feature vector arrives.This is logically equivalent to one RNN per user with shared weights and separate hidden-state sequences.
Baseline Models
The evaluation compares the proposed DNN and RNN anomaly detectors with established one-class SVM, isolation forest, and PCA baselines.
- Baseline Models: The study evaluates its DNN and RNN models against popular anomaly, novelty, and outlier detection methods.
- Baseline Models: The baselines include one-class support vector machine, isolation forest, and principal component analysis.
- Baseline Models: The one-class SVM and isolation forest implementations come from scikit-learn’s novelty and outlier detection functionality.
Experiments
The experiments use the CERT v6.2 dataset, chronological development/test splitting, and recall-oriented tuning to evaluate online anomaly detection models. Results examine feature choices, prediction timing, covariance structure, baselines, anomaly rankings, and analyst-budget recall.
- Data: CERT v6.2 contains 135,117,169 events from 4,000 users over 516 days, covering five event sources and five injected insider-threat scenarios.The dataset also includes categorical user metadata.
- Data: The dataset is split chronologically into approximately 85% development and 15% test data, with predictions evaluated at user-day granularity.The test set contains over 40% of threat user-days despite only 15% of events, and experiments retain weekdays only.
- Tuning: Models are tuned with random hyper-parameter search using CR-k, the summed recall across analyst budgets up to k, with CR-1000 selected for model selection.CR-k approximates area under the recall curve and reflects fixed daily budgets, rare positives, and higher missed-detection costs.
- Feature representation: Excluding categorical features improves LSTM performance, so subsequent experiments use count features only.The authors suggest the added categorical-feature complexity may hinder trainability or be unnecessary for this dataset.
- Prediction mode: Same-time-step prediction outperforms next-time-step prediction for both DNN and LSTM models, while the two architectures perform equivalently.The authors suspect CERT lacks enough multi-day temporal patterns for LSTM to provide an advantage.
- Covariance and baselines: Diagonal covariance outperforms identity covariance, and Isolation Forest is the strongest baseline, ranking third behind DNN-Diag and LSTM-Diag.The diagonal model can account for variance trends and, in contextual settings, condition mean and variance on local context.
- Anomaly analysis: After an initial burn-in period, all malicious user-days rank above the 50th anomaly percentile, with most above the 95th percentile.The model begins untrained and quickly learns users’ normal behavior from observed examples.
- Analyst-budget analysis: With a daily budget of 425, DNN-Diag, LSTM-Diag, and Isolation Forest reach 100% recall; LSTM-Diag reaches 90% recall at budget 250.The latter corresponds to a reported 93.5% reduction in data analysts need to review.
Conclusions
The paper presents an online unsupervised deep learning system that models normal user behavior to identify and explain anomalous activity in streaming logs. Its evaluation supports the approach, while the authors identify broader streaming tasks and finer-grained temporal analysis as important extensions.
- The system models normal user behavior rather than explicitly modeling insider threat, which can take new and different forms.
- Its online architecture filters high-volume streaming logs into manageable events for analyst review and provides feature-level explanations for anomalous user behavior.Examples include abnormal file uploads during a particular time window.
- On the CERT Insider Threat v6.2 dataset, DNN and LSTM models outperformed Isolation Forest, SVM, and PCA baselines.
- The authors propose applying the domain-agnostic anomaly-detection model to a wider range of streaming tasks.They report that the LSTM performed equivalently to the DNN in these experiments but may perform better with more complicated temporal patterns.
- Daily aggregation by user may miss anomalous patterns within a single day, motivating future event-level sequence modeling with LSTM hidden states.The authors suggest this could reduce feature engineering and narrow the events analysts inspect.