Source-linked AI summary
Tiresias: Predicting Security Events Through Deep Learning
Yun Shen, Enrico Mariconti, Pierre-Antoine Vervier, Gianluca Stringhini
TL;DR
Existing security-event predictors generally forecast binary outcomes rather than the specific actions attackers will take. Tiresias uses recurrent neural networks to predict the next event from observed event histories, reaching precision up to 0.93 on large-scale intrusion-prevention data while supporting monitoring and retraining when performance drops.
Problem
Prior systems predict binary outcomes but not the specific attacker techniques or actions, while labeled data and changing attacker behavior complicate model construction and maintenance.
Method
Tiresias trains a recurrent neural network to model the probability distribution of the next security event from historical event sequences.
Results
Precision reached up to 0.93 on 3.4 billion security events collected from 740k machines over 27 days, with stable models and automatic retraining after sudden precision drops.
Takeaways & Limitations
Long-term recurrent memory was key for accurate prediction in complex, noisy multi-step attacks, outperforming simpler short-term-memory systems.
Takeaways & Limitations
Because the data is passively collected from predefined intrusion-prevention signatures, Tiresias can predict only events observed by that product and may miss rare or newly exploited attacks.
Abstract
from arXiv · showhide
With the increased complexity of modern computer attacks, there is a need for defenders not only to detect malicious activity as it happens, but also to predict the specific steps that will be taken by an adversary when performing an attack. However this is still an open research problem, and previous research in predicting malicious events only looked at binary outcomes (e.g., whether an attack would happen or not), but not at the specific steps that an attacker would undertake. To fill this gap we present Tiresias, a system that leverages Recurrent Neural Networks (RNNs) to predict future events on a machine, based on previous observations. We test Tiresias on a dataset of 3.4 billion security events collected from a commercial intrusion prevention system, and show that our approach is effective in predicting the next event that will occur on a machine with a precision of up to 0.93. We also show that the models learned by Tiresias are reasonably stable over time, and provide a mechanism that can identify sudden drops in precision and trigger a retraining of the system. Finally, we show that the long-term memory typical of RNNs is key in performing event prediction, rendering simpler methods not up to the task.
1 INTRODUCTION
Modern attacks unfold through heterogeneous, multi-step behaviors, while prior prediction systems mainly forecast binary outcomes. Tiresias addresses this gap by predicting specific future security events from past system events.
- Attackers combine heterogeneous steps, including spearphishing, drive-by downloads, and privilege escalation, to compromise systems.
- Existing prediction systems mainly forecast binary outcomes, such as whether an attack or data breach will occur.
- 3.4 billion security events from 740k machines over 27 days were used to evaluate Tiresias.
- Precision reached up to 0.93 for predicting the specific malicious event that would occur next.
- Tiresias models were fairly stable, but sudden precision drops triggered automatic retraining after changes in attacks or systems.
- Recurrent Neural Networks outperformed simpler short-term-memory systems such as Markov Chains on complex, noisy multi-step attacks.
2 BACKGROUND AND MOTIVATION
Security-event prediction must handle variable attack sequences, unrelated noise, and potentially concurrent attacks from different adversaries. Tiresias therefore targets the next event in a context-rich, multi-attack environment.
- Coordinated attacks can follow different event orders across endpoints, making a single fixed sequence pattern inadequate.
- Endpoints may observe unrelated security events between events belonging to a coordinated attack.
- Different attacks from multiple adversary groups may occur simultaneously on the same endpoint.
- The desired model must understand noisy events, recognize attacks in context, and forecast the upcoming security event.
3 METHODOLOGY
Tiresias collects and preprocesses endpoint event sequences, trains and validates a recurrent model, predicts the next event distribution, and monitors operational performance. Its recurrent memory design supports prediction amid noisy sequential inputs.
- Tiresias uses four phases: data collection and preprocessing, model training and validation, security-event prediction, and performance monitoring.
- A recurrent neural network models the probability distribution of the next event from a rollback window of historical events.
- Validation uses data from different machines to assess generalization beyond the training endpoints.
- At prediction time, Tiresias ranks the next-event distribution and selects the event with the maximum probability.
- The recurrent memory array uses forget, input, and output gates together with cell and hidden states.
- A stochastic design samples the memory-cell distribution and activates only the most likely cell to improve resilience to noisy input.
4 DATASETS
The evaluation uses anonymized security-event data from Symantec’s intrusion prevention product, organized into datasets spanning 27 days and billions of events. These datasets support tests of predictive effectiveness, stability over time, and reusability.
- Data source: Tiresias is evaluated on passively collected, anonymized security events from Symantec’s intrusion prevention product.Each event includes a machine ID, timestamp, event ID, description, system actions, and optional metadata; machine IDs reconstruct per-machine event sequences before disposal.
- Dataset construction: 27 days of data comprise two datasets containing over 3.4 billion security events.D1 contains over 2.2 billion events across 17 days, while D2 contains 1.2 billion events collected across selected dates from November 2017 through February 2018.
- Evaluation design: D1 supports validation, baseline comparison, training-period variation, and short-term stability analysis.The first five days validate the approach and compare three baselines; the first seven days train models evaluated on the remaining ten days.
- Evaluation design: D2 tests longer-term accuracy and compares models trained on January data with models trained on the older D1 period.The comparison focuses on whether prediction performance remains reusable when training data are months older than part of the evaluation data.
- Data limitations: The model can predict only events observed by Symantec’s intrusion prevention product.Passively collected data exclude events blocked by other products and events that do not match predefined signatures.
5 EVALUATION
The evaluation tests Tiresias on exact next-event prediction, comparisons with simpler sequence models, varying training periods, and long-term stability. Results show strong, relatively stable precision, with longer training helping mainly for unusual attack behavior.
- Evaluation design: The evaluation measures exact upcoming-event prediction across multiple experimental settings, including baseline comparisons, training-period variation, and model reliability over time.The experiments address performance, baseline comparisons, training duration, reuse without retraining, and the influence of recurrent long-term memory.
- Overall prediction results: Over 80% precision, recall, and F1-measure were achieved when predicting the final event on machines from sequences of previous events.The evaluation considered 4,495 possible security events and used 14,396 machines excluded from the initial training set.
- Overall prediction results: Precision, recall, and F1 decreased by less than 0.05 in the worst case across same-day and following-day evaluation, although precision declined 4% from 0.83 to 0.79 for one model.The dominant events differed between training and test data, while the three metrics remained balanced and on the same scale.
- Baseline comparison: Tiresias outperformed first-order Markov chains, 3-gram models, and spectral learning, while 3-grams ranked above Markov chains and Markov chains above spectral learning.The ordering among baselines indicates that sequence memory mattered in this comparison.
- Training-period length: 0.819 precision was achieved on average with one week of training, 0.3% higher than models trained on one day, producing similar overall results.The comparison tested five one-day models and one seven-day model on ten days of later data.
- Training-period length: A precision standard deviation of approximately 0.02 over ten days indicates stable short-term performance, while week-long training better handled deviating or rare attack behaviors.One-day and one-week models were similarly effective for test data close in time to training; longer training was more useful for outlying behavior.
- Stability over time: Months after training, precision did not decrease dramatically, and day-long training performed comparably to week-long training, although a December distribution shift increased precision unexpectedly.The results support rapid convergence with one or a few days of training and good aging of the model over several months.
- Sequence analysis: Successful guesses were associated with longer, more distinctive event sequences, whereas incorrect guesses involved frequent sequences differing mainly in their final event.The sequence analysis links prediction confidence and correctness to the structure and uniqueness of preceding events.
6 CASE STUDIES
Tiresias is evaluated in real-world case studies involving coordinated multi-step attacks and coarser attack-category predictions. The system predicts event sequences step by step, adapts using observed context, and achieves strong precision across different settings.
- Multi-step attack detection: Tiresias identifies candidate coordinated attacks by analyzing event sequences with similar frequencies across machines and applying a support threshold.The method allows variability with a 10% frequency error margin.
- Sequential prediction: Tiresias predicts upcoming events sequentially, feeding actual observations back into its memory after both correct and incorrect predictions.This feedback helps the system remain aligned with the observed attack sequence.
- Sequential prediction: Tiresias can recover from repeated incorrect predictions by using contextual information, eventually producing a confident correct prediction.In the illustrated case, the model initially missed e6 twice after e7, then corrected its behavior.
- Prediction granularity: Category-level predictions capture shared attack traits when exact event prediction fails, including properties such as targeted software, protocol, or attack type.The category representation is coarser than the original IPS event labels.
- Prediction granularity: 88.9% precision was achieved for predicting whether events should be blocked or allowed, an 8% increase over exact-event prediction on the same day.The value of this relaxed granularity depends on category accuracy and the desired prediction granularity.
7 DISCUSSION
Tiresias performs well across changing time periods, but its performance is constrained by rare events and radical changes in observed attack patterns. New attack sequences may require retraining or may remain undetected until represented in the data.
- Limitations: Rare intrusion attempts may be predicted incorrectly because the recurrent neural network lacks enough training samples for them.The paper notes that existing statistical and machine-learning methods also lack a satisfactory solution for rare-event prediction.
- Limitations: Tiresias may need retraining when newly observed security events or radical changes in attack data alter the observed distribution.The discussion contrasts this with slower precision decline when the training set is older but the data remains comparatively stable.
- Limitations: A zero-day attack may be detected when it reuses known action sequences, but a novel multi-step sequence may not be detected.The boundary depends on whether the new attack follows previously observed event patterns.
8 RELATED WORK
Prior work applied forecasting and recurrent neural networks to security-related tasks, while DeepLog targeted anomaly detection in narrow, low-variety log environments. Tiresias instead addresses coordinated multi-step attacks in noisy environments with wider event variety.
- Prior security forecasting work predicted future website maliciousness and the existence of private proof-of-concept exploits.
- RNNs have been applied to binary analysis, function identification, function-type recovery, and malware classification.
- DeepLog focused on anomaly detection in Hadoop and OpenStack logs containing only 29 and 40 event types, respectively.
- Tiresias targets multi-step coordinated attacks in noisy environments with a wider variety of events than DeepLog’s specific log settings.
9 CONCLUSIONS
The paper presents Tiresias for security-event prediction and evaluates it on an extensive intrusion-prevention-system alarm dataset. It reports high precision and stable results when models are trained months before testing.
- Tiresias predicts security events using an extensive intrusion-prevention-system alarm dataset from a major security-firm product.
- Tiresias reaches high precision on the complex task of security-event prediction.
- Models trained months before application to a test set still produce stable results.