PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Anomaly Detection in Log Data: A Survey

Max Landauer, Sebastian Onder, Florian Skopik, Markus Wurzenberger

arXiv:2207.03820v2cs.LG

TL;DR

The paper addresses the difficulty of detecting relevant incidents in large, unstable, and unstructured log data, alongside limited state-of-the-art overview. It conducts a systematic literature review of deep-learning architectures, preprocessing, detection methods, and evaluations, finding diverse applicable architectures and identifying data-set, benchmarking, and explainability issues.

  • Problem

    Large log volumes make manual analysis infeasible, while existing surveys insufficiently cover deep-learning architectures, preprocessing, and challenges for log anomaly detection.

  • Method

    The paper conducts a systematic literature review of model architectures, unstructured-log transformations, anomaly detection methods, and evaluations.

  • Results

    The survey finds diverse suitable architectures, varied training features and numeric encodings, and detection through classification or other network outputs.

  • Takeaways & Limitations

    The review emphasizes the need for adequate diverse data sets, state-of-the-art benchmarks, and improved explainability to strengthen applicability and reliability.

Abstract

from arXiv · show

Automatic log file analysis enables early detection of relevant incidents such as system failures. In particular, self-learning anomaly detection techniques capture patterns in log data and subsequently report unexpected log event occurrences to system operators without the need to provide or manually model anomalous scenarios in advance. Recently, an increasing number of approaches leveraging deep learning neural networks for this purpose have been presented. These approaches have demonstrated superior detection performance in comparison to conventional machine learning techniques and simultaneously resolve issues with unstable data formats. However, there exist many different architectures for deep learning and it is non-trivial to encode raw and unstructured log data to be analyzed by neural networks. We therefore carry out a systematic literature review that provides an overview of deployed models, data pre-processing mechanisms, anomaly detection techniques, and evaluations. The survey does not quantitatively compare existing approaches but instead aims to help readers understand relevant aspects of different model architectures and emphasizes open issues for future work.

I. INTRODUCTION

Log data can reveal failures and security incidents, but its scale and variability make manual analysis and fixed signatures inadequate. The survey therefore reviews deep-learning approaches, their input transformations, detection methods, and evaluations.

  • Modern logs contain traces of normal operations as well as failures, availability issues, and security incidents requiring timely operator action.
  • Manual analysis is infeasible for modern log volumes, while fixed keyword signatures cannot handle scenarios that were not known beforehand.
  • Anomaly detection learns a normal-behavior baseline and reports deviations as potentially adverse activities for operator attention.
  • Existing surveys provide limited coverage of deep-learning log anomaly detection and do not adequately resolve feature extraction, input transformation, or architecture selection.
  • The systematic review surveys model architectures, unstructured-log transformations, anomaly differentiation methods, and evaluation practices.
  • The survey organizes its investigation around challenges, algorithms, preprocessing, anomaly types, evaluation, labels and incremental learning, and reproducibility.

A. Preliminary Definitions

The paper frames log data as heterogeneous chronological event records and anomaly detection as identifying rare or unexpected behavior. It highlights challenges spanning representation, instability, labeling, scale, streaming, and explainability.

  • Log data are chronological application-generated events ranging from structured vectors to semi-structured objects and unstructured heterogeneous messages.
  • Parsers extract log templates and parameters, enabling event identifiers and values to be derived from otherwise unstructured messages.
  • Anomalies are rare or unexpected instances whose conformity can be assessed using associated attributes and similarity measures.
  • Challenges: Deep-learning systems face heterogeneous numeric-input requirements, evolving events and baselines, class imbalance, diverse anomaly manifestations, and limited anomaly labels.
  • Challenges: Interleaved distributed logs, poor data quality, continuous streams, and very large volumes complicate sequence recovery, reliable learning, single-pass processing, and real-time deployment.
  • Challenges: Neural-network approaches generally provide less explainability than conventional machine-learning methods, complicating justified responses to critical behavior and security incidents.

III. SURVEY METHOD

The survey method combines a database search with explicit literature-collection and evaluation procedures. Its search string targets log data, anomaly detection, and deep learning while including terminology variants.

  • The review method first collects relevant literature and then analyzes retrieved papers using defined evaluation criteria.
  • Figure 1 presents the composition of the search string used to retrieve relevant literature.
  • The search string targets publications containing the three concepts log data, anomaly detection, and deep learning, with alternative terminology to reduce missed studies.
  • Seven scientific databases were searched in January 2022, returning 2925 publications for subsequent selection.

2) Selection of Relevant Publications:

The selection procedure narrows the searched literature through topical, methodological, quality, and format criteria. A two-stage screening reduced the collection to 62 included papers.

  • The main inclusion criterion required a multilayer neural-network model designed for anomaly detection in heterogeneous and unstructured log data.
  • 2) Selection of Relevant Publications:: Exclusions covered methods without log-data applicability, superseded similar studies, unmodified existing approaches, non-English works, unavailable electronic publications, and non-paper formats.
  • 2) Selection of Relevant Publications:: The review imposed no publication time range but sought to omit studies failing minimum scientific standards.
  • 2) Selection of Relevant Publications:: Quality screening required explicit study findings, rigorous model descriptions, sound evaluations, referenced or described data sets, and clear visualizations.

B. Reviewed features

The survey evaluates reviewed approaches across deep-learning models, preprocessing, anomaly detection, operation, and reproducibility. Its feature matrix records these assessments for systematic analysis.

  • Deep learning and operation: The survey assesses deep-learning models, training losses, online learning, and supervised-learning modes.
  • Preprocessing: Preprocessing questions cover raw-log handling, feature extraction, and vector representation.
  • Anomaly detection: Anomaly-detection questions examine anomaly types, neural-network outputs, and differentiation from normal samples.
  • Evaluation and reproducibility: Evaluation questions cover data sets, metrics, runtime performance, benchmarks, and public availability of data and source code.
  • Evaluation and reproducibility: The resulting feature matrix supports the survey’s analyses and discussions.

IV. SURVEY RESULTS

The surveyed literature shows rapid recent growth in deep learning for log anomaly detection, while citation influence is concentrated in foundational work on event sequences.

  • 1) Publications per Year:: 58 out of 62 reviewed publications were published in 2019 or later.The literature search was conducted at the beginning of 2022, so only two included publications came from that year.
  • 2) Citations:: DeepLog had the highest citation count among the reviewed publications as of January 2023.The 2017 paper was the first reviewed approach using deep learning to detect anomalous event sequences.
  • 2) Citations:: Yang et al.’s earlier paper had significantly fewer citations because it focused on tokens in single log events rather than event sequences.

1) Deep Learning Models:

The survey finds broad architectural diversity, with recurrent neural networks most common and specialized models addressing unsupervised learning, attention, graphs, and combined architectures.

  • Recurrent architectures: 36 out of 62 reviewed approaches used recurrent neural networks for anomaly detection.RNN feedback mechanisms retain state over time, supporting sequential log patterns.
  • Recurrent architectures: LSTM approaches commonly train on event-occurrence sequences to disclose unusual sequential patterns.Bi-LSTM experiments suggest higher performance than LSTM, while GRUs are computationally more efficient.
  • Unsupervised models: Autoencoders create codes and reconstruct inputs, avoiding the need for labeled input data.
  • Specialized architectures: Transformers use self-attention to weight inputs according to occurrence context, whereas GNNs ingest graph inputs such as session graphs.
  • Combined architectures: Some approaches combine architectures, such as an MLP combining outputs from a VAE and an adversarially trained Transformer.
  • Training objectives: Cross-entropy is the most common reviewed loss function, while other studies use hyper-sphere, mean-squared-error, probabilistic, custom, or adversarial objectives.14 publications do not state their loss function.

3) Operation mode:

Real-world log detection must address continuous, changing streams and unstructured inputs. Reviewed approaches use online adaptation, varied supervision assumptions, parsing or tokens, event grouping, and vectorization.

  • 3) Operation mode:: Online or incremental learning processes each data instance once with approximately linear runtime as log streams change over time.
  • 3) Operation mode:: Most supervised approaches train offline because labels require manual analysis or validation of delimited data sets.
  • 3) Operation mode:: Many approaches labeled unsupervised actually assume anomaly-free training data and therefore operate semi-supervised.Anomalies in training data can alter network weights and deteriorate subsequent detection.
  • Data preparation: Raw logs are commonly preprocessed with parsers that extract event identifiers and parameters, while token strategies split messages into cleaned word lists.
  • 2) Event grouping:: Deep learning usually groups events into windows or sessions to analyze sequential patterns and temporal correlations.Session identifiers can represent parallel program workflows, but not all log data provides such identifiers.
  • 2) Event grouping:: Time-window grouping makes event frequencies harder to treat as time series when resulting windows span varying durations.

3) Feature extraction:

The survey describes how unstructured logs are transformed into structured feature vectors and alternative representations suitable for neural-network analysis.

  • Feature extraction: Parsing and token-based preprocessing extract structured features from otherwise unstructured logs.Token sequences can represent each log line as a sentence, while token counts require comparing and counting tokens.
  • Event features: Event sequences encode chronologically ordered event-type identifiers within fixed, sliding, or session windows.Event counts instead form d-dimensional vectors recording each log key’s occurrence frequency.
  • Statistical features: Additional event statistics include inter-arrival times, seasonal-log percentages, message lengths, activity rates, entropy scores, and occurrence bursts.
  • Vector representations: Common neural-network inputs include event-ID sequences, count vectors, semantic vectors, one-hot encodings, embeddings, and parameter vectors.Representations may encode event types, tokens, learned semantics, or actual parsed parameter values.
  • Alternative representations: Graphs and transfer matrices provide less common alternatives for encoding dependencies between log events.Graphs transform event sequences into session graphs, while transfer matrices encode transitions between event types.

D. Anomaly Detection Techniques

Reviewed approaches target different anomaly types and convert neural-network outputs into labels, scores, or predictions that support anomaly decisions.

  • Anomaly types: The survey distinguishes outlier, sequential, frequency, and statistical anomalies according to the event properties that deviate from normal behavior.Sequential anomalies involve changed execution paths, while frequency and statistical anomalies concern event counts or other quantitative properties.
  • Anomaly types: Most reviewed approaches focus on collective sequential anomalies rather than isolated outlier events.
  • Network outputs: Neural networks produce scalar or vector outputs that can serve as anomaly scores, including reconstruction errors from autoencoders.Large reconstruction errors indicate samples that are difficult to reconstruct from models trained on normal data.
  • Detection decisions: Binary and multi-class classification assign anomaly labels, whereas prediction-based methods flag events absent from the top n predicted log keys.The candidate count n regulates the tradeoff between true-positive and false-positive rates.
  • Detection decisions: RE, TRA, and VEC outputs are typically thresholded anomaly scores, while PRD outputs are compared with the highest-probability log-key candidates.The survey notes exceptions, including supervised approaches based on reconstruction errors.

E. Evaluation & Reproducibility

The survey finds that evaluations rely heavily on a small set of public log datasets and standard metrics, while reproducibility is limited by scarce original implementations.

  • Evaluation scope: The survey presents commonly used public datasets and evaluation metrics while also considering benchmark selection and implementation availability.
  • Data sets: Evaluations commonly use only a few datasets, with HDFS, BGL, Thunderbird, and related datasets comprising the vast majority of reviewed evaluations.
  • Data sets: The datasets cover system failures, hardware and software problems, injected errors, and logs from conventional and distributed computing systems.Some datasets instead represent malicious activity, including data exfiltration, unauthorized access, backdoors, and brute-force attacks.
  • Evaluation metrics: Evaluation results count true positives, false positives, false negatives, and true negatives, with aggregation choices affecting how samples and detections are judged.Session-level labels, especially in HDFS, require evaluation procedures that differ from individually labeled events.
  • Evaluation metrics: Precision, recall or true positive rate, false positive rate, and F1-score are the metrics used by almost all reviewed publications.Accuracy appeared in 15 publications, while area under precision-recall curves was less common.

3) Benchmark approaches:

Benchmark comparisons are widespread, with DeepLog dominating usage; however, original source-code publication remains uncommon across the reviewed literature.

  • Benchmark approaches: DeepLog is the most commonly used benchmark, appearing in 38 of the 62 reviewed publications.It uses LSTM RNNs to predict upcoming events and flags observed events with low predicted probabilities.
  • Conventional benchmarks: Other benchmarks use PCA, invariant mining, clustering, support vector machines, logistic regression, decision trees, and isolation forests on event-count or sequence representations.
  • Deep-learning benchmarks: LogAnomaly and LogRobust extend sequence-based benchmarking with semantic vectors, with LogRobust specifically addressing unknown events arising during software evolution.
  • Benchmark approaches: Only eight of the 62 reviewed publications do not involve any benchmark approach for comparison.
  • Reproducibility: Only 8 publications provided original source code, although re-implementations were available in the deep-loglizer toolbox.The survey treats public data and original source code as requirements for reproducibility.

V. DISCUSSION

The survey finds that deep-learning log anomaly detection is dominated by sequential representations and RNN-based models, while evaluation practices and anomaly coverage remain narrow. It identifies simple baselines, limited dataset diversity, sequence-level evaluation, class imbalance, and low explainability as important open issues.

  • Challenges: Data instability is addressed mainly by semantic vectors, while imbalanced data is handled with sampling and context-aware embeddings.Semantic representations allow previously unknown or changed events to be compared with known events through similarity measures.
  • Architectures: RNNs are the most commonly applied models because they capture sequential log patterns; CNNs offer an efficient alternative, while autoencoders and transformers support unsupervised learning.GANs, MLPs, GNNs, and EGNNs appear less often, although the survey identifies properties that make them worth considering.
  • Pre-processing: Tokenization, parsing, and parameter extraction are the three main feature-extraction strategies, with deep-learning approaches generally favoring semantic vectors and sequential features.Event counts, one-hot encodings, and embedding layers are used less often or as contributing features.
  • Detection scope: Most reviewed methods detect sequential anomalies, whereas event counts, isolated log-line outliers, event parameters, and inter-arrival times receive comparatively little attention.This narrow focus leaves diverse anomaly artifacts only partially addressed.
  • Evaluation: HDFS, BGL, Thunderbird, and OpenStack dominate evaluations, and simple heuristics achieve F1 = 90.41% and ACC = 99.48% on HDFS.The survey argues that competitive simple baselines and the limited dataset variety make broader benchmarking necessary.
  • Evaluation: Evaluations usually label whole sequences rather than individual anomalous elements, rely heavily on F-score under class imbalance, and mostly use offline training.Only 8 of 62 reviewed approaches support incremental learning, compared with 54 that support only offline training.

VI. CONCLUSION

The survey covers 62 deep-learning approaches for detecting anomalous events or processes in system logs. It finds that diverse architectures and feature types are suitable for this task.

  • The survey examines 62 scientific approaches for detecting anomalous events or processes in system log data using deep learning.
  • Suitable architectures include recurrent and convolutional neural networks, transformers, Autoencoders, and generative adversarial networks.
  • Training and detection features include event or token sequences and counts, parameter values, and event-derived statistics.
Loading 2207.03820v2…