PaperlayerView exact revision on arXiv

Source-linked AI summary

Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection

Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, Michael R. Lyu

arXiv:2107.05908v2cs.SEcs.LG

TL;DR

Large-scale systems produce log volumes that make manual and traditional ML-based anomaly detection impractical, while rigorous comparisons of neural detectors and reusable implementations are limited. The paper reviews and implements six neural-network-based methods, evaluates them on HDFS and BGL, and provides a toolkit; it concludes that the study can guide future research and industrial applications.

  • Problem

    Massive and complex log volumes make conventional anomaly detection impractical, while rigorous comparisons and reusable implementations of neural-network-based detectors are lacking.

  • Method

    The paper reviews five neural networks used by six log anomaly detectors, evaluates four unsupervised and two supervised methods, and releases an open-source toolkit.

  • Results

    The methods are systematically evaluated on HDFS and BGL, containing nearly 16 million log messages and 0.4 million anomaly instances, using accuracy, robustness, and efficiency.

  • Takeaways & Limitations

    The study aims to help researchers and practitioners understand detector characteristics, reduce re-implementation effort, and support industrial applications and future research.

  • Takeaways & Limitations

    Real-world deployment remains constrained by insufficient anomaly labels and privacy policies that restrict access to detailed customer logs.

Abstract

from arXiv · show

Logs have been an imperative resource to ensure the reliability and continuity of many software systems, especially large-scale distributed systems. They faithfully record runtime information to facilitate system troubleshooting and behavior understanding. Due to the large scale and complexity of modern software systems, the volume of logs has reached an unprecedented level. Consequently, for log-based anomaly detection, conventional manual inspection methods or even traditional machine learning-based methods become impractical, which serve as a catalyst for the rapid development of deep learning-based solutions. However, there is currently a lack of rigorous comparison among the representative log-based anomaly detectors that resort to neural networks. Moreover, the re-implementation process demands non-trivial efforts, and bias can be easily introduced. To better understand the characteristics of different anomaly detectors, in this paper, we provide a comprehensive review and evaluation of five popular neural networks used by six state-of-the-art methods. Particularly, four of the selected methods are unsupervised, and the remaining two are supervised. These methods are evaluated with two publicly available log datasets, which contain nearly 16 million log messages and 0.4 million anomaly instances in total. We believe our work can serve as a basis in this field and contribute to future academic research and industrial applications.

1 INTRODUCTION

Modern software systems generate massive, complex log volumes, making manual and traditional ML-based anomaly detection impractical. This paper reviews, implements, and systematically evaluates six neural-network-based detectors to support research and industrial use.

  • Motivation: Large-scale services depend on logs for monitoring, troubleshooting, and detecting abnormal system behavior.Logs record runtime status and help ensure system reliability and service quality.
  • Motivation: Manual keyword searches and rule matching become labor-intensive and error-prone as log volume, variety, and velocity increase.Traditional ML methods add automation but retain practical deployment limitations.
  • Limitations: Traditional methods often provide insufficient interpretability and weak adaptability to previously unseen log events.These limitations hinder fault localization and deployment in evolving systems.
  • Study scope: The study reviews five representative neural networks used by six state-of-the-art log anomaly detectors, covering four unsupervised and two supervised methods.The selected methods include LSTMs, Transformer, Autoencoder, CNN, and attentional BiLSTM.
  • Evaluation: All methods are evaluated on HDFS and BGL, which together contain nearly 16 million log messages and 0.4 million anomaly instances.Evaluation considers accuracy, robustness, and efficiency.
  • Contributions: The paper releases an open-source toolkit and benchmarks the selected models against traditional machine-learning counterparts.The toolkit is intended to reduce re-implementation effort and support reuse.

2 LOG ANOMALY DETECTION OVERVIEW

Log-based anomaly detection transforms collected semi-structured logs into structured, numerical sequences before identifying abnormal instances. The overview covers parsing, partitioning, feature extraction, and sequence-level or event-level detection.

  • Framework: The overall framework comprises log collection, log parsing, feature extraction, and anomaly detection.These phases connect raw runtime records to downstream anomaly decisions.
  • Log parsing: Log parsing separates each raw log line into a constant log event and variable parameter values.For example, changing block identifiers, sizes, and addresses can be replaced with <*> tokens.
  • Log partitioning: Timestamp- or identifier-based partitioning groups logs into sequences for downstream analysis.Fixed windows do not overlap, sliding windows may overlap, and identifier-based sequences can vary in length.
  • Feature extraction: Deep-learning methods commonly consume log-event sequences represented by event indices or semantic embedding vectors.Semantic features are formed by embedding words in log events and aggregating the resulting vectors.
  • Anomaly detection: Traditional detectors often classify an entire sequence from event counts, whereas deep-learning methods can assess normality for individual log events.Deep-learning approaches first learn normal log patterns before detecting deviations.

3 LOG ANOMALY DETECTION

Neural log anomaly detectors differ in architecture, loss function, supervision, and use of log semantics. The paper organizes them around forecasting, reconstruction, and supervised classification objectives.

  • Method taxonomy: The study examines six methods using four unsupervised and two supervised neural-network-based detectors.The methods vary in model architecture, loss, and whether they use semantic information.
  • Forecasting loss: Forecasting loss learns normal sequential patterns by predicting the next log event from preceding observations.Anomalies are detected when observed sequences violate learned normal patterns.
  • Reconstruction loss: Reconstruction loss trains an autoencoder to reproduce normal input windows, with anomalous samples expected to produce larger reconstruction loss.The model learns to reconstruct normal sequences before measuring abnormality.
  • Supervised loss: Supervised loss uses anomaly labels to learn features that distinguish abnormal samples from normal ones.Common supervised losses include cross-entropy and mean squared error.
  • Unsupervised methods: DeepLog forecasts log events from sequential event indices, while LogAnomaly adds template semantics through template2Vec representations.Logsy uses a Transformer and can operate on event indices or semantic sequences.
  • Supervised methods: LogRobust uses attention-weighted BiLSTM representations to classify anomalies under changing log conditions.Its attention mechanism assigns different weights to log events before softmax classification.

3.3 Tool Implementation

The authors implemented the six studied detectors as a reusable Python toolkit with unified interfaces and configurable model settings. PyTorch supplies the deep-learning implementation components.

  • Implementation: The toolkit implements six anomaly detection methods in Python using around 3,000 lines of code.The methods are packaged for reuse rather than requiring separate re-implementations.
  • Implementation: Standardized input/output interfaces make the studied methods easier to use consistently.The toolkit is designed as a common package for the evaluated detectors.
  • Configuration: Users can configure model settings such as loss functions and whether to use log semantic information.This exposes key design choices across the studied methods.
  • Implementation: PyTorch provides the basic building blocks for implementing the deep-learning models.The framework supports the toolkit’s neural-network implementations.

4 EVALUATION

The evaluation benchmarks six deep-learning log anomaly detectors on accuracy, robustness to unknown log events, and efficiency. These criteria reflect the practical requirements of distinguishing anomalies, handling evolving logs, and processing massive data volumes.

  • Evaluation Criteria: Six DL-based detectors are evaluated on accuracy, robustness, and efficiency, the key qualities considered for industrial deployment.The study measures anomaly discrimination, detection with unknown log events, and model speed.
  • Evaluation Criteria: Accuracy measures how well a method distinguishes anomalous logs from normal ones.Both false positives and false negatives matter because they respectively waste engineering effort or risk missing critical failures.
  • Evaluation Criteria: Robustness measures whether a detector can identify anomalies when unknown log events are present.This issue is increasingly important as modern software systems evolve rapidly.
  • Evaluation Criteria: Efficiency measures the time an anomaly detector takes during training and testing.The criterion addresses the stringent processing demands created by daily terabytes or petabytes of generated data.

4.1 Experiment Design

The experiment uses two public log datasets, standard accuracy metrics, controlled training and testing procedures, and a common hardware environment. Dataset partitioning differs by source, while unsupervised methods are trained without anomalies.

  • Datasets: HDFS and BGL provide the evaluation data, with HDFS containing 11,175,629 messages and BGL containing 4,747,963 messages.HDFS uses identifier-based partitioning, whereas BGL uses timestamp-based partitioning because it lacks job-execution identifiers.
  • Evaluation Metrics: Precision, recall, and F1 score measure accuracy for the binary log-anomaly classification task.Precision counts correctly identified anomalous sequences among predicted anomalies, while recall counts detected anomalies among all actual anomalies.
  • Experiment Setup: Experiments use four NVIDIA Titan V GPUs, 20 Xeon Gold 6148 CPUs, and 256GB of RAM, with each method run five times.Parameters are fine-tuned for best results, and the reported result is the best run.
  • Experiment Setup: Chronologically partitioned logs are shuffled at the sequence level, with the first 80% used for training and the remaining 20% for testing.Input windows are not shuffled; unsupervised training data exclude anomalies because these methods learn normal log patterns.

4.2 Accuracy of Log Anomaly Detection

Supervised detectors generally outperform unsupervised ones, while log semantics particularly benefit unsupervised methods. Training anomalies hurt forecasting-based methods more than reconstruction-based methods, and traditional methods remain competitive on HDFS.

  • Accuracy Without Log Semantics: Supervised methods reach F1 scores of 0.97 on HDFS and 0.983 on BGL, exceeding the best unsupervised scores of 0.944 and 0.961.CNN achieves the HDFS supervised result, attentional BiLSTM achieves the BGL result, and LSTM provides the best unsupervised results.
  • Comparison with Traditional ML: Decision Tree achieves 0.998 on HDFS, ranking best overall there, while other traditional methods are generally outperformed by DL-based counterparts.The comparison includes traditional unsupervised and supervised methods across HDFS and BGL.
  • Comparison with Traditional ML: Shuffling log sequences improves BGL results for both DL-based and traditional ML-based methods by alleviating unseen logs in testing data.The order of log events within each input window remains preserved.
  • Accuracy with Log Semantics: Log semantics improve almost all methods, with Autoencoder gaining nearly 15% and BGL best F1 scores reaching 0.967 for LSTM and 0.989 for CNN.HDFS best F1 scores remain almost unchanged, and Decision Tree remains undefeated on HDFS.
  • Varying Anomaly Ratio: Anomalies in training data quickly deteriorate forecasting-based methods, whereas reconstruction-based methods are more resistant.This experiment tests the effect of anomalous training instances on unsupervised deep-learning methods.

4.3 Robustness of Log Anomaly Detection

Unprecedented logs substantially reduce anomaly-detection performance, with unsupervised methods more vulnerable than supervised methods. Logs’ semantic information can improve robustness, particularly for supervised models.

  • 20% noisy logs leave supervised DL models near 0.9 F1, while LSTM and Transformer fall below 0.5.With semantic information, supervised models maintain this performance under injected noise.
  • 5% noisy logs reduce LSTM and Transformer F1 scores by more than 20%.Unsupervised methods are substantially more vulnerable to injected noise than supervised methods.
  • Unprecedented logs significantly affect anomaly detection, while supervised methods and log semantics improve robustness.

4.4 Efficiency of Log Anomaly Detection

The evaluation measures training and testing time on HDFS and BGL. Deep learning methods often take longer than traditional machine-learning methods, although testing is generally fast and some ML methods are real-time capable.

  • Most models train in tens of seconds and test in around five seconds, with BGL requiring less time because it is smaller.
  • LSTM and Autoencoder are the most time-consuming models to train on HDFS, whereas supervised models require more training time on BGL.
  • Logistic Regression, Decision Tree, SVM, and PCA train faster than DL models, while SVM and PCA can operate in real time.
  • Invariant Mining requires thousands of seconds for pattern mining on HDFS, while most methods test in tens of milliseconds.
  • Compared with traditional ML methods, DL-based methods often require more time for training and testing.

5 INDUSTRIAL PRACTICES

The paper reports a 14-month Huawei Cloud deployment of automated log anomaly detection and identifies practical challenges that limit reliability and interpretability in complex production environments.

  • Deployment Architecture: The production system uses offline training and online serving, with Kafka streaming logs from separate services to the anomaly-detection model.The deployment architecture separates model training from online analysis and assigns each service a Kafka topic.
  • Deployment Outcomes: After 14 months, engineers confirmed more than 85% of predicted anomalies for one large-scale service system.The result is based on more than one year of predicted anomalies retrieved from the selected service.
  • Challenges: Production logs create high complexity through interleaved events and overwhelming numbers of log events.These characteristics distinguish production logs from the HDFS and BGL datasets used in evaluation.
  • Challenges: Offline-trained thresholds may require repeated adjustment because their optimal settings do not always transfer to online deployment.The paper identifies threshold re-determination as an operational requirement after deployment.
  • Challenges: Concept drift from new log events and changing patterns requires frequent retraining when online learning is unavailable.The paper links this challenge to continuous feature upgrades in modern software systems.
  • Challenges: Inconsistent logging styles, meaningless tokens, and limited system-level explanations constrain parsing and interpretability.The authors also identify an incorrect model strategy, because many anomalies arise from error logs rather than event-order violations.
  • Challenges: Ambiguous anomaly labels and privacy restrictions can prevent access to detailed logs and make prompt detection more difficult.The paper presents labeling and privacy as additional production challenges.

6 RELATED WORK

Related work spans log parsing, traditional and deep-learning-based anomaly detection, and empirical studies of logging practices and industrial use.

  • Log Analysis: Log analysis commonly combines log parsing with log mining, supporting tasks such as anomaly detection, failure prediction, and failure diagnosis.The paper distinguishes streaming from non-streaming parsing and describes log mining as a second major step.
  • Anomaly Detection: Traditional log-mining methods include PCA, invariant mining, and clustering, while deep-learning methods increasingly use neural networks for anomaly detection.The paper categorizes anomaly-detection methods into traditional ML-based and DL-based approaches.
  • Deep Learning Methods: DeepLog introduced unsupervised LSTM-based detection, followed by semantic, robustness, CNN, and other neural-network extensions.The cited methods address semantic information, unseen logs, insufficient labels, and alternative neural architectures.
  • Empirical Studies: Prior empirical studies examined logging practices, industrial use, developer behavior, and the lifecycle of log analysis.The paper positions its own study against a benchmark of traditional methods and emphasizes practical issues such as unseen logs and anomalies in training data.

7 CONCLUSION

The paper addresses the difficulty of applying log anomaly detection to modern systems by reviewing and evaluating deep-learning methods against practical deployment needs. It concludes that semantic information improves robustness to noise and releases a toolkit to support further use and development.

  • Conclusion: Modern systems produce unprecedented log volumes that overwhelm existing statistical and traditional machine-learning approaches.This motivates the development and evaluation of deep-learning-based anomaly detectors.
  • Conclusion: The study reviews five neural networks and evaluates six state-of-the-art methods for accuracy, robustness, and efficiency.It also examines semantic information, log instability, and comparisons with traditional ML-based methods.
  • Conclusion: Logs’ semantics improve model robustness against noise in both training and testing data.The paper presents this as a principal result of its comparative evaluation.
  • Conclusion: The authors release an open-source toolkit intended to support model customization and improvement in academic and industrial settings.The toolkit is presented as a way to help address the gap between research methods and practical use.
Loading 2107.05908v2…