PaperlayerView exact revision on arXiv

Source-linked AI summary

Log-based Anomaly Detection with Deep Learning: How Far Are We?

Van-Hoang Le, Hongyu Zhang

arXiv:2202.04301v2cs.SEcs.LG

TL;DR

Log-based anomaly detection models often report very high accuracy, but comprehensive evidence about their reliability across evaluation settings remains limited. The paper systematically evaluates five deep learning models on four datasets across training-data selection, grouping, dataset characteristics, and early detection, finding that evaluation choices substantially affect results and the problem remains unsolved.

  • Problem

    High reported accuracy has not established how well log-based anomaly detection models perform across different evaluation settings.

  • Method

    The paper systematically evaluates five representative deep learning models on four datasets across training-data selection, log grouping, dataset characteristics, and early anomaly detection.

  • Results

    Evaluation factors substantially affect model performance, and the studied models do not always generalize well across experimental settings.

  • Takeaways & Limitations

    The problem of log-based anomaly detection has not been solved yet, motivating further research into model evaluation and improved detection methods.

  • Takeaways & Limitations

    The four public datasets may not represent all characteristics of log data or cover all real-world scenarios.

Abstract

from arXiv · show

Software-intensive systems produce logs for troubleshooting purposes. Recently, many deep learning models have been proposed to automatically detect system anomalies based on log data. These models typically claim very high detection accuracy. For example, most models report an F-measure greater than 0.9 on the commonly-used HDFS dataset. To achieve a profound understanding of how far we are from solving the problem of log-based anomaly detection, in this paper, we conduct an in-depth analysis of five state-of-the-art deep learning-based models for detecting system anomalies on four public log datasets. Our experiments focus on several aspects of model evaluation, including training data selection, data grouping, class distribution, data noise, and early detection ability. Our results point out that all these aspects have significant impact on the evaluation, and that all the studied models do not always work well. The problem of log-based anomaly detection has not been solved yet. Based on our findings, we also suggest possible future work.

1 INTRODUCTION

Log-based anomaly detection addresses the operational risks of large software-intensive systems, but existing deep learning evaluations may overlook important experimental factors. This paper evaluates five representative models across four datasets and examines how evaluation settings affect their conclusions.

  • Motivation: Large software-intensive systems generate substantial operational risk because small problems can cause user dissatisfaction and financial loss.Anomaly detection supports quality assurance for these systems.
  • Research questions: Existing work overlooks important aspects of experimental datasets, evaluation metrics, and experimental settings.The paper asks whether deep learning methods are as effective as claimed and which factors affect performance.
  • Approach: The study evaluates five representative deep learning models on four public datasets under controlled experimental settings.It examines training-data selection and grouping, dataset noise and class distribution, and early anomaly detection.
  • Findings: Training-data selection and log-data grouping significantly influence model performance, while shorter log sequences reduce accuracy.These findings make evaluation design a central concern for anomaly-detection studies.
  • Findings: Highly imbalanced classes and small amounts of data noise substantially affect effectiveness, and Precision, Recall, and F-measure are not comprehensive enough for imbalanced data.The study also reports that existing models do not generalize well across experimental settings.

2 LOG-BASED ANOMALY DETECTION WITH DEEP LEARNING

Deep learning anomaly detection commonly transforms raw logs through parsing, grouping, representation, and model-based detection. The reviewed models differ in how they represent log sequences and whether they use forecasting or classification.

  • Detection models: DeepLog and LogAnomaly forecast the next log event, whereas PLELog, LogRobust, and CNN use classification-based detection.The reviewed literature includes supervised, semi-supervised, and unsupervised approaches.
  • Common workflow: The common workflow has four steps: log parsing, log grouping, log representation, and anomaly detection through deep learning models.These stages convert raw logs into features used by detection models.
  • Log parsing: Log parsing converts each log message into an event template containing constant text and associated variable parameters.For example, a parameterized message can be represented by a template such as “Verification succeeded for *”.
  • Log grouping: Log grouping separates logs into finite chunks called log sequences, using mechanisms such as fixed, sliding, or session windows.The resulting sequences provide the units from which features are extracted.
  • Log representation: DL models use sequential, quantitative, or semantic vectors to represent log events and their context.DeepLog uses sequential vectors, LogAnomaly combines sequential and quantitative vectors, and LogRobust uses semantic vectors.

3.1 Motivation

The paper motivates re-evaluation of deep learning anomaly detection because reported accuracy can depend on data selection, grouping, class imbalance, noise, and detection timing. These factors create practical concerns for real-world and online use.

  • Motivation: Reported high accuracy on common datasets, including HDFS F-measure values above 90%, motivates testing whether deep learning has nearly solved anomaly detection.The paper evaluates actual effectiveness rather than relying on prior reported results.
  • Training data selection: Randomly selecting training sequences can expose models to future data, causing leakage and making evaluation unsuitable for historical-log scenarios.Chronological selection better reflects the setting in which only past logs are available for training.
  • Log grouping: Window size creates a trade-off: small windows can miss anomalies spanning sequences, while large windows can combine multiple anomalies and confuse detection.The study evaluates windows from 20 to 200 messages, plus half-hour, one-hour, and session windows.
  • Class distribution: Anomalous sequences may comprise only 0.5% to 15% of a dataset, creating highly imbalanced classes that challenge detection.The paper therefore questions whether commonly used evaluation metrics adequately reflect effectiveness.
  • Data quality: Small amounts of labeling and log-parsing noise can downgrade anomaly-detection performance.Public datasets are manually inspected and labeled, so false positives, false negatives, and parsing errors are possible.
  • Early detection: Online detection requires identifying early anomaly signals while maintaining high detection accuracy so mitigation actions can begin sooner.This motivates evaluating early-detection capability.

3.2 Evaluated Models

The study evaluates five representative deep learning models whose implementations are publicly available and whose original results can be reproduced. The models span forecasting-based and classification-based detection designs.

  • Evaluated models: The evaluated models are DeepLog, LogAnomaly, PLELog, LogRobust, and CNN.Their publicly available implementations enabled reproduction of results reported in the original papers.

3.3 Datasets

The experiments use four public log datasets—HDFS, BGL, Spirit, and Thunderbird—with differing scales and anomaly proportions. Table 1 summarizes their dataset statistics.

  • Four public datasets—HDFS, BGL, Thunderbird, and Spirit—are used for the experiments.
  • HDFS contains 11,175,629 log messages from more than 200 Amazon EC2 nodes, organized into 16,838 block-based log windows.Anomalous blocks account for 2.93% of the dataset.
  • BGL contains 4,747,963 manually labelled log messages, including 348,460 anomalies.The anomalous proportion is 7.34%.
  • Spirit contributes the first 5 million log lines of its original dataset, including 764,500 abnormal messages.This subset has an anomaly proportion of 15.29%.
  • The Thunderbird data includes 4,934 abnormal log messages, representing 0.49% of the log lines.

3.4 Research Questions

The study examines how training selection, grouping, class distribution, noise, and detection timing affect deep-learning anomaly detection. Its research questions use controlled comparisons across datasets and data conditions.

  • RQ1: Training Data Selection: RQ1 compares random and chronological training-data selection to assess whether models maintain accuracy under different strategies.Chronological training uses earlier logs while reserving later logs for testing.
  • RQ2: Data Grouping: RQ2 evaluates fixed windows of different sizes, message-count windows, and session-window grouping to measure grouping effects.
  • RQ3: Class Distribution: RQ3 tests model performance under systematically varied imbalanced class distributions created by removing normal or abnormal sequences.The subject datasets can have anomaly ratios as low as 0.1%.
  • RQ4: Data Noise: RQ4 measures robustness to mislabeled logs and parsing noise by injecting label errors and comparing four log parsers.Synthetic mislabeled proportions range from 1% to 10%.
  • RQ5: Early Detection: RQ5 evaluates online early-detection ability by recording how many log messages are examined before each model raises an anomaly.

3.5 Experimental Setup

The experimental pipeline parses logs, groups them into labeled sequences, converts sequences into model-specific vectors, and trains representative deep-learning models. Experiments are repeated across the specified datasets and settings.

  • The setup begins by preprocessing log data before applying deep-learning-based anomaly detection.
  • Log Parsing: Drain parses each log message into a log template, producing a sequence of parsed events for later processing.
  • Log Grouping: Logs are grouped into sessions for HDFS and fixed windows for BGL, Spirit, and Thunderbird, with sequence sizes varied by research question.Grouping sizes include 20, 100, and 200 messages, plus 0.5- and 1-hour windows.
  • Log Representation: The models use different numerical representations, including sequential vectors, quantitative event-count vectors, and semantic template vectors.
  • Deep Learning Models: DeepLog and LogAnomaly use two-layer LSTMs, LogRobust uses an attention-based Bi-LSTM, PLELog uses a GRU, and CNN uses convolutional layers with max pooling.
  • Experiments are repeated five times and averaged to reduce randomness-related bias.They run on a Windows Server 2012 R2 system with an NVIDIA Tesla K40c GPU.

3.6 Evaluation Metrics

Model effectiveness is evaluated with Precision, Recall, Specificity, and F-Measure. These metrics distinguish correctly detected anomalies, missed anomalies, false alarms, and correctly identified normal sequences.

  • Precision measures the percentage of detected abnormal sequences that are correctly abnormal.It is defined as TP/(TP+FP).
  • Recall measures the percentage of real anomalies that the model correctly identifies.It is defined as TP/(TP+FN).
  • Specificity measures the percentage of real normal sequences that the model correctly identifies as normal.It is defined as TN/(TN+FP).
  • F-Measure is the harmonic mean of Precision and Recall.
  • TP counts correctly detected abnormal sequences, FP counts normal sequences wrongly flagged as anomalous, and FN counts missed abnormal sequences.

4 RESULTS AND FINDINGS

Evaluation outcomes vary substantially with training-data selection, grouping, class distribution, noise, and model design. These findings show that strong reported accuracy does not consistently generalize across realistic settings.

  • Training data selection: 0.927 versus 0.426: DeepLog’s BGL F-measure falls when training changes from random to chronological selection.Random selection exposes semi-supervised models to future log events, causing data leakage and overly high accuracy.
  • Data grouping: Different fixed-window sizes produce unstable results, while session grouping improves BGL performance and yields above-0.9 F-measures on HDFS.On BGL, the 100-log setting reduces F1 by 4.84% to 50.31% versus one-hour logs; session grouping captures execution-path relations.
  • Class distribution: Highly imbalanced data impairs detection, and Precision, Recall, and F-measure can misrepresent performance without Specificity.For LogRobust on BGL, increasing the anomaly percentage improves Precision by 63.8%, Recall by 10.1%, Specificity by 2.6%, and F-measure by 38.4%.
  • Data noise: 1% mislabelled logs causes LogRobust and CNN F-measures on HDFS to drop by 37.2% and 36.2%, respectively.At 10% noise, their F-measures fall to 0.161 and 0.263, while forecasting-based models are more robust to mislabelled logs.
  • Log parsing errors: 0.755 to 0.609: DeepLog’s Spirit F-measure declines when parsing changes from IPLoM to Drain, while Specificity drops from 0.545 to 0.099.Extra log events from Drain hinder forecasting-based models; semantic-vector methods handle parsing errors better.
  • Early detection: Forecasting-based models detect anomalies earlier than classification-based models, whereas PLELog may be unsuitable for online detection because of delayed detection and high time cost.The studied forecasting models are DeepLog and LogAnomaly; classification-based models include PLELog, LogRobust, and CNN.

5 DISCUSSION

The studied models do not consistently perform as claimed, and evaluation outcomes depend strongly on datasets, settings, and detection requirements. The discussion identifies model-specific trade-offs, validity threats, and research directions.

  • Findings: The five studied models do not always perform as well as reported in their original papers.Different scenarios affect anomaly-detection performance in different ways.
  • Model trade-offs: DeepLog reduces model-construction effort and supports early detection, but performs poorly on complex datasets and is sensitive to log-parsing errors.It uses only normal logs and ignores semantic meaning by relying on log-template indices.
  • Model trade-offs: LogAnomaly reduces the impact of mislabelled logs, handles large data using normal logs, and improves matching through semantic vectors.Its sequential and quantitative vectors and template matching provide these advantages.
  • Model trade-offs: PLELog can learn from historical anomalies while using only normal logs, but is time-consuming, noise-sensitive, and weak on early detection.Its clustering-based probabilistic label estimation adds training cost.
  • Model trade-offs: LogRobust and CNN can achieve high accuracy, but supervised training requires substantial labeled data that is mostly unavailable in practice.LogRobust handles parsing noise through contextual modeling, while CNN captures relationships within log context and semantic embeddings.
  • Future research: Future research should improve evaluation across datasets, handle limited labels, support early detection, and model evolving logs and richer event relations.The proposed directions include more datasets, better semi-supervised or unsupervised methods, sufficient lead time, semantic modeling, and logical or interactive relationships.
  • Threats and future work: Evaluation is threatened by limited models, datasets, data quality, and implementation changes, motivating broader and more controlled studies.The authors used public implementations, four datasets, synthetic data, and injected mislabelled logs to address some threats.

6 CONCLUSION

The paper analyzes deep learning models for log-based anomaly detection across evaluation dimensions and finds that results are often worse than expected. Its released code and data provide resources for future benchmarking.

  • Conclusion: Evaluation factors including training-data selection, dataset characteristics, and early-detection capability strongly affect results, while model performance is often worse than expected.The conclusion states that log-based anomaly detection has not yet been solved.
  • Conclusion: The source code, experimental data, and synthetic datasets are available as resources for evaluating future log-based anomaly-detection models.The datasets can serve as a benchmark for future work.
Loading 2202.04301v2…