Source-linked AI summary
LogBERT: Log Anomaly Detection via BERT
Haixuan Guo, Shuhan Yuan, Xintao Wu
TL;DR
Online systems need timely detection of anomalous events, and existing sequence models have difficulty using complete context and explicitly encoding shared normal patterns. LogBERT applies BERT with masked log key prediction and hypersphere minimization, and outperforms state-of-the-art approaches on three log datasets.
Problem
Timely log anomaly detection matters for protecting online computer systems, while RNN-based objectives do not explicitly encode patterns shared by all normal sequences.
Method
LogBERT uses BERT with masked log key prediction and volume of hypersphere minimization to learn normal log-sequence patterns from normal sequences.
Results
LogBERT outperforms state-of-the-art approaches for log anomaly detection on three log datasets.
Takeaways & Limitations
The framework detects anomalous log sequences after learning normal-sequence patterns with two self-supervised training tasks.
Takeaways & Limitations
The hyper-parameters g and r are tuned using a validation set.
Abstract
from arXiv · showhide
Detecting anomalous events in online computer systems is crucial to protect the systems from malicious attacks or malfunctions. System logs, which record detailed information of computational events, are widely used for system status analysis. In this paper, we propose LogBERT, a self-supervised framework for log anomaly detection based on Bidirectional Encoder Representations from Transformers (BERT). LogBERT learns the patterns of normal log sequences by two novel self-supervised training tasks and is able to detect anomalies where the underlying patterns deviate from normal log sequences. The experimental results on three log datasets show that LogBERT outperforms state-of-the-art approaches for anomaly detection.
1 Introduction
LogBERT addresses limitations in log anomaly detection by using BERT to model complete sequence context and two self-supervised tasks to learn normal patterns. It reports the best performance among state-of-the-art baselines on three log datasets.
- 1 Introduction: LogBERT uses BERT to capture information from whole log sequences rather than only preceding events.This targets the limitation that recurrent models cannot encode both left and right context for each log.
- 1 Introduction: Masked log key prediction learns to reconstruct randomly masked log keys in normal sequences.The task is designed to capture normal-sequence patterns through self-supervised training.
- 1 Introduction: Volume of hypersphere minimization brings normal log sequences close together in the embedding space.This provides a spherical objective for regulating the distribution of normal sequences.
- 1 Introduction: LogBERT derives an anomaly criterion after training on normal sequences and detects sequences whose patterns deviate from normal behavior.The framework is motivated by the difficulty of training binary classifiers under imbalanced log data.
- 1 Introduction: LogBERT achieves the best log anomaly-detection performance compared with various state-of-the-art baselines on three datasets.
2 Related Work
Related work progresses from keyword and rule-based detection to feature-based unsupervised methods and deep models that capture sequential log patterns. LogBERT extends this trajectory by exploring BERT for log-sequence information.
- 2 Related Work: Traditional approaches use keywords or regular expressions, but may miss attacks whose individual log entries appear normal while their operation sequence is anomalous.
- 2 Related Work: Rule-based methods were proposed to identify anomalous events in response to sequence-level detection challenges.
- 2 Related Work: A typical learning-based pipeline parses messages into log keys, extracts sequence features, and applies an unsupervised detector.
- 2 Related Work: Deep learning approaches mainly use LSTM or GRU models to represent normal log-key sequences and derive anomaly scores.
- 2 Related Work: LogBERT explores BERT to capture information from log sequences and introduces two self-supervised training tasks.
3 LogBERT
LogBERT models normal log sequences with a Transformer encoder and two self-supervised tasks, then detects anomalies using prediction behavior and sequence representations.
- Framework: LogBERT uses a Transformer encoder to model log sequences and capture contextual relations among log keys.Each self-attention lets every log key attend to all keys in the input sequence.
- Framework: The input represents each log key as the sum of a learned log-key embedding and a sinusoidal position embedding.A special DIST token is added at the beginning to represent the whole sequence.
- Self-supervised tasks: Masked Log Key Prediction trains LogBERT to recover randomly masked log keys, encouraging it to encode bidirectional context from normal sequences.Masked-token representations are passed through a softmax over the log-key set and optimized with cross-entropy loss.
- Self-supervised tasks: Volume of Hypersphere Minimization concentrates normal sequence representations around a center in the embedding space.The DIST representation is averaged to obtain the center, and training minimizes the distance of normal sequences to it.
- Training and detection: The total training objective combines masked-key prediction and hypersphere minimization, with α balancing the two tasks.After training, anomaly scores are derived from masked-key prediction results; thresholds g and r are tuned on a validation set.
4 Experiments
LogBERT is evaluated against traditional and deep-learning baselines on HDFS, BGL, and Thunderbird, with ablations, visualization, and parameter analyses examining its behavior.
- Experimental Setup: The evaluation uses HDFS, BGL, and Thunderbird, with around 5000 normal log sequences from each dataset used for training.HDFS sequences average 19 logs, BGL 562, and Thunderbird 326.
- Experimental Setup: Baselines include PCA, OCSVM, Isolation Forest, LogCluster, DeepLog, and LogAnomaly, alongside LogBERT.Drain parses log messages into log keys before evaluation.
- Experimental Results: LogBERT achieves the highest F1 scores on all three datasets, with large margins over the baselines.DeepLog and LogAnomaly outperform the traditional approaches, while PCA, Isolation Forest, and OCSVM have poor balanced performance.
- Ablation Studies: Using both self-supervised tasks improves performance, especially for relatively short log sequences.The improvement is less significant on BGL and Thunderbird, whose sequences are much longer than HDFS sequences.
- Visualization: The VHM task separates normal and anomalous HDFS sequences in latent space, whereas sequences remain mixed without it.Normal sequences group together when VHM is incorporated.
- Parameter Analysis: Increasing the mask ratio from 0.1 to 0.5 slightly increases F1, but higher ratios worsen performance; larger candidate sets raise precision and reduce recall.The candidate-set trend requires balancing precision and recall.
5 Conclusion
LogBERT is trained on normal log sequences using two self-supervised tasks and then detects anomalous sequences. On three log datasets, it outperformed state-of-the-art approaches.
- LogBERT is a BERT-based log anomaly detection model trained only on normal log sequences.The model learns normal-sequence patterns without requiring anomalous training examples.
- The first self-supervised task predicts masked log keys in log sequences.
- The second self-supervised task brings normal log sequences closer together in embedding space.
- After training on normal log sequences, LogBERT detects anomalous log sequences.
- Experimental results on three log datasets show that LogBERT outperforms state-of-the-art approaches for log anomaly detection.