PaperlayerView exact revision on arXiv

Source-linked AI summary

LAnoBERT: System Log Anomaly Detection based on BERT Masked Language Model

Yukyung Lee, Jina Kim, Pilsung Kang

arXiv:2111.09564v3cs.LGcs.CL

TL;DR

System log anomaly detection must identify rare anomalies promptly while minimizing human intervention, but parser-based approaches can lose log information. LAnoBERT uses parser-free BERT masked language modeling with efficient inference, achieving the best unsupervised performance and comparable or superior performance to supervised models across three benchmark datasets.

  • Problem

    System log anomaly detection requires prompt, low-intervention identification of anomalies, while parser-based methods may lose information through predefined templates.

  • Method

    LAnoBERT uses parser-free BERT masked language modeling for unsupervised anomaly detection and an efficient log-dictionary-based inference process.

  • Results

    LAnoBERT achieved the best performance among unsupervised models and superior or similar performance to supervised models on HDFS, BGL, and Thunderbird.

  • Takeaways & Limitations

    LAnoBERT provides a parser-independent anomaly detection model with an efficient inference process for practical system-log monitoring.

  • Takeaways & Limitations

    Benchmark comparisons were constrained because AUROC could not be calculated for some models lacking official implementation code.

Abstract

from arXiv · show

The system log generated in a computer system refers to large-scale data that are collected simultaneously and used as the basic data for determining errors, intrusion and abnormal behaviors. The aim of system log anomaly detection is to promptly identify anomalies while minimizing human intervention, which is a critical problem in the industry. Previous studies performed anomaly detection through algorithms after converting various forms of log data into a standardized template using a parser. Particularly, a template corresponding to a specific event should be defined in advance for all the log data using which the information within the log key may get lost. In this study, we propose LAnoBERT, a parser free system log anomaly detection method that uses the BERT model, exhibiting excellent natural language processing performance. The proposed method, LAnoBERT, learns the model through masked language modeling, which is a BERT-based pre-training method, and proceeds with unsupervised learning-based anomaly detection using the masked language modeling loss function per log key during the test process. In addition, we also propose an efficient inference process to establish a practically applicable pipeline to the actual system. Experiments on three well-known log datasets, i.e., HDFS, BGL, and Thunderbird, show that not only did LAnoBERT yield a higher anomaly detection performance compared to unsupervised learning-based benchmark models, but also it resulted in a comparable performance with supervised learning-based benchmark models.

1. Introduction

System log anomaly detection seeks prompt, low-intervention monitoring, but existing approaches face parser dependence, information loss, long-sequence limitations, and unrealistic abnormal-data requirements. LAnoBERT addresses these issues with parser-free BERT-based unsupervised detection and an efficient inference process, achieving strong benchmark performance.

  • Motivation: Real-time operator monitoring is slow and vulnerable to human error or bias, while rule-based methods still require human intervention.Machine-learning-based monitoring is pursued to minimize human involvement.
  • Motivation: Parser-based preprocessing requires expert-defined templates, can lose crucial information, and makes performance depend on parser compatibility.These limitations motivate parser-free detection.
  • Motivation: RNN-based feature embedding struggles with long sequences and long-term dependencies, motivating transformer-based architectures.The paper positions BERT as a response to this limitation.
  • Results: LAnoBERT achieved the best performance among unsupervised models and comparable or superior performance to supervised models on HDFS, BGL, and Thunderbird.The method was trained under less favorable conditions while maintaining practical detection performance.
  • Proposed method: LAnoBERT is a BERT-based, unsupervised, log parser-free anomaly detection framework.It uses simple preprocessing rather than predefined log templates.
  • Proposed method: Its inference process uses a log dictionary database to reduce BERT's computational burden and handle redundant log sequences.The design targets efficient practical deployment.

2. Related Work

Prior log anomaly detection methods use parsing-based or parser-free preprocessing, neural feature extraction, and either anomaly detection or classification. LAnoBERT avoids parser dependence while using the log sequence itself as input, addressing template-related information loss and flexibility concerns.

  • Parsing-based methods: Parsing-based methods standardize logs with parsers such as Drain, but template substitution can discard information from log messages.Parser compatibility can constrain practical extensibility.
  • Parsing-based methods: DeepLog learns normal template patterns with an unsupervised LSTM and flags sequences whose patterns fall outside a top-g candidate set.Its decisions depend on the selected candidate threshold.
  • Parsing-based methods: LogRobust and HitAnomaly use parser-generated templates with classification-based anomaly detection, while HitAnomaly combines two transformer representations through attention.These approaches retain parser dependence and classification formulations.
  • Parsing-based methods: LogBERT trains on normal logs with masked log key prediction and hypersphere minimization, but still relies on Drain for log sequence refinement.Its inference selects top-g predicted log keys as candidates.
  • Parsing-free methods: LAnoBERT avoids predefined templates by refining logs with simple preprocessing and using the log sequence itself as model input.This allows application to previously unseen log sequences.
  • Parsing-free methods: LogSy and NeuralLog are parser-free alternatives, but both formulate anomaly detection as classification-based learning.Their training setups use combinations of normal and abnormal data across systems.

3. Background

System logs are unstructured, ordered sequence data that can be standardized with parsers or modeled directly using BERT. BERT’s masked language modeling provides an unsupervised way to learn contextual representations without task-specific labels.

  • 3.1. Log parser for anomaly detection: Log parsers convert unstructured log messages into standardized templates, but model performance can become dependent on parser compatibility.In the BGL system, Drain reduced 4,747,964 messages to 376 events.
  • 3.2. BERT: BERT is a transformer encoder pretrained with masked language modeling and next sentence prediction on unlabeled data.Masked language modeling replaces selected tokens with [MASK] and predicts the missing tokens.
  • 3.3. BERT for anomaly detection: LAnoBERT applies BERT to system-log anomaly detection so log features, natural-language content, semantics, and context can be learned without a separate parser.The tokenizer is intended to preserve natural-language information that parser-based templating may discard.

4. Proposed Method

LAnoBERT is a parser-free log anomaly detector that trains BERT with masked language modeling on normal logs and scores abnormality from masked-token predictions. It also reduces inference cost by caching scores for repeated log sequences.

  • Abnormal Score: At test time, LAnoBERT masks each log key in turn and computes prediction error and predictive probability for every resulting sequence.This produces a number of test inputs equal to the log-sequence length.
  • Model Training: The model trains BERT with masked language modeling on normal logs, using 20% random masking and omitting the next-sentence prediction objective.The approach relies on normal-log context to distinguish abnormal logs during testing.
  • Preprocessing: LAnoBERT uses regular expressions instead of a log parser, replacing numbers, dates, and IPs while minimizing information loss.Preprocessed sequences are tokenized with WordPiece.
  • Abnormal Score: The final abnormal score aggregates the top k prediction-error and probability values, treating larger error or lower probability as stronger anomaly evidence.Error and probability are computed independently before aggregation.
  • Efficient Inference: Naive testing requires computations proportional to the total number of log sequences multiplied by sequence length, making direct inference inefficient for actual systems.The proposed inference process addresses this cost by removing repeated computations for duplicated sequences.
  • Efficient Inference: LAnoBERT caches abnormal-error and abnormal-probability values in a log dictionary and computes values only when an incoming key is not already stored.The dictionary uses a log sequence as a key and stores the corresponding anomaly values.

5. Experimental Setting

The experiments compare LAnoBERT with parser-based and parser-free benchmark models across HDFS, BGL, and Thunderbird using F1 score and AUROC-based evaluation criteria.

  • 5.1. Datasets: HDFS, BGL, and Thunderbird provide labeled logs from different systems for comparing anomaly-detection performance and assessing generalization.HDFS contains logs from a private cloud environment, BGL logs come from the Blue Gene/L supercomputer, and each BGL sequence has a normal or abnormal label.
  • 5.2. Benchmark Methods: The benchmark set spans supervised models, unsupervised models, parser-based methods, and parser-free transformer approaches.Compared methods include LogSy, LogRobust, HitAnomaly, PCA, iForest, OCSVM, LogCluster, DeepLog, LogAnomaly, and LogBERT.
  • 5.2. Benchmark Methods: Benchmark methods differ in preprocessing and anomaly determination, including specialized parsers, tokenizers, next-word prediction, classification, clustering, and abnormal scores.Figure 6 summarizes structural differences, preprocessing during training and testing, and anomaly-detection procedures.
  • 5.3. Evaluation Metrics: F1 score and AUROC are used because F1 depends on the threshold, whereas AUROC evaluates performance across threshold candidates.AUROC uses false-positive and true-positive rates, with values closer to 1 indicating better anomaly-detection performance.
  • 5.3. Evaluation Metrics: AUROC cannot be calculated for LogSy, LogCluster, and LogAnomaly because official implementation code was unavailable, so comparisons use available F1 results and LAnoBERT AUROC.Best F1 thresholds are selected theoretically on the test dataset because training uses only normal data.

6. Results

LAnoBERT was evaluated against supervised and unsupervised log-anomaly detectors across HDFS, BGL, and Thunderbird, including parser-based and parser-free settings. Predictive probability produced the strongest reported LAnoBERT results, while performance varied with dataset complexity and scoring method.

  • 6.1. Anomaly Detection Performance: The experiment compared supervised and unsupervised models while recording parser use because log-anomaly performance varies substantially with preprocessing.Table 3 covers LAnoBERT and ten additional models across HDFS, BGL, and Thunderbird.
  • 6.1. Anomaly Detection Performance: LogBERT recorded F1 scores of 0.8232 on HDFS, 0.9083 on BGL, and 0.9664 on Thunderbird, outperforming the other unsupervised baselines listed in the comparison.The comparison attributes its results to BERT-based contextual understanding and reports benefits from MLKP and VHM tasks.
  • 6.1. Anomaly Detection Performance: Predictive loss yielded F1 scores of 0.9123 on HDFS, 0.6932 on BGL, and 0.5142 on Thunderbird, with limitations on long and complex log data.The paper explains that token-level incorrect predictions can make loss-based scoring unsuitable for evaluating ordered log-key context.
  • 6.1. Anomaly Detection Performance: LAnoBERT achieved F1 scores of 0.9645 on HDFS, 0.8749 on BGL, and 0.9990 on Thunderbird using predictive probability, outperforming the other models except on BGL.The results identify mask probability as a critical factor in performance improvement.
  • 6.1. Anomaly Detection Performance: LAnoBERT performed comparably to benchmark models on BGL and better than some benchmarks on HDFS and Thunderbird in the most realistic parser-free unsupervised setting.The paper characterizes this setting as the most realistic but also the most challenging.
  • 6.2. Performance according to the BERT learning method: Natural-language pre-training improved BGL F1 by 0.0271 to 0.9020 but reduced HDFS F1 by 0.0341 to 0.9304 versus training from scratch.The paper relates the difference to BGL’s more complex vocabulary and HDFS’s simpler log structure.

7. Conclusion

The paper concludes that LAnoBERT detects abnormal logs without a parser by learning normal-log context with masked language modeling and using prediction-based scores. It reports robust performance, while identifying dataset-specific training, computational cost, and limited preprocessing as future challenges.

  • 7. Conclusion: LAnoBERT is an unsupervised, parser-free model that learns normal-log context with MLM and detects anomalies using prediction error and predictive probability.The method also introduces a score calculation approach for logs whose normal and abnormal data have similar characteristics.
  • 7. Conclusion: LAnoBERT can be a robust and parser-independent log anomaly detection model despite benchmark performance depending heavily on log-parser use.This conclusion follows the paper’s comparison of parser-dependent benchmarks with the proposed parser-free approach.
  • 7. Conclusion: LAnoBERT requires individual training for each log dataset, limiting its direct coverage of diverse log structures across distributed systems, supercomputers, and server applications.The paper proposes a unified framework as a future research direction.
  • 7. Conclusion: LAnoBERT’s Transformer architecture has higher stated computational complexity than RNN-based models because self-attention costs O(n^2 · d) versus O(n · d^2) for recurrence.The paper suggests parameter-efficient methods such as LoRA and Adapter to address computational inefficiency.
Loading 2111.09564v3…