PaperlayerView exact revision on arXiv

Source-linked AI summary

Log-based Anomaly Detection Without Log Parsing

Van-Hoang Le, Hongyu Zhang

arXiv:2108.01955v3cs.SEcs.AI

TL;DR

Existing anomaly-detection approaches are affected by information loss from log-parsing errors, including OOV words and semantic misunderstandings. NeuralLog instead encodes raw log messages as semantic vectors and applies a Transformer-based classifier to log sequences; across four public datasets, it achieves F1-scores above 0.95 and outperforms existing approaches.

  • Problem

    Log-parsing errors caused by OOV words and semantic misunderstandings limit existing log-based anomaly detection.

  • Method

    NeuralLog encodes raw log messages with BERT-derived semantic representations and detects anomalies using a Transformer-based classification model over log sequences.

  • Results

    NeuralLog achieves F1-scores above 0.95 on four public datasets and outperforms existing log-based anomaly detection approaches.

  • Takeaways & Limitations

    The results support parsing-free semantic representation and contextual sequence modeling for log-based anomaly detection.

  • Takeaways & Limitations

    The evaluation uses datasets from a limited number of distributed and supercomputer systems that do not cover all domains.

Abstract

from arXiv · show

Software systems often record important runtime information in system logs for troubleshooting purposes. There have been many studies that use log data to construct machine learning models for detecting system anomalies. Through our empirical study, we find that existing log-based anomaly detection approaches are significantly affected by log parsing errors that are introduced by 1) OOV (out-of-vocabulary) words, and 2) semantic misunderstandings. The log parsing errors could cause the loss of important information for anomaly detection. To address the limitations of existing methods, we propose NeuralLog, a novel log-based anomaly detection approach that does not require log parsing. NeuralLog extracts the semantic meaning of raw log messages and represents them as semantic vectors. These representation vectors are then used to detect anomalies through a Transformer-based classification model, which can capture the contextual information from log sequences. Our experimental results show that the proposed approach can effectively understand the semantic meaning of log messages and achieve accurate anomaly detection results. Overall, NeuralLog achieves F1-scores greater than 0.95 on four public datasets, outperforming the existing approaches.

I. INTRODUCTION

NeuralLog addresses the information loss caused by inaccurate log parsing in anomaly detection by representing raw log messages semantically and modeling log-sequence context. Evaluated on four public datasets, it achieves F1-scores above 0.95 and outperforms existing approaches.

  • Problem: Existing log-based anomaly detection methods rely on parsers that can introduce errors and directly reduce detection performance.The study identifies changing logging statements and parsing inaccuracies as important sources of error.
  • Approach: NeuralLog avoids log parsing by transforming each raw log message into a semantic vector.The approach is designed to retain semantic information that parsing errors may remove.
  • Approach: A Transformer-based classification model uses sequences of semantic vectors to capture contextual information between log messages.The model applies multi-head self-attention to contextual information in log sequences.
  • Evaluation: NeuralLog was evaluated on four public datasets for log-based anomaly detection.The evaluation examines the approach on real-world log data.
  • Results: All F1-scores exceeded 0.95, and NeuralLog outperformed existing log-based anomaly detection approaches.The reported results also indicate that NeuralLog understands log semantics and handles OOV words.

B. Log Parsing Methods

Log parsing converts raw messages into event templates, but parser errors—especially from OOV words—can distort log-event representations and undermine anomaly detection. The paper’s empirical analysis shows that OOV words remain prevalent across datasets and can create extra events that existing detection methods handle poorly.

  • Parsing approaches: Log parsers remove variable parameters and retain keywords to convert semi-structured messages into event templates.The paper reviews frequent-pattern, clustering, language-modeling, and heuristic parsing techniques, including Drain, AEL, IPLoM, and Spell.
  • Detection approaches: Existing anomaly detectors use parsed events in different ways, including log count vectors, next-event prediction, and sequence models.PCA, IM, and ADR analyze log count vectors, while DeepLog predicts the next parsed template and LogAnomaly and LogRobust use learned representations with neural models.
  • OOV analysis: 94.12% of unique words in the BGL testing set were unseen during training under the 60/40 split, although they occurred in 8.51% of testing log messages.The testing set contained 362,123 unseen unique words concentrated in 160,403 of 1,885,398 log messages.
  • OOV analysis: 30.4% of words remained out of vocabulary in the Thunderbird testing set even when 80% of its logs were used for training.On BGL, more than 80% of testing words were unseen across training proportions from 20% to 80%.
  • OOV analysis: At an 80% training split, OOV words appeared in 6.7% of BGL testing messages and 1.7% of Thunderbird testing messages.OOV-containing templates were more widespread: above 80% on BGL and above 60% on Thunderbird at the 80% split.
  • Detection impact: OOV words can create extra log events, causing fixed-dimensional count-vector models to reject new events and DeepLog to classify them as anomalies.The BGL dataset included 1,165 events appearing only once, and these rare extra events formed a majority of log templates.

B. Log Parsing Errors Introduced by Semantic Misunderstanding

Log parsers can misunderstand log semantics in two ways: treating parameters as keywords or removing essential keywords, causing distinct events to collapse or multiply incorrectly.

  • Semantic misunderstanding produces two parsing-error cases: misidentifying parameters as keywords and misidentifying keywords as parameters.
  • Parameters mistaken for keywords split messages from one template into multiple templates, creating extra log events.Drain produced about 80% extra log events on BGL and 72% on Thunderbird.
  • Removing essential keywords can merge normal and anomalous messages into one event, making their states difficult to distinguish.For example, “machine check enable” and “machine check interrupt” are parsed as “machine check ∗”.
  • On BGL, Spell, AEL, and IPLoM produced 58,228, 20,154, and 31,298 misidentified log messages, respectively.
  • On Thunderbird, Spell, AEL, and IPLoM produced 3,851, 1,463, and 5,687 misidentified log messages, respectively.

C. The Impact of Log Parsing Errors on Anomaly Detection

Existing anomaly-detection methods depend on parsed log events, so parsing inaccuracies and parser choice affect their detection performance.

  • Existing approaches parse messages into log events, construct log sequences, and then apply unsupervised or supervised anomaly-detection models.
  • Different parsers produce different anomaly-detection results, demonstrating that performance is affected by parsing accuracy.SVM performed better with Drain and AEL because those parsers produced fewer inaccurate log events.
  • Fixing parsing errors on BGL was used to test whether more accurate parsing improves SVM and LogRobust detection accuracy.
  • OOV handling failures lose semantic information, while semantic misunderstanding introduces additional parsing errors that limit log-event-based detection.

IV. NEURALLOG: LOG-BASED ANOMALY DETECTION WITHOUT LOG PARSING

NeuralLog detects anomalies directly from raw log messages without log parsing by combining BERT-based semantic representations with Transformer classification.

  • NeuralLog directly uses raw log messages to detect anomalies without relying on log parsing.
  • The approach preprocesses logs, encodes each message into a semantic vector with BERT, and applies a Transformer to detect anomalies.

A. Preprocessing

NeuralLog preprocesses raw messages by tokenizing and normalizing text, while retaining textual information from message headers and content for semantic representation.

  • Preprocessing tokenizes each log message using common delimiters, lowercases capital letters, and removes non-character tokens.
  • NeuralLog uses textual information from verbosity, component, and content rather than analyzing message content alone.
  • WordPiece tokenization is adopted as the first step for handling OOV words.
  • WordPiece builds vocabulary iteratively by selecting character or symbol pairs that maximize training-data likelihood.
  • WordPiece was chosen because it handles OOV words effectively and reduces vocabulary size compared with space-, stemming-, or camel-case-based strategies.

2) Log Message Representation:

NeuralLog represents preprocessed log messages as semantic vectors with BERT, then uses positional information and a Transformer encoder to classify log sequences as anomalous or normal.

  • Log Message Representation: BERT is preferred over Word2Vec because contextual representations distinguish words whose meanings vary with position and context.
  • Log Message Representation: BERT encodes each tokenized log message into a fixed-dimensional semantic vector.NeuralLog uses BERT base with 12 transformer-encoder layers and 768 hidden units per layer.
  • Log Message Representation: WordPiece decomposes OOV words into subwords, allowing BERT to represent unseen words from their subword collections.
  • Anomaly Detection Model: A Transformer encoder receives semantic vectors with positional embeddings and predicts whether each log sequence is anomalous.The pipeline preprocesses new messages, vectorizes them, feeds the sequence to the trained model, and produces an anomaly classification.

2) Datasets:

NeuralLog is evaluated on four public log datasets, using dataset-specific sequence construction and standard precision, recall, and F1-score metrics.

  • Datasets: NeuralLog is evaluated on four public datasets: HDFS, Blue Gene/L, Thunderbird, and Spirit.The HDFS dataset contains 11,175,629 log messages, while BGL contains 4,747,963 log messages.
  • Datasets: HDFS log sequences are formed by grouping messages with the same block ID, whereas the other datasets are ordered by timestamp.
  • Evaluation Metrics: The evaluation uses Precision, Recall, and F1-Score to measure anomaly-detection effectiveness.
  • Evaluation Metrics: Precision measures the proportion of detected abnormal sequences that are correct, while Recall measures the proportion of real anomalies identified.
  • Evaluation Metrics: F1-Score is the harmonic mean of Precision and Recall.

B. RQ1: How effective is NeuralLog?

On four public datasets, NeuralLog generally outperforms existing approaches, achieving the best results on BGL, Thunderbird, and Spirit and comparable results on HDFS.

  • RQ1 Results: NeuralLog achieves the best results on BGL, Thunderbird, and Spirit, with F1-scores of 0.98, 0.96, and 0.97, respectively.
  • RQ1 Results: On HDFS, NeuralLog achieves Recall of 1.00 with high precision and comparable overall results to existing approaches.Recall of 1.00 means NeuralLog identifies all anomalies captured by the dataset.
  • RQ1 Results: Existing methods based on parsed log events perform poorly when parsing is inaccurate, particularly on BGL and Thunderbird.They achieve high F1-Score on Spirit, where the parsing error rate is only 0.1%.
  • RQ1 Results: LogRobust obtains F1-scores of 0.75 on BGL and 0.68 on Thunderbird, partly because Drain inaccurately parses many messages.
  • RQ1 Results: NeuralLog averages 14.3 minutes for encoding, 5.2 minutes for training, and 3.1 milliseconds per log sequence for detection.Parsing-based baselines require 102 minutes on average for preprocessing, while Log2Vec averages 314 minutes.

semantic meaning of log data?

NeuralLog’s raw-message representation and WordPiece tokenization improve semantic understanding by avoiding parser-induced information loss and preserving OOV meaning.

  • Representation Evaluation: NeuralLog directly encodes preprocessed raw log messages, while its variants use parsed template indexes or parsed template embeddings.
  • Representation Evaluation: On BGL, template-index and template-embedding variants achieve F1-scores of 0.46 and 0.90, compared with 0.98 for NeuralLog.
  • OOV Evaluation: WordPiece enables NeuralLog to retain OOV meaning by splitting unseen words into subwords rather than removing them.
  • OOV Evaluation: On Thunderbird, NeuralLog achieves an F1-score of 0.96, versus 0.80 for NeuralLog-Word2Vec and 0.90 for NeuralLog-NoWordPiece.

D. RQ3: Effectiveness of NeuralLog under different settings

NeuralLog remains effective across alternative pretrained encoders and Transformer settings, while its performance depends somewhat on retaining sufficient model capacity. Its raw-message, semantic representation also addresses information loss from parsing and removes a limitation of conventional methods.

  • Pretrained language models: GPT2, Roberta, and BERT can all understand log-message semantics and achieve promising anomaly-detection results.GPT2 and Roberta replace BERT using base models with 12 layers, 12 attention heads, and 768 hidden units.
  • Transformer hyperparameters: Reducing Transformer attention heads and feed-forward network size can slightly hurt NeuralLog’s performance across four datasets.The resulting F1-scores are reported in Figure 11 for varied hyperparameter settings.
  • Representation design: NeuralLog directly processes raw log messages into semantic representations, avoiding information loss caused by log parsing.Its representations capture semantic information in individual messages and relationships between messages.
  • Representation design: NeuralLog removes words containing numbers and special characters, but those tokens may carry useful anomaly-detection information in some scenarios.Examples include node IDs, task IDs, IP addresses, and exit codes.

B. Threats to Validity

The study’s validity is bounded by limited subject-system coverage, possible implementation discrepancies, manually introduced label noise, and known variability in log parsers and log data. The authors therefore identify broader datasets, data-quality analysis, and careful comparison procedures as important boundaries on interpretation.

  • Subject datasets: The four datasets cover distributed systems and supercomputers but still represent a limited number of subject systems and domains.The authors plan to evaluate NeuralLog on more datasets from a wider variety of systems.
  • Tool comparison: Comparison results may be affected by the correctness of related-tool implementations despite checks against results reported in the original papers.The evaluation used implementations from Loglizer and the tool authors with prior default settings.
  • Labeling noise: Manual labeling of the four public datasets may introduce false-positive or false-negative noise into the experiments.The authors believe any noise is small but identify data-quality investigation as future work.
  • Data and preprocessing: Log-parser accuracy and preprocessing choices can affect downstream log-mining results and may require manual parameter tuning.Prior evaluations also found that some parsers do not scale well with log volume.
  • Evaluation scope: NeuralLog’s reported effectiveness is evaluated on four public datasets and compared with related anomaly-detection approaches.The conclusion describes the approach as effective and efficient for log-based anomaly detection.
Loading 2108.01955v3…