PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs

Sasho Nedelkoski, Jasmin Bogatinovski, Alexander Acker, Jorge Cardoso, Odej Kao

arXiv:2008.09340v1cs.LGcs.IRstat.ML

TL;DR

Log anomaly detectors often rely on fixed representations that generalize poorly to unseen logs. Logsy learns representations by classifying target-system normal logs against auxiliary-system samples with a self-attention encoder and hyperspherical objective, improving anomaly-detection performance over prior methods.

  • Problem

    Existing one-class methods use predefined log representations that cannot adequately capture semantic differences between normal and anomalous logs, limiting generalization to unseen logs.

  • Method

    Logsy uses a self-attention encoder and hyperspherical classification objective to separate target-system normal logs from auxiliary-system samples and learn compact representations.

  • Results

    Logsy achieves the best scores overall, with average F1 scores of 0.448 on Blue Gene/L, 0.99 on Thunderbird, and 0.77 on Spirit.

  • Takeaways & Limitations

    Logsy improves precision while preserving high recall, including outperforming baselines with only 10% of target-system data.

Abstract

from arXiv · show

The detection of anomalies is essential mining task for the security and reliability in computer systems. Logs are a common and major data source for anomaly detection methods in almost every computer system. They collect a range of significant events describing the runtime system status. Recent studies have focused predominantly on one-class deep learning methods on predefined non-learnable numerical log representations. The main limitation is that these models are not able to learn log representations describing the semantic differences between normal and anomaly logs, leading to a poor generalization of unseen logs. We propose Logsy, a classification-based method to learn log representations in a way to distinguish between normal data from the system of interest and anomaly samples from auxiliary log datasets, easily accessible via the internet. The idea behind such an approach to anomaly detection is that the auxiliary dataset is sufficiently informative to enhance the representation of the normal data, yet diverse to regularize against overfitting and improve generalization. We propose an attention-based encoder model with a new hyperspherical loss function. This enables learning compact log representations capturing the intrinsic differences between normal and anomaly logs. Empirically, we show an average improvement of 0.25 in the F1 score, compared to the previous methods. To investigate the properties of Logsy, we perform additional experiments including evaluation of the effect of the auxiliary data size, the influence of expert knowledge, and the quality of the learned log representations. The results show that the learned representation boost the performance of the previous methods such as PCA with a relative improvement of 28.2%.

I. INTRODUCTION

Logsy addresses poor generalization in log anomaly detection by learning semantic log representations that distinguish target-system normal data from auxiliary anomalies. It combines self-attention, auxiliary data, and a hyperspherical objective to produce compact representations.

  • Logs record runtime system events and support anomaly detection for reliability, stable operation, and loss mitigation.
  • One-class methods learn normal behavior from mostly unlabeled data, but existing representations generalize poorly to previously unseen log messages.
  • Labeled-data approaches require costly expert annotation and repeated retraining while still leaving unseen logs between retraining epochs unresolved.
  • Logsy separates target-system normal logs from auxiliary datasets to improve representations, regularize overfitting, and generalize to unseen logs.
  • The method uses self-attention and a hyperspherical decision boundary to learn compact representations and distance-based anomaly scores.
  • The study evaluates Logsy on Blue Gene/L, Thunderbird, and Spirit datasets and reports significant improvements over previous studies.

II. RELATED WORK

Prior work spans supervised, unsupervised, deep-learning, and NLP-based log anomaly detection, but target-system labels are costly and template-index methods struggle with new events. Logsy instead uses auxiliary log data to provide anomalous-distribution bias without target-system anomaly labels.

  • Supervised log anomaly detection assumes normal and anomalous samples, but obtaining system-specific labels is costly and often infeasible.
  • PCA-based detection groups log keys by sessions and represents each session through log-key occurrence counts.
  • LSTM-based methods forecast log events or model normal and anomalous sequences, but one-hot template representations cannot handle newly appearing events.
  • NLP-based approaches use TF-IDF, word2vec, traditional classifiers, or attention-based Bi-LSTM models for log analysis.
  • Logsy adds anomalous-distribution bias through easily accessible auxiliary datasets and evaluates without target-system labels.
  • The method assumes realistic auxiliary log samples provide more informative descriptions of normal and anomalous data than noise or no auxiliary data.
  • Logsy formulates target logs as normal and auxiliary logs as anomalous, then maps log-message embeddings to representations and anomaly scores.

IV. SELF-ATTENTIVE ANOMALY DETECTION WITH CLASSIFICATION-BASED OBJECTIVE

The method represents unstructured log messages as token sequences with contextual embeddings and defines target and auxiliary datasets for anomaly detection. Its learned space is intended to place similar messages near one another and different messages farther apart.

  • A log is a temporally ordered sequence of unstructured messages generated by software logging instructions.
  • Each message contains a constant log template and an optional varying part called variables.
  • Tokens are the smallest inseparable objects in messages and may be words, word pieces, or characters depending on tokenization.
  • A token’s context consists of preceding and subsequent tokens, while an embedding vector is a d-dimensional real-valued representation.
  • The learned vector space should place semantically similar log messages close together and substantially different messages farther apart.
  • The target dataset supplies test-time logs, whereas auxiliary data from unrelated systems is used only for training and contains no target-system anomaly data.

B. Logsy

Logsy tokenizes log messages and uses a Transformer encoder with multi-head self-attention to learn contextual log representations. Its [EMBEDDING] token output provides the final vector representation used by the model.

  • Tokenization: Logsy preprocesses raw messages into word-token sequences, removing endpoints, capitalization, special characters, numerical tokens, and other common tokens.
  • Operation modes: Logsy operates offline for parameter training and hyperparameter selection, then online to generate a vector representation and anomaly score for each message.
  • Model: The model combines learned token embeddings with positional encodings before applying a Transformer encoder with multi-head self-attention.Positional information is merged with token embeddings as x′ = x + n.
  • Encoder: Residual connections and layer normalization update attention outputs, while feed-forward layers enrich each row without changing the representation dimensionality.The attention transformation is combined as X′ = norm(X′ + X′′).
  • Output representation: A final linear layer extracts the transformed [EMBEDDING] token as the log vector representation used by the objective function and as a log message embedding.

C. Objective function

Logsy replaces the standard classifier’s half-space boundary with a hyperspherical objective that pulls normal samples toward a center while separating auxiliary anomalies. This is intended to produce compact, distance-meaningful representations without trivial constant mappings.

  • Spherical loss: Logsy uses a spherical loss that enforces a compact hyperspherical decision region for normal samples while learning distinctions from anomalous samples.
  • Training data: Training combines normal logs from the target system with auxiliary anomaly data, encoding each message into a learned p-dimensional representation.
  • Motivation: A standard sigmoid classifier can scatter normal representations because its decision boundary is a half-space rather than a compact region.
  • Hyperspherical classifier: The Gaussian radial basis function replaces the standard classifier function to enforce compactness of log-message representations.
  • Optimization behavior: Normal samples are pulled toward sphere center c = 0, anomalous samples are favored at larger distances, and the auxiliary term prevents constant-vector solutions.

D. Anomaly score and detecting anomalies

Logsy defines anomaly scores as distances between log representations and the hypersphere center. Low scores indicate normal messages, while scores above a threshold identify anomalies.

  • Anomaly score: The anomaly score is the distance from a log vector, obtained from the [EMBEDDING] token, to the hypersphere center c.
  • Detection rule: Messages with A(x_i) > E are classified as anomalies; otherwise, they are considered normal.

E. Including expert knowledge

Logsy can incorporate operator-labeled samples as additional anomaly information. These samples may improve realism and performance but require labeling and retraining or fine-tuning.

  • Expert knowledge: Operator-labeled samples can be added alongside auxiliary data or used for fine-tuning after pre-training with normal and auxiliary data.
  • Trade-off: These labels provide more realistic anomaly examples that may improve performance, but obtaining them is costly.

F. Vector representations of the logs

Logsy learns numerical log representations that compactly group normal samples while separating anomalies in a hyperspherical space. These representations can also replace weaker features in other anomaly-detection methods.

  • F. Vector representations of the logs: Logsy produces numerical log representations for anomaly detection and can replace less powerful representations such as TF-IDF in PCA.The learned vectors are used by Logsy’s objective function and can also enhance other log-based methods.
  • F. Vector representations of the logs: The [EMBEDDING] token vector summarizes each log message and is the model’s only output to the loss function.This forces the vector to encode the message context.
  • F. Vector representations of the logs: The hyperspherical decision boundary places normal samples close together around the sphere’s center while anomalies disperse toward the boundary.The paper illustrates this ideal arrangement in a lower-dimensional representation space.

V. EVALUATION

The evaluation uses three evolving HPC log datasets as target systems and additional HPC logs as auxiliary data. Timestamp-based splits deliberately test generalization to previously unseen messages.

  • V. EVALUATION: Logsy is evaluated on the Blue Gene/L, Spirit, and Thunderbird HPC datasets, with HPC RAS logs added as auxiliary data.The RAS dataset lacks labels and therefore cannot serve as a target evaluation dataset.
  • V. EVALUATION: Auxiliary data comes from remaining systems, preventing target-system information leakage and using logs accessible from other systems.No labeled target-system samples are included in the auxiliary data.
  • V. EVALUATION: The datasets contain many new log messages over time, making system evolution central to the generalization evaluation.The systems were collected between 2004 and 2006 and include anomaly and normal messages identified by category tags.
  • V. EVALUATION: Thunderbird and Spirit exceed 200 million messages, but evaluation restricts them to the first 5 million timestamp-sorted messages while preserving unseen test logs.Blue Gene/L is retained in full because it contains fewer than 5 million messages.
  • V. EVALUATION: Five timestamp-ordered train-test splits range from 10% training and 90% testing to 80% training and 20% testing.This design ensures that test data contains log messages unseen during training.
  • V. EVALUATION: Every split contains previously unseen test messages, and reducing training data increases the number of novel test messages.This directly supports empirical testing of generalization.

1) Evaluation methods:

Logsy is compared with DeepLog and PCA using standard classification metrics across three datasets and multiple splits. It generally improves F1 and precision, especially for unseen messages and limited target-system training data.

  • 1) Evaluation methods:: Evaluation uses F1-score, precision, recall, and accuracy, with anomalous logs designated as the positive class.The metrics are computed from true and false positive and negative predictions.
  • 2) Baselines:: Logsy is compared with publicly available PCA and DeepLog baselines whose parameters are tuned for their best F1 score.LogAnomaly is not directly evaluated because no public implementation was available.
  • B. Results and discussion: Logsy achieves average F1 scores of 0.448 on Blue Gene/L, 0.99 on Thunderbird, and 0.77 on Spirit, exceeding both baselines in all experiments.The baselines retain high recall but show lower precision.
  • B. Results and discussion: Logsy preserves high recall while improving precision by correctly classifying unseen messages and reducing false positives.On Blue Gene/L, DeepLog and PCA have precision values 2–4 times lower than Logsy.
  • B. Results and discussion: 0.32 F1-score is achieved by Logsy with 10% Blue Gene/L training data, versus 0.24 for the strongest baseline.On Thunderbird, Logsy reaches an F1-score of 0.99 with the first 10% of training data.

1) The effect of the auxiliary data on the evaluation scores:

The experiments examine how auxiliary and labeled anomaly data affect Logsy's anomaly-detection performance and learned representations. Increasing auxiliary data improves scores up to a plateau, while limited target anomaly labeling reaches strong performance quickly.

  • Auxiliary-data experiments use Blue Gene/L, Thunderbird, and Spirit as target systems under a 20%-train, 80%-test split.
  • Increasing auxiliary data from 1 to 250000 raises all evaluation scores, but adding samples from 100000 to 250000 produces no further change.The authors report that the information present in the auxiliary data is already represented in 100000 random samples.
  • A single auxiliary sample can regularize the hypersphere loss and prevent trivial solutions, while more diverse datasets may further improve performance.
  • Including expert labeling: Increasing labeled anomaly samples in Blue Gene/L improves performance, with 2% labeled data already achieving the best reported 0.8 F1-score.
  • The learned Blue Gene/L representations concentrate normal samples near a hypersphere centre and disperse most anomalies outside it, enabling distance-threshold anomaly scoring.

3) Utilization of the learned log embeddings in related approaches:

The learned Logsy embeddings can replace TF-IDF representations in PCA, improving a previous approach across all three HPC datasets. This supports the embeddings' usefulness beyond Logsy itself.

  • Replacing PCA's original TF-IDF representations with Logsy embeddings improves F1-score by 0.09 for Blue Gene/L, 0.11 for Thunderbird, and 0.01 for Spirit.
  • The average relative improvement from using Logsy embeddings in PCA is 28.2% in F1-score.
  • The conclusion reports that adopting Logsy log vectors in PCA improves F1-score by 0.07, corresponding to 28.2%.
Loading 2008.09340v1…