Source-linked AI summary
Seeing the Needle in the Haystack: Towards Weakly-Supervised Log Instance Anomaly Localization via Counterfactual Perturbation
Yutszyuk Wong, Wentai Wu, Yuen-Ying Yeung, Weiwei Lin
TL;DR
Large-scale log systems make instance-level anomaly annotations costly, leaving weakly supervised localization as an important challenge. LogMILP combines prototype-guided multi-instance learning with counterfactual perturbation consistency, and experiments on three public datasets report competitive detection with more reliable instance localization.
Problem
Instance-level log anomaly annotations are prohibitively expensive at scale, while weakly supervised MIL still faces unreliable localization and interpretation.
Method
LogMILP uses learnable prototypes, multi-head attention, and counterfactual perturbation consistency within a MIL framework trained only with bag-level labels.
Results
Experiments on BGL, Spirit, and ZooKeeper show competitive bag-level detection and clear advantages on instance-level localization metrics such as Loc@3 and SR.
Takeaways & Limitations
LogMILP provides a practically viable weakly supervised approach that jointly supports anomaly detection, instance localization, and more reliable localization interpretation.
Takeaways & Limitations
Robustness to noisy data remains untested.
Abstract
from arXiv · showhide
Log anomaly detection is a critical task for system operations and security assurance. However, in networked systems at scale, log data are generated at massive scale while instance-level annotations are prohibitively expensive, posing great difficulties to fine-grained anomaly localization. To address this challenge, we propose LogMILP (Log anomaly localization based on Multi-Instance Learning enhanced by prototypes and Perturbation), a weakly supervised framework that enables both bag-level anomaly detection and instance-level anomaly localization using only bag-level labels. Our method guides the model to pinpoint the critical log entries using prototype-guided structural modeling with counterfactual perturbation consistency regularization, thereby improving localization reliability and interpretability under coarse-grained supervision. Experimental results on three public datasets demonstrate that LogMILP achieves competitive detection performance while yielding significantly more reliable instance-level localization. Our code is open-sourced at https://github.com/YUK1207/LogMILP.
I. INTRODUCTION
LogMILP addresses weakly supervised log anomaly localization, where large-scale systems make instance-level labels difficult to obtain and existing MIL methods can mislocalize critical entries. It combines prototype-guided modeling, multi-head attention, and counterfactual perturbation consistency to improve detection and localization.
- Large-scale, semantically complex log data make efficient anomaly detection and precise critical-entry localization difficult.
- Weakly supervised methods reduce labeling demands but struggle with instance localization, while MIL attention can be distracted by frequent patterns and fail to reveal causal contribution.
- LogMILP uses learnable prototypes and instance-prototype similarity statistics to guide attention allocation and bag-level prediction.
- Counterfactual perturbation removes key identified instances during training to focus the model on decisive evidence and improve localization reliability and interpretability.
- The method evaluates instance localization with Loc@k and Success Rate alongside traditional bag-level detection, using experiments on BGL, Spirit, and ZooKeeper.
- LogMILP achieves clear advantages in both detection performance and localization reliability.
II. RELATED WORK
Related work spans supervised, unsupervised, self-supervised, and weakly supervised log anomaly detection. MIL matches coarse window-level labeling but still faces noisy localization and unreliable interpretation when attention is treated as contribution.
- Supervised log anomaly methods can perform strongly with annotations but depend on manual instance-level labeling, limiting scalability.
- Self-supervised contextual models such as LogBERT and LogFormer improve session-level detection, but primarily target detection accuracy rather than instance localization.
- Weakly supervised MIL treats log batches or time windows as bags and uses bag-level labels to support anomaly detection and localization when exact timestamps are unavailable.
- Prior MIL methods can be affected by noisy logs, high-frequency templates, and statistical bias during instance localization.
- Prototype mechanisms add structural constraints and characterize representative latent patterns, helping separate instances without instance-level labels.
D. Perturbation Consistency and Interpretability
The paper formulates windowed log analysis as MIL and uses counterfactual perturbation to test whether highly attended instances are genuinely important. LogMILP combines this reliability mechanism with prototype-guided attention and contextual instance representations.
- Attention or saliency scores may not reflect the true basis of model decisions, motivating deletion, masking, or replacement of model-identified critical inputs.
- Removing a highest-attention instance with little prediction change indicates potentially spurious localization, so perturbation consistency provides a reliability check.
- The task treats each time window as a bag and each contained log entry as an instance, with training using only bag-level labels.
- LogMILP encodes instance representations, aggregates them through prototype-guided multi-head attention, and trains with key-instance perturbation consistency.
- A log sequence is split into fixed-size, strided subsequences that serve as MIL bags.
C. Instance Encoding
The model encodes log instances into contextual latent representations, relates them to learnable prototypes, and combines prototype statistics with attention for bag-level prediction. Counterfactual perturbation and multiple regularization objectives encourage attention toward decision-critical instances using only bag-level labels.
- Instance representation encoding: Input embeddings are linearly projected into a hidden latent space and then contextually encoded with a two-layer Transformer.The transformations are H = XW + b and Z = Ψ(H).
- Prototype-guided modeling: Learnable prototypes characterize latent log patterns through normalized instance–prototype distances and similarity scores.Each instance receives a maximum prototype similarity and an anomaly-candidate bias of 1 − mi.
- Prototype-guided modeling: Prototype statistics comprising maximum similarity, assignment entropy, and average activation are concatenated as an auxiliary descriptor for prediction.The descriptor is not itself a direct anomaly score.
- Perturbation consistency: Counterfactual perturbation zeroes the highest-attention key instance, compares predictions before and after removal, and penalizes insufficient confidence drops.This encourages the model to focus on truly critical anomalous evidence rather than incidental attention patterns.
- Optimization: Focal classification loss is jointly optimized with prototype, attention-entropy, and consistency regularization while training uses only bag-level labels.Instance-level labels are excluded from parameter optimization.
F. Localizing Instance-level Anomalies
Instance localization selects candidate log entries from the attention head with minimum entropy, using the top-k attention-weighted instances in each positive bag. Localization accuracy is then evaluated with dedicated metrics against ground-truth anomalous instances.
- Candidate selection: For each positive bag, the model examines the attention head with minimum entropy and selects its top-k highest-weighted instances as anomaly candidates.The selected candidate set is denoted Stop B.
- Evaluation: Loc@k and Success Rate are used to evaluate instance-level anomaly localization accuracy.The supplied section introduces these metrics before defining Success Rate separately.
- Localization metric: Loc@k compares selected candidates with ground-truth anomalous instances using their overlap within each positive bag.The ground-truth anomalous set is denoted Sa B.
- Training procedure: The training logic processes mini-batches, computes classification and auxiliary losses, and updates model parameters through the combined objective.The algorithm includes positive and negative bag indexing and back-propagation through the total loss.
2) Success Rate:
Success Rate evaluates whether removing the selected key instance changes the predicted anomaly probability as expected. Higher Success Rate indicates greater reliance on decision-critical instances rather than incidental correlated patterns.
- Success Rate definition: Success Rate compares each positive bag’s predicted anomaly probabilities before and after removing its key instance.The two predictions are denoted Porig and Ppert.
- Interpretation: A higher Success Rate indicates that localization relies more on truly decision-critical instances than on incidental correlated patterns.The metric therefore assesses the reliability of the selected instance under perturbation.
A. Experimental Setup
The evaluation uses three public log anomaly detection datasets, unified preprocessing, and bag-level protocols. Comparisons include classical sequence models, pretrained semantic or Transformer methods, and the closely related weakly supervised MIL baseline MIDLog.
- Datasets and preprocessing: LogMILP is evaluated on BGL, Spirit, and ZooKeeper after unified preprocessing into multi-instance bags.BGL and ZooKeeper use sliding time windows, while Spirit uses non-overlapping blocks aggregated into fixed-size bags.
- Baselines: The comparison includes DeepLog, LogAnomaly, LogBERT, LogFormer, and MIDLog under a unified bag-level evaluation protocol.MIDLog is the weakly supervised MIL baseline most closely related to LogMILP.
- Evaluation protocol: For instance localization and perturbation evaluation, adaptations compute Loc@3 and Success Rate without modifying the compared models’ core bag-level detection logic.The adaptations are applied offline to LogBERT and LogFormer, whose original designs do not directly target these evaluations.
3) Evaluation Protocols:
The evaluation uses bag-level F1, precision, recall, and instance-level Loc@3 and SR under weak supervision, with LogMILP compared against baselines across datasets. LogMILP achieves the strongest reported bag-level F1 scores and reliable instance-level localization, though some baseline localization results are supplementary adaptations.
- Evaluation Protocols: F1 is the primary bag-level metric, with validation-selected threshold τ applied to test probabilities for precision, recall, and F1.
- Evaluation Protocols: Loc@3 measures localization accuracy and SR measures causal reliability, computed only during testing from available instance-level ground truth.
- Bag-level Anomaly Detection: LogMILP achieved the best F1 scores of 0.9342, 0.9295, and 0.9967 across the three datasets, with a recall gap exceeding 10% over the second-best method on BGL.
- Instance-level Anomaly Localization: LogMILP achieved high SR across all three datasets and substantially outperformed baselines in Loc@3 on Spirit.
C. Ablation Study
The ablation study tests whether counterfactual perturbation consistency contributes to localization. Removing the consistency loss reduces localization reliability, supporting its role in learning critical anomalous evidence without instance-level labels.
- C. Ablation Study: The ablation compares the full model with a version without consistency loss while keeping all other components unchanged.
- C. Ablation Study: Consistency regularization significantly improves localization reliability, particularly on SR.
- C. Ablation Study: Counterfactual perturbation encourages the model to learn anomaly sources without relying on instance-level labels.
- C. Ablation Study: The framework unifies prototype guidance, multi-head attention aggregation, and key-instance perturbation consistency training using only bag-level labels.
- Limitations: Robustness to noisy data remains untested, defining a stated limitation of the proposed approach.