Source-linked AI summary
Anomaly Detection using Autoencoders in High Performance Computing Systems
Andrea Borghesi, Andrea Bartolini, Michele Lombardi, Michela Milano, Luca Benini
TL;DR
HPC anomaly detection is difficult because systems are large and supervised approaches require scarce, correctly labeled healthy and faulty data. The paper trains autoencoders on normal behavior and uses reconstruction error to detect abnormal states, achieving 88%-96% accuracy on previously unseen anomalies.
Problem
HPC systems generate abundant monitoring data, but faulty behavior is sporadic and uncontrolled, making the correctly labeled datasets required by supervised approaches difficult or often impossible to obtain.
Method
The approach trains autoencoders on healthy system behavior and classifies inputs using the magnitude of their reconstruction error relative to a threshold.
Results
88%-96% accuracy was achieved in distinguishing normal and anomalous states, including anomaly types unseen during initial training.
Takeaways & Limitations
The method detects faulty conditions without injecting anomalies during training, requiring neither system logs nor changes to standard user workflows.
Takeaways & Limitations
Faulty behavior is sporadic and uncontrolled, and abnormal conditions are not necessarily recorded in logging systems, constraining the availability of labeled anomaly data.
Abstract
from arXiv · showhide
Anomaly detection in supercomputers is a very difficult problem due to the big scale of the systems and the high number of components. The current state of the art for automated anomaly detection employs Machine Learning methods or statistical regression models in a supervised fashion, meaning that the detection tool is trained to distinguish among a fixed set of behaviour classes (healthy and unhealthy states). We propose a novel approach for anomaly detection in High Performance Computing systems based on a Machine (Deep) Learning technique, namely a type of neural network called autoencoder. The key idea is to train a set of autoencoders to learn the normal (healthy) behaviour of the supercomputer nodes and, after training, use them to identify abnormal conditions. This is different from previous approaches which where based on learning the abnormal condition, for which there are much smaller datasets (since it is very hard to identify them to begin with). We test our approach on a real supercomputer equipped with a fine-grained, scalable monitoring infrastructure that can provide large amount of data to characterize the system behaviour. The results are extremely promising: after the training phase to learn the normal system behaviour, our method is capable of detecting anomalies that have never been seen before with a very good accuracy (values ranging between 88% and 96%).
1 Introduction
HPC anomaly detection is difficult because complex systems generate abundant monitoring data while faulty-state labels are scarce and hard to obtain. The paper proposes semi-supervised autoencoders trained on normal behavior to detect previously unseen anomalies, reporting 88%-96% accuracy.
- HPC systems contain many concurrently operating components whose failures, misconfigurations, or undesired application states can degrade performance.
- Monitoring data are abundant, but labeled healthy and unhealthy examples are scarce, making supervised anomaly detection difficult to train.
- The proposed semi-supervised approach trains autoencoders to learn normal system behavior rather than requiring labeled anomalies.
- The method is intended to detect faulty conditions unseen during training without injecting anomalies, requiring system logs, or changing users’ workflows.
- 88%-96% accuracy was reported while identifying new anomaly types without large amounts of labeled data.
2 Related Works
Prior HPC anomaly-detection approaches include supervised classifiers, density-estimation methods, and log-based clustering. Their limitations include dependence on labeled or synthetic training data and restricted anomaly coverage, motivating alternative learning strategies.
- Supervised methods classify HPC behavior using statistical features and labeled data, which the paper identifies as poorly suited to the HPC context.
- CADE combines real and artificially generated data to train a Random Forest classifier that ranks data points by anomalousness.
- The paper identifies supervised learning as the main limitation of the CADE-based approach.
- A k-means method detects faults from console logs, but it is limited to faults recognizable and recorded by individual nodes.
- Deep-learning anomaly detection has been studied in other domains, while several cited fault-diagnosis methods remain supervised.
- An industrial approach learns normal behavior through probability-density estimation and can handle unseen anomaly types without supervision.
3 Data Collection
The study uses D.A.V.I.D.E., a 45-node supercomputer with integrated monitoring designed to collect heterogeneous system measurements at scale. Storage constraints require retaining coarse-grained rather than raw historical data.
- D.A.V.I.D.E. is a 45-node CINECA supercomputer with 990 TFlops peak performance and four Tesla P100 GPUs per node.
- Examon provides fine-grained, lightweight, scalable monitoring and integrates heterogeneous data into a uniform repository describing system state.
- Raw monitoring data cannot be stored indefinitely because storage space is limited.
- The study uses coarse-grained data aggregated in 5-minute intervals and 166 node-level metrics, including loads, temperatures, fan speed, and power consumption.
4 The Autoencoder-based Approach
The approach assigns one autoencoder to each computing node and trains it on normal operating data. Anomalies are identified when new inputs produce unusually large reconstruction errors.
- Separate autoencoder models are created for individual nodes to learn each node’s normal feature correlations.
- The reconstruction error measures how well a trained autoencoder reproduces new inputs and increases when they differ from healthy training data.
- Each autoencoder uses 166 input neurons, a 1660-neuron sparse intermediate layer, and a 166-neuron linear output layer.
- The workflow creates node-specific autoencoders, trains them on normal data, and detects anomalies from reconstruction errors.
5 Experimental Evaluation
Experiments on the D.A.V.I.D.E. supercomputer evaluated autoencoder-based detection of injected frequency-governor misconfigurations using months of monitored data. Reconstruction errors separated normal and anomalous behavior, supporting threshold-based classification with high F-scores, although optimal thresholds depended on anomaly type.
- Experimental Setup: The experiments modeled conservative as normal and injected powersave and performance frequency-governor policies as anomalies.Conservative follows CPU load, whereas powersave and performance statically select the lowest and highest allowed frequencies.
- Experimental Setup: The evaluation used 83 days of Examon data: 66 normal days and 13 days with injected anomalies.Normal data were randomly split into training and anomaly-free test sets; anomalous periods formed a separate test set.
- Reconstruction Error-Based Detection: Anomalous periods produced higher reconstruction errors than normal periods, with five powersave periods and one performance period observed for node davide45.The method uses relative error magnitude rather than requiring the reconstruction error to be exactly zero.
- Detection Accuracy: The normal and anomalous error distributions were clearly separated, and F-score results were generally very good across the evaluated node groups.Performance-dominant nodes were harder to distinguish because their behavior was more similar to the normal condition.
- Detection Accuracy: The threshold classifier labels a point abnormal when its reconstruction error E_i exceeds θ, with θ chosen as a node-specific percentile of normal errors.For davide45, the 95th percentile gives θ = 0.082.
- Detection Accuracy: Higher thresholds generally performed best for powersave-dominant nodes, while performance-dominant nodes did not consistently benefit from increasing the threshold.For davide27 and davide28, n = 94 was empirically best because higher thresholds increased false negatives.
6 Conclusion
The paper presents an autoencoder-based approach that uses large datasets from a scalable monitoring framework to distinguish normal and anomalous HPC system states. Further validation across a broader anomaly set and real-time deployment remain future goals.
- The approach uses autoencoders to distinguish normal and anomalous HPC system states.
- The method relies on large datasets collected through a lightweight, scalable monitoring framework.
- The authors plan to validate the method on a broader set of anomalies.
- Future work includes classifying different anomaly types and sources, alongside an online prototype for real-time detection.