Source-linked AI summary
When Log Parsing Removes What Matters: Detecting Failures in Distributed GPU Workloads
Boyuan Cui, Yuyang Zeng, Yifang Tian, Hans-Arno Jacobsen
TL;DR
Distributed-GPU failure detection lacks a common basis for evaluating established methods and their operating costs. The paper introduces GF-Bench and a labeled dataset, then finds that parsed logs yield limited coverage of both fault classes while raw logs improve ICL-LogPrompt’s score at higher cost.
Problem
There is no common framework and dataset for measuring how log-anomaly detectors perform on controlled distributed-GPU failures and what they cost to run.
Source: §1 · pp. 1–2Method
GF-Bench generates controlled multi-GPU failures and collects observability data, which the paper uses to evaluate established detectors on parsed logs and LogPrompt on raw logs.
Source: §1 · p. 2Results
Parsed-log methods do not reliably detect both failure classes and reach at most 0.667 overall F1; raw-log ICL-LogPrompt reaches 0.983 at 9.0× input-token usage and 8.9× per-run cost.
Source: §3.2 · pp. 3–4, §4 · p. 4Takeaways & Limitations
Drain-based parsing improves efficiency but can discard information that affects distributed-failure detection.
Source: §3.2 · p. 4The findings are initial, in-distribution evidence from one two-A40-GPU configuration, two injected fault classes, NCCL logs, and Drain as the only parser.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Large AI models increasingly rely on distributed GPU execution, yet it remains unclear whether established anomaly detection methods transfer effectively to this setting. We introduce GF-Bench, a PyTorch-based framework for generating controlled failures in distributed GPU workloads, and a dataset produced with it. The framework supports extensible workload and fault-injection logic; based on this design, the current release targets failures visible in multi-GPU communication logs and provides 220 labeled two-GPU runs covering normal behavior, CUDA out-of-memory fail-stop faults, and persistent-straggler fail-slow faults. Using GF-Bench, we evaluate three sequence-based methods (DeepLog, LogAnomaly, and LogBERT) and the large language model (LLM)-based LogPrompt with in-context learning (ICL) and chain-of-thought (CoT) prompting strategies on parsed logs, and additionally evaluate LogPrompt on raw logs. On parsed inputs, no method reliably detects both failure classes, and the highest overall F1 score is 0.667. Raw inputs raise the overall F1 score of ICL-LogPrompt to 0.983, at 9.0x its average input-token usage and 8.9x its per-run cost. These results reveal a trade-off between effectiveness and efficiency: parsed inputs reduce LLM token use and inference cost but can discard the distributed execution context required for accurate failure detection. GF-Bench provides a reproducible basis for studying cost-efficient anomaly detection methods and can be extended to additional GPU workloads and failure types.
1 Introduction
The paper addresses the lack of a common basis for generating labeled failures in distributed GPU workloads and evaluating detectors on them. It introduces GF-Bench and a 220-run dataset, then uses them to compare detection methods and the costs of parsed versus raw logs.
1 Introduction
- Existing work lacks a framework for producing controlled, labeled distributed-GPU failures and measuring detector effectiveness and operating cost on them.Prior work offers general log-anomaly detection or low-level GPU fault injection, but not an end-to-end bridge between them.Source: §1 · pp. 1–2
- GF-Bench executes configurable multi-GPU workloads, injects faults, and collects logs, with workload, injection, and collection components designed to be extended independently.The framework is built on PyTorch and initially targets NCCL-visible failures.Source: §1 · p. 2
- The released dataset contains 220 labeled runs covering normal behavior, CUDA OOM fail-stop faults, and persistent-straggler fail-slow faults.These fault types instantiate two distinct failure modes in distributed training.Source: §1 · p. 2
- The evaluation compares four log-anomaly detection methods on parsed logs and measures how removing parsing changes detection effectiveness and token and monetary costs.LogPrompt is also evaluated on raw logs.Source: §1 · p. 2
2 GF-Bench Design
GF-Bench represents each run through its collected observability data and evaluates detectors on both predictive quality and operational cost. Its design combines configurable multi-GPU execution, fault injection, and automated data collection.
2 GF-Bench Design
- Each run’s observability data is transformed by a representation strategy into detector input; the run label distinguishes normal execution from one of K failure classes and induces a binary anomaly label.The collected data may include logs, metrics, and traces from each GPU and its associated process.Source: §2 · p. 2
- The framework launches a supervising driver and one GPU-bound process per GPU, supporting customizable workloads with varied computation and communication patterns.The described training simulation combines tensor computations with collective communication such as AllReduce.Source: §2 · p. 2
- A workload’s fault-injection interface pairs a condition checker, which determines when a fault triggers, with an injection function that implements it.A customizable global timeout terminates unfinished processes so runs reach a recorded outcome.Source: §2 · p. 2
- The collection pipeline captures workload outputs, including standard streams and log files, in a designated directory for each experiment run.NCCL logs can be collected through the NCCL_DEBUG_FILE environment variable.Source: §2 · p. 2
- New workloads and failure scenarios can be added through custom workload logic and a fault-injection interface; fault timing can vary by changing the condition checker.The same injection function can be reused for variants such as triggering OOM at different training iterations.Source: §2 · p. 2
3 Evaluation and Results
The evaluation tests established anomaly detectors on Drain-parsed NCCL logs and LogPrompt on both parsed and raw logs, using a two-GPU dataset with normal, OOM, and straggler runs. Parsed methods do not reliably detect both fault classes; raw logs improve ICL-LogPrompt’s F1 while increasing token use and cost.
3.1 Experimental Setup
- The dataset covers normal execution, CUDA OOM fail-stop faults, and persistent-straggler fail-slow faults in a simulated data-parallel workload.Each iteration performs tensor computations and a delay, then an AllReduce; OOM and straggler faults are injected at a designated rank.Source: §3.1 · p. 3
- The 220-run evaluation uses a held-out half of the dataset; sequence models train on normal runs, while ICL-LogPrompt receives one labeled demonstration per class.Per-rank logs are concatenated in ascending rank order because the logs lack timestamps.Source: §3.1 · p. 3
3.2 Results
- 0.667 is the highest overall F1 on parsed inputs, but ICL-LogPrompt misses every OOM run while detecting every straggler.CoT-LogPrompt detects no stragglers; DeepLog and LogAnomaly predict every test run as normal.Source: Table 2 · p. 3, §3.2 · p. 3
- Raw input raises CoT-LogPrompt’s overall F1 from 0.418 to 0.547 and its false positive rate to 0.98.With raw logs, CoT begins detecting stragglers.Source: §3.2 · p. 3, Table 3 · p. 3
- Raw logs let ICL-LogPrompt identify an OOM run from rank 1’s missing Abort COMPLETE lines, a pattern it misses on parsed input.Drain maps Abort COMPLETE messages from different ranks to the same template by treating source-rank fields as variables.Source: §3.2 · p. 3, Figure 2 · p. 4, Figure 3 · p. 4
- 0.983 overall F1 for raw-input ICL-LogPrompt, versus 0.667 on parsed logs; OOM and straggler F1 both reach 0.983.Raw inputs also raise ICL input-token usage 9.0× and per-run cost 8.9×.Source: Table 3 · p. 3, §3.2 · p. 3
- Raw input uses 9.0× as many input tokens and costs 8.9× as much per run for ICL-LogPrompt as parsed input.Average ICL latency rises from 1.012 s to 1.943 s.Source: Table 4 · p. 4, §3.2 · p. 4
4 Discussion
The reported findings are initial, in-distribution evidence from one two-A40 configuration, two injected fault classes, NCCL logs, and Drain parsing. The conclusion reports poor transfer to parsed NCCL logs and a raw-log effectiveness–cost trade-off.
4 Discussion
- The evaluation is limited to two A40 GPUs, two injected fault classes, NCCL logs, Drain parsing, and one ICL demonstration per class.The sequence models do not receive demonstrations; results may also depend on rank-ordered concatenation, demonstration choice, GPT-5.4, and LLM nondeterminism.Source: §4 · p. 4
- On Drain-parsed logs, no method reliably detects both failure classes, and the highest overall F1 is 0.667.Raw logs raise ICL-LogPrompt’s F1 to 0.983, with 9.0× input-token usage and 8.9× per-run cost.Source: §4 · p. 4
- Additional parsers, observability sources, fault mechanisms, and cross-environment replication are needed to establish generality.The parsing stage’s overhead and false positive rate under operational prevalence were not measured.Source: §4 · p. 4