PaperlayerView exact revision on arXiv

Source-linked AI summary

TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training

Zhipeng Xia, Haotian Xu, Siyu Yun, Liqi Lin, Hu Liu, Yu Li, Cheng Zhuo

arXiv:2608.30769v1cs.LG

TL;DR

Silent data corruption can propagate through LLM training without immediate detection, while existing defenses lack detailed knowledge of computation-level vulnerability. The paper characterizes forward and backward fault propagation and introduces TrainSDC, which keeps training close to fault-free execution under sparse and dense faults.

  • Problem

    SDC vulnerability across major Transformer computations and its propagation through training remain insufficiently understood, limiting the design of targeted protections.

  • Method

    The paper systematically injects faults across major Transformer computations in both passes and uses the resulting characterization to combine Q/K recomputation, residual monitoring, and exponent-aware gradient scaling.

  • Results

    TrainSDC maintains training behavior close to fault-free execution on Llama 3.2-1B and Qwen3-0.6B under sparse and dense faults.

  • Takeaways & Limitations

    Forward and backward passes require different protection strategies because forward vulnerability is location-dependent while backward vulnerability varies less across components.

  • Takeaways & Limitations

    The fault model covers transient arithmetic computation errors but does not model persistent storage failures.

Abstract

from arXiv · show

LLM training is increasingly vulnerable to silent data corruption (SDC), yet existing protection methods largely treat Transformer computations uniformly because their vulnerability remains poorly understood. We present the first systematic characterization of SDC vulnerability across major computation interfaces in both the forward and backward passes of Transformer training. Our analysis reveals two distinct error propagation mechanisms: forward-pass vulnerability is highly location dependent, with faults on the Q/K path producing persistent training deviations, whereas backward-pass vulnerability is largely governed by gradient exponent distributions rather than computation locations. Motivated by these observations, we propose TrainSDC, a characterization-guided protection framework consisting of Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling. Experiments on Llama 3.2-1B and Qwen3-0.6B show that TrainSDC maintains training behavior close to fault-free execution under both sparse and dense fault injection while introducing only 1.65%-6.76% runtime overhead.

1 Introduction

The paper characterizes SDC vulnerability across Transformer computations and uses the findings to design TrainSDC, which protects forward and backward passes differently. TrainSDC keeps training close to fault-free behavior under sparse and dense faults.

  • Motivation: SDC can silently corrupt activations and gradients, allowing incorrect values to influence parameter updates without an immediate error signal.Its effects may appear later as loss spikes, failed convergence, or degraded final quality.
  • Characterization: The study provides a systematic characterization of SDC vulnerability across major Transformer modules in both forward and backward passes.Faults are injected at major component outputs and evaluated by their largest and remaining training effects.
  • Findings: Q/K-path faults cause more persistent forward-pass damage, while value, attention-output, and MLP faults cause larger but shorter-lived loss changes.This distinction separates faults that silently alter later optimization from disturbances attenuated by subsequent computation and updates.
  • Findings: Backward vulnerability varies less across components and depends on the exponent distribution of gradients.This motivates broad rather than module-specific backward protection.
  • TrainSDC: TrainSDC combines Q/K-operation recomputation, residual monitoring, flagged-step replay, and power-of-two loss scaling.The framework assigns different protection mechanisms to forward and backward computation.
  • Results: TrainSDC maintains outcomes close to fault-free training on Llama 3.2-1B and Qwen3-0.6B under sparse and dense fault injection.Under dense faults that make unprotected training diverge, TrainSDC preserves convergence with limited overhead.

2 Related Work

Prior SDC studies examine production behavior, controlled injections, or coarse system-level effects, but do not fully isolate Transformer computations. This paper compares major Transformer computations across forward and backward passes and uses that characterization for differentiated protection.

  • Prior characterization: Production measurements document SDC prevalence, while unhealthy-node experiments preserve realistic hardware behavior but do not control fault location or timing.They also do not separate individual computations within attention and feed-forward networks.
  • Prior characterization: Other studies analyze aggregate gradients, faulty devices, or recovery rather than local Transformer computations.These perspectives differ from computation-level vulnerability analysis.
  • Paper scope: This paper compares major Transformer computations separately in the forward and backward passes and uses the results to protect the two passes differently.TrainSDC avoids continuously duplicating the full training computation.

3 Fault Characterization

The characterization evaluates transient faults by location, propagation, and persistence across Transformer training. It finds path-dependent forward vulnerability but broadly distributed backward vulnerability, motivating heterogeneous protection.

  • Study design: The study traces whether transient computation errors are amplified, attenuated, or retained across module outputs and training stages.These observations provide the basis for the protection design.
  • Fault model: The study models transient arithmetic errors and excludes persistent storage failures.Transient errors affect only the current tensor computation and do not retain corrupted values or bit states across executions.
  • Fault model: The fault model targets activation outputs in the forward pass and corresponding output gradients in the backward pass, flipping four random bits in selected elements.Parameters, optimizer states, and stored training data are not modified.
  • Metrics: Normalized metrics distinguish the largest loss deviation during training from the deviation at the final step relative to clean-run variation.A value above one exceeds every observed clean-to-clean difference for the corresponding metric.
  • Forward results: Q/K-path outputs are the most vulnerable forward interfaces across both models, while V, normalization, MLP, and block outputs produce larger immediate but more attenuated deviations.Q/K faults instead yield modest immediate perturbations and the largest final loss deviations.
  • Backward results: Most monitored backward gradients produce similarly large training deviations, indicating weaker dependence on module location.Backward protection therefore requires broad coverage rather than module-specific protection.
  • Design implication: The protection design directly verifies weakly observable computations and uses lightweight monitoring where propagation signatures remain detectable.This follows the distinct propagation behavior of forward faults.
  • Propagation implications: Q/K faults are strongly attenuated after softmax and residual addition, making them weakly observable despite continued influence on parameter updates.V, normalization, attention-output, and MLP faults remain visible at residual writeback and are suitable for lightweight monitoring.

4 Protection Design

TrainSDC uses heterogeneous protection because forward-pass vulnerability depends strongly on computation location, whereas backward-pass vulnerability is driven mainly by gradient exponent behavior. It recomputes the Q/K path, monitors residual writebacks, and applies exponent-aware gradient scaling.

  • Forward Pass Protection: Forward protection directly verifies computations whose faults become weakly observable after propagation, while lightweight monitoring covers computations with detectable propagation signatures.This characterization-guided split avoids redundant execution where monitoring is sufficient.
  • Forward Pass Protection: TrainSDC recomputes the complete Q/K path, including projections, normalization, and rotary position encoding, because faults before attention can persist through training.The two executions use the same input and network parameters; a mismatch indicates an incorrect Q/K result under the transient fault model.
  • Forward Pass Protection: 128-bit XOR fingerprints compactly represent each Q or K tensor for comparing original and recomputed executions without retaining the full tensors.Indexed mixing reduces cancellation between changes at different tensor positions.
  • Forward Pass Protection: Residual-gain monitoring detects transient faults outside the Q/K path by measuring amplitude changes at residual writebacks and comparing worker ranks against current references.The guard reuses token-wise mean-square values from root mean square normalization and calibrates thresholds using recent finite cross-rank deviations.
  • Backward Pass Protection: Backward-pass vulnerability depends primarily on gradient magnitude and exponent distributions rather than Transformer module location.Exponent-bit flips can amplify small gradients, propagating into global parameter updates, while attenuation suppresses their contribution.
  • Selecting the Scaling Exponent: Exponent scaling shifts gradient exponent codes by k during backpropagation, then reverses the scaling before clipping and updating so the parameter update remains unchanged.The selected k minimizes predicted vulnerability within the finite numerical range, balancing amplification risk against non-finite upper-tail values.
  • Selecting the Scaling Exponent: ρ = 0.650 with pperm = 0.067 for Llama-3.2-1B and ρ = 0.867 with pperm = 0.005 for Qwen3-0.6B support the ranking of scaling candidates, with subsequent experiments fixing k = 15.Each network used n = 9 candidates.

5 Evaluation

The evaluation uses controlled transient-fault experiments, common training conditions, and component ablations to assess TrainSDC’s protection and overhead. TrainSDC preserves convergence and stays close to fault-free outcomes under sparse and dense faults, while its components provide complementary protection.

  • Experimental setup: The evaluation uses Llama 3.2-1B and Qwen3-0.6B from a shared step-1221 checkpoint through step 2035 on six NVIDIA A100 GPUs.Runs use random weights, a common document-disjoint SmolLM2-derived split, and approximately 100M training tokens per run.
  • Experimental setup: The protocol evaluates sparse and dense injections, with 10 or 100,000 corrupted activation elements per injection event.Methods share checkpoints, data order, injection schedules, and the unified evaluation metrics include Overhead, Dfinal, DMaximum, ∆PPL, Top-1, and W L2.
  • Protection configuration: TrainSDC applies Q/K-path recomputation, residual-gain monitoring, and exponent scaling with k = 15.The residual guard samples selected layers, while alarms discard accumulated gradients and replay microbatches before updating the network.
  • Main results: At 100,000 corrupted elements, unprotected training diverges, whereas TrainSDC preserves fault-free-consistent convergence with Dfinal of 6.32×10^-4 / 6.15×10^-4.The corresponding DMaximum is 9.14 × 10^-3 / 1.48 × 10^-2 and ∆PPL is 0.173 / 0.175 for Llama/Qwen.
  • Ablation study: Exponent scaling is the strongest individual component at 10 corrupted elements, reducing Dfinal to 3.03 × 10^-4 / 1.12 × 10^-3 and ∆PPL to 0.0728 / 0.250.Under strong faults, Q/K-only and residual-only produce Dfinal near 10^-1 and ∆PPL between 21.4 and 29.1, while the complete method achieves 6.32 × 10^-4 / 6.15 × 10^-4 Dfinal.

6 Conclusion

The paper identifies distinct forward- and backward-pass SDC vulnerabilities and uses those findings to build TrainSDC. On two language models, the framework mitigates sparse and dense faults with limited overhead and restores outcomes close to fault-free training.

  • Conclusion: Q/K-path faults cause more persistent damage, while value, attention-output, and MLP faults cause larger but shorter-lived loss changes.Backward vulnerability varies less across components and depends on the gradient exponent distribution.
  • Conclusion: TrainSDC combines Q/K-path recomputation, the Residual Gain Guard, and exponent scaling.These components correspond to the distinct forward- and backward-pass propagation findings.
  • Conclusion: TrainSDC mitigates sparse and dense faults on Llama 3.2-1B and Qwen3-0.6B with limited overhead and outcomes close to fault-free training.The conclusion reports this result across both evaluated models and fault regimes.

A.1 Robustness across Fault Rate, Layer, and Injection Budget

Additional SmolLM2-135M experiments vary fault-step rate, target layer, and injection budget independently. Across all three variants, forward sensitivity remains path-dependent: Q/K faults persist, while several other faults attenuate during training.

  • Experimental variants: The robustness study varies fault-step rate, target layer, and injection budget while keeping the main characterization’s remaining settings unchanged.Rate robustness uses Layer 20 and 0.1%; layer robustness uses Layer 10 and 1%; budget robustness corrupts exactly 1% of each target tensor.
  • Results: Across all three variants, Q/K-path faults consistently leave final deviations above the fault-free envelope.This pattern holds under changes in fault-step rate, target layer, and injection budget.
  • Results: V, MLP, and block-output faults mainly produce large peak deviations that are attenuated during subsequent training.Normalization outputs are also identified as vulnerable in the varied forward-pass experiments.

A.2 Operating Characteristics of Forward Protection

Forward protection combines calibrated residual-gain monitoring with Q/K-path recomputation, then aggregates alarms into replayed optimizer steps. With α = 2.5, it maintains low clean false-positive behavior while detecting protected Q/K corruptions and improving residual-guard recall as fault density increases.

  • Calibration and online state: The detector uses a 64-observation sliding window containing 320 rank deviations per monitored layer and path.The first 64 calibration steps seed the window, while all 100 calibration steps determine threshold floors; deployment begins without additional warm-up.
  • Threshold selection: α = 2.5 is selected as the smallest candidate with zero held-out clean alarms on both networks and no recall reduction.The sweep used 200 held-out fault-free steps and compared α values from 1 to 3.
  • Alarm handling: All rank, path, and layer decisions reduce to one optimizer-step alarm that triggers a single replay before the update.Rejected attempts are excluded from detector history, and clean deployment traces yield a combined step-level FPR of 0.061%.
  • Evaluation protocol: The complete forward-protection evaluation reports clean false-positive rate and alarm recall across fault runs containing 403 valid injection events.Missed forward alarms are further analyzed by composition across both networks and three spatial intensities.
  • Detection behavior: Every direct corruption of protected Q/K-path targets is detected, while residual-guard recall rises with corruption density.Combined recall increases from 43.42% to 86.35% for Llama and from 40.94% to 78.41% for Qwen.

A.3 Temporal Stability of the Exponent-Scaling Parameter

Exponent scaling is evaluated across the full Pythia-1B checkpoint trajectory to determine whether a selected exponent can remain fixed during training. After warm-up, long admissible intervals make only one later reassessment necessary.

  • Selection criterion: The exponent is selected from clean backward-gradient exponent histograms and must remain admissible over a substantial training interval.A held value remains admissible while its predicted risk is at most 25% above the pointwise minimum.
  • Checkpoint sweep: The checkpoint sweep covers all 154 released Pythia-1B checkpoints from step 0 through step 143,000 without fault injection.Profiling uses WikiText-103 validation sequences and BF16 exponent histograms collected at all 64 Transformer linear projections.
  • Temporal stability: After k = 15 is selected at step 4,000, it remains admissible through step 78,000, spanning 74,000 training steps.The relative excess risk stays below the 25% limit during this interval.
  • Temporal stability: One reassessment at step 79,000 selects k = 13, which remains admissible through the final checkpoint at step 143,000.Thus, the trajectory after the initial post-warm-up operating point requires only one additional reassessment.
Loading 2608.30769v1…