PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Resilience of Text-to-Video Diffusion Models to Hardware Faults

Zachary Coalson, A M Aahad, Stella Doehring, Zane Ma, Sanghyun Hong

arXiv:2608.29598v1cs.LG

TL;DR

T2V diffusion models face unexplored resilience risks from random bitwise hardware faults, whose effects may differ from previously studied architectures. This paper systematically injects memory and computational faults into three T2V models and finds modest aggregate degradation but potentially substantial semantic and visible-output changes.

  • Problem

    The resilience of T2V diffusion models to random bitwise hardware faults remains unexplored, despite GPU faults propagating into model weights or activations and T2V's iterative and spatiotemporal dependencies.

  • Method

    The study systematically injects 2-bit memory faults and 1-bit computational faults during inference across three T2V models and evaluates 16 video-quality and semantic-correctness metrics.

  • Results

    Aggregate degradation is modest, but semantic correctness is more affected than visual quality, memory faults are more damaging than computational faults, and 7–28% of faults produce visible alterations.

  • Takeaways & Limitations

    Single random hardware faults can alter T2V output semantics, revealing reliability risks for deployed T2V systems.

  • Takeaways & Limitations

    The study considers only 2-bit memory and 1-bit computational faults and focuses on random faults, while other fault models and targeted adversarial effects remain open directions.

Abstract

from arXiv · show

We present the first systematic study of the resilience of text-to-video (T2V) diffusion models under random hardware-level faults. While T2V models are widely used for automated video generation due to their ability to produce high-quality, temporally coherent, and realistic videos, their iterative denoising process and spatiotemporal dependencies introduce unique failure modes. We perform an extensive fault-injection study covering both computational and memory faults across three T2V models and a representative benchmark. Our results show that (1) a single fault can degrade overall performance by up to 3.7\%, with semantic correctness more affected than perceptual quality; (2) memory faults are more damaging than computational faults, high-order exponent bits are particularly vulnerable, and the widely-used bfloat16 is more susceptible than alternative formats; and (3) 7-28\% of faults cause visible artifacts, including semantic changes such as added objects, suggesting that single faults are sufficient to alter output semantics. Our findings reveal reliability risks in deployed T2V systems and motivate further research on improving fault resilience. Code: \href{https://github.com/ztcoalson/T2V-Resilience}{https://github.com/ztcoalson/T2V-Resilience}.

1 Introduction

T2V diffusion models face distinct hardware-fault risks because iterative denoising, spatiotemporal dependencies, and backbone reuse can propagate perturbations. This study introduces a systematic fault-injection evaluation and finds modest aggregate degradation but substantially greater semantic vulnerability.

  • Motivation: T2V models generate coherent videos through iterative denoising, while deployment on GPUs creates exposure to random bitwise faults.The DiT backbone uses billions of parameters and incurs substantial compute and memory overhead.
  • Research gap: The resilience of T2V models remains unexplored because diffusion inference, spatiotemporal dependencies, and continuous output quality distinguish them from prior fault-studied architectures.
  • Study design: The study evaluates memory and computational faults across three T2V models, 16 individual metrics, 3 aggregate metrics, more than 300,000 faults, and 550 prompts.
  • Key findings: 0.3–3.7% aggregate performance degradation occurs under random bitwise faults, while semantic correctness degrades up to 6.6× more than visual quality.
  • Key findings: 2.4–8.6× greater impact comes from 2-bit memory faults than 1-bit computational faults because memory corruption persists.
  • Key findings: The most significant exponent bit is disproportionately vulnerable, whereas sensitivity is largely uniform across Transformer blocks and layer types.
  • Key findings: bfloat16 is more susceptible to faults than alternative numerical representations.
  • Key findings: 7–28% of faults produce visible semantic alterations or severe distortions, indicating that individual faults can substantially change output semantics.

2 Background and Related Work

T2V diffusion extends image diffusion into the spatiotemporal domain, using iterative latent denoising to generate coherent videos. Its DiT backbone is reused across denoising steps, motivating targeted resilience analysis alongside prior work on other architectures.

  • Background: T2V models generate coherent video sequences by iteratively denoising representations in a compressed latent space.
  • Background: A typical architecture combines a text encoder, spatiotemporal DiT backbone, and VAE decoder, with the DiT reusing weights across T denoising steps.
  • Related work: Prior fault-resilience studies primarily examined classification networks and later Transformer or language-model inference.

3 Our Fault Injection Framework

The framework models uncorrected memory and computational faults during inference by injecting bit flips into the DiT backbone. It evaluates three representative models on VBench using broad quality and semantic metrics.

  • Fault models: The study models 2-bit memory faults in weights and 1-bit computational faults in activations, reflecting faults that escape or are not corrected by modern GPU protections.
  • Fault setting: Faults are assumed to occur once per inference and are injected only into the DiT generation process, excluding the text encoder and VAE.
  • Injection procedure: Memory faults flip two uniformly sampled weight bits, whereas computational faults flip one output bit at a sampled layer and diffusion timestep using a PyTorch hook.
  • Models and benchmark: The evaluation covers OpenSora, CogVideoX-2B, and Latte, whose DiT backbones range from 1B–2B parameters.
  • Models and benchmark: VBench supplies 550 prompts across 11 categories for evaluating generated-video quality and semantic fidelity.
  • Metrics: Sixteen VBench metrics are summarized by Quality, Semantic, and Total Scores.

4 Empirical Evaluation

Across three T2V models, random hardware faults modestly reduce aggregate scores but affect semantic correctness more than visual quality. Memory faults are substantially more damaging than computational faults, while fault timing shows little systematic effect and visible artifacts remain possible.

  • Main resilience results: Total Score decreases by 0.3–3.7% across fault models and T2V models.Performance drops are similar across models, within approximately 1%.
  • Main resilience results: Semantic Score drops by 1.1–8.8%, compared with 0.2–2.5% for Quality Score.Semantics are affected up to 6.6× more than quality, indicating altered entity attributes and relationships rather than only visual rendering changes.
  • Main resilience results: 2-bit memory faults are 2.4–8.6× more impactful than 1-bit computational faults.The reported drops are 1.4–8.8% for memory faults versus 0.2–2.8% for computational faults, likely because memory corruption persists across denoising steps.
  • Resilience analysis: Computational fault effects vary from -3.6% to +2.6% across normalized diffusion timesteps, with no clear trend as timesteps increase.Together with bit-position results, this suggests fault magnitude matters more than location or timing.
  • Numerical representation: bfloat16 is most vulnerable under 2-bit memory faults, with Total Score drops of 3.1% versus 1.8% for float32 and 1.2% for float16.The authors attribute this pattern to bfloat16 allocating 8 of 16 bits to the exponent while retaining float32’s range.
  • Qualitative analysis: 28% of 2-bit memory faults cause nonminor visual changes, compared with 7% of 1-bit computational faults.Most faults produce minor changes, but major alterations and complete distortion also occur; examples include added objects, changed ground tiling, and static.

5 Discussion

The discussion prioritizes targeted protections and fault-mitigation strategies for the most damaging bit flips, while identifying broader fault models, adversarial faults, and better evaluation metrics as open needs.

  • Potential countermeasures: Selective protection of high-order exponent bits is motivated because they account for most degradation, avoiding the full cost of ECC or triplication.The discussion also considers range-based clipping and restart sampling as lower-cost mitigation strategies with stated trade-offs.
  • Limitations and future work: The study covers only 2-bit memory and 1-bit computational faults, leaving other practical fault models for future investigation.Its random-fault design characterizes expected naturally occurring behavior but does not establish whether targeted adversarial effects can be induced.
  • Limitations and future work: VBench metrics may understate visible fault effects, motivating metrics better suited to quantifying fault impact.This limitation follows from the qualitative analysis showing visible failures despite modest quantitative degradation.

6 Conclusion

The study finds that T2V diffusion models are vulnerable to random bitwise hardware faults, with modest aggregate degradation masking semantic and visual alterations. An open-source fault-injection framework supports further resilience research.

  • At most 3.7% aggregate degradation masks greater effects on semantic correctness than visual quality, with up to 28% of faults producing visible alterations.Reported alterations include added objects or distortion that standard metrics may not capture.
  • The released fault-injection framework enables further study of T2V resilience and evaluation of future defenses.

A Detailed Experimental Setup

The experiments use VBench to examine fault impacts on individual video-quality and semantic metrics, reporting averages over 100 trials for fault-free, memory-fault, and computational-fault conditions. Effects vary substantially across metrics, with semantic measures generally less resilient than quality measures.

  • The experiments use Python 3.10.19, PyTorch 2.2.2, the official VBench repository, and a system with 8 NVIDIA A40 GPUs.
  • Figure 8 compares fault-free, 2-bit memory-fault, and 1-bit computational-fault conditions using average scores over 100 trials.The reported 95% confidence intervals are within ±0.95.
  • 14.2–23.7% degradation affects Multiple Objects, Human Action, and Object Class, whereas Subject and Background Consistency change by -0.3% to +1.4%.
  • Semantic metrics change by -23.7% to +11.2%, while quality metrics change by -8.5% to +4.5%.The passage notes substantial variation across individual metrics.

C Full Resilience Analysis Results

OpenSora and CogVideoX show resilience patterns broadly consistent with Latte: the most significant exponent bit is disproportionately vulnerable, while vulnerability is largely uniform across Transformer blocks and layer types. OpenSora also exhibits higher variance and comparatively greater attention-layer resilience.

  • The most significant exponent bit is disproportionately vulnerable across OpenSora and CogVideoX, while vulnerability is largely uniform across Transformer blocks and layer types.
  • Figures 9 and 10 report average changes across VBench metrics by maximum bit location, Transformer block index, and layer type for OpenSora and CogVideoX.
  • OpenSora shows positive average changes up to +3.9% at bit positions other than the most significant exponent bit.Higher per-trial variance and wider confidence intervals make these signs less reliable as systematic effects.
  • OpenSora attention layers are more resilient than MLPs, with average changes of -0.7% and -2.7%, respectively.

D VBench Metric Definitions

VBench measures visual quality, temporal behavior, semantic fidelity, and prompt alignment through independently normalized metrics. Aggregate scores summarize quality, semantic correctness, and their weighted combination.

  • Metric families: Temporal metrics assess background stability, flickering, motion smoothness, and dynamic degree.
  • Metric families: Quality metrics assess aesthetic appeal and low-level imaging fidelity, while Overall Consistency captures holistic prompt alignment and stylistic coherence.
  • Metric families: Semantic metrics assess object composition, human actions, colors, spatial relationships, scenes, styles, overall consistency, and object classes.
  • Metric framework: VBench metrics are independently computed per video, normalized to [0, 1], and reported as percentages using empirical minimum and maximum reference values.
  • Aggregate metrics: Quality Score is a weighted average of seven metrics, while Semantic Score is a weighted average of nine prompt-level correctness metrics.
  • Aggregate metrics: Total Score combines Quality and Semantic Scores with 4× greater emphasis on quality.
  • Metric framework: The metric suite enables coarse- and fine-grained analysis of how faults affect video generation.
Loading 2608.29598v1…