Source-linked AI summary
6Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Models
Rundong Su, Jintao Zhang, Zhihang Yuan, Haojie Duanmu, Jianfei Chen, Jun Zhu
TL;DR
Video Diffusion Transformers are expensive to deploy, and static quantization overlooks activation sensitivity that changes across denoising timesteps. 6Bit-Diffusion combines dynamic NVFP4/INT8 allocation with Temporal Delta Cache to reduce these costs, achieving 1.92× acceleration and 3.32× memory reduction while maintaining original video quality.
Problem
Video Diffusion Transformers require high memory and computational cost, while static bit-width policies overlook timestep-varying activation quantization sensitivity.
Method
The framework dynamically assigns NVFP4 to temporally stable layers and INT8 to sensitive layers using block differences, while Temporal Delta Cache skips redundant computations across adjacent timesteps.
Results
1.92× inference speedup and 3.32× memory reduction are achieved on CogVideoX while maintaining the original video quality.
Takeaways & Limitations
Dynamic precision allocation and temporal caching enable aggressive, training-free Video DiT inference without compromising generation quality.
Abstract
from arXiv · showhide
Diffusion transformers have demonstrated remarkable capabilities in generating videos. However, their practical deployment is severely constrained by high memory usage and computational cost. Post-Training Quantization provides a practical way to reduce memory usage and boost computation speed. Existing quantization methods typically apply a static bit-width allocation, overlooking the quantization difficulty of activations across diffusion timesteps, leading to a suboptimal trade-off between efficiency and quality. In this paper, we propose a inference time NVFP4/INT8 Mixed-Precision Quantization framework. We find a strong linear correlation between a block's input-output difference and the quantization sensitivity of its internal linear layers. Based on this insight, we design a lightweight predictor that dynamically allocates NVFP4 to temporally stable layers to maximize memory compression, while selectively preserving INT8 for volatile layers to ensure robustness. This adaptive precision strategy enables aggressive quantization without compromising generation quality. Beside this, we observe that the residual between the input and output of a Transformer block exhibits high temporal consistency across timesteps. Leveraging this temporal redundancy, we introduce Temporal Delta Cache (TDC) to skip computations for these invariant blocks, further reducing the computational cost. Extensive experiments demonstrate that our method achieves 1.92$\times$ end-to-end acceleration and 3.32$\times$ memory reduction, setting a new baseline for efficient inference in Video DiTs.
1 Introduction
Video DiTs offer strong generation quality but remain costly to deploy because of high memory and computation demands. 6Bit-Diffusion addresses timestep-varying quantization sensitivity with dynamic precision allocation and temporal caching, achieving faster, lower-memory inference while preserving comparable quality.
- Motivation: Large video DiTs incur substantial deployment overhead, including out-of-memory failures on consumer devices and 22-minute generation for a 49-frame 1080p video.The 22-minute measurement is reported for CogVideoX with two billion parameters on an NVIDIA RTX-5090.
- Motivation: Static quantization policies are limited because activation sensitivity changes drastically across denoising timesteps, causing precision loss or wasted compression.Uniform quantization uses one bit-width, while static mixed-precision assignments remain fixed during inference.
- Dynamic precision allocation: A block’s previous-timestep input-output difference linearly predicts current-timestep sensitivity, enabling INT8 for volatile blocks and NVFP4 for stable ones.The predictor dynamically estimates precision requirements online with minimal overhead.
- Temporal caching: Temporal Delta Cache reuses highly consistent residual updates across adjacent timesteps to skip redundant Transformer-block computations without degrading video quality.TDC complements dynamic mixed-precision quantization as part of the unified acceleration framework.
- Results: 1.92× speedup and 3.32× GPU memory reduction are achieved on CogVideoX while preserving comparable video quality to the full-precision baseline.The framework combines Dynamic Mixed-Precision Quantization with Temporal Delta Cache.
2 Related Works
Related work reduces DiT costs through post-training quantization and feature caching. Existing approaches include smoothing or rotating quantization difficulty and using fixed or dynamically selected computation-skipping schedules.
- Diffusion Transformers: Diffusion Transformers replaced U-Nets as a standard visual-generation backbone, using self-attention to capture long-range dependencies and structural relationships.Examples include Open-Sora, CogVideoX, and HunyuanVideo.
- Post-Training Quantization: Post-training quantization compresses weights and activations without retraining, while SmoothQuant and QuaRot address activation outliers through scaling or randomized Hadamard rotations.These methods target memory-footprint and computation reductions for DiTs and related architectures.
- Feature Caching: Feature-caching methods exploit temporal redundancy by reusing cached features or attention states, progressing from fixed schedules to step-dependent skipping decisions.DeepCache and FORA use fixed schedules, whereas AdaCache and TeaCache make more dynamic decisions based on feature changes or timestep embeddings.
3 Preliminaries of Quantization
Quantization maps full-precision tensors to discrete low-bit representations and reconstructs them through scale and zero-point parameters. The preliminaries describe general quantization, dequantization, and NVFP4’s block-scaled FP4 representation.
- General quantization: Quantization reduces memory footprints and computational overhead by compressing full-precision weights and activations into low-bit formats.The general formulation starts from a full-precision tensor X.
- General quantization: The general process quantizes X using a scale and zero-point within a target representable range, then dequantizes it as X̂ = (Xq − z) · s.Xq and X̂ are the quantized and dequantized tensors; s is the scaling factor and z is the zero-point.
- NVFP4 quantization: NVFP4 represents values with an E2M1 FP4 format and a shared FP8 scaling factor for contiguous blocks, with maximum representable value 6.0.The cited formulation uses s = max(|X|)/6.0 and applies CastToFP4 before dequantization.
4 Methods
The framework dynamically routes activation precision using block-level temporal differences and skips computations when block updates are temporally redundant. It combines DMPQ’s sensitivity-aware NVFP4/INT8 assignment with TDC’s error-guided caching.
- Dynamic Mixed-Precision Quantization: DMPQ predicts each internal linear layer’s current quantization sensitivity from the preceding timestep’s block input-output relative L1 difference.The layer-wise relative quantization error is modeled as a linear function of the block transformation magnitude, with coefficients fitted offline on calibration data.
- Dynamic Mixed-Precision Quantization: DMPQ measures block transformation magnitude as the relative L1 distance between its preceding-timestep input Xt−1 and output Yt−1.The relative quantization error Erel is a normalized L2 distance between full-precision and selectively quantized block outputs.
- Dynamic Mixed-Precision Quantization: Large block differences route activations to INT8, while stable blocks use NVFP4 to reduce memory while preserving robustness.Inference compares Γt−1 with a layer-specific threshold τΓ derived from an acceptable relative error threshold.
- Temporal Delta Cache: TDC predicts current update stability from the similarity of the previous two residual updates and reuses cached deltas to skip redundant block computation.The block forward pass uses the unified text-and-visual input plus a residual delta when computation is skipped.
- Temporal Delta Cache: Residual updates remain strongly temporally consistent across most of the diffusion process, but their redundancy varies across transformer layers.This layer-dependent consistency motivates adaptive rather than uniformly scheduled caching.
- Temporal Delta Cache: TDC uses accumulated prediction error to refresh cached updates and limit drift during continuous skipping.The mechanism tracks the last fully computed timestep, adds a penalty during repeated skips, and evaluates cache viability using a global threshold.
5 Experiments
Experiments on CogVideoX and standard video-generation benchmarks show that the framework preserves quality under aggressive quantization while reducing memory and inference cost. Ablations further isolate the benefits of DMPQ, TDC, and PDR.
- Quantitative Results: DMPQ achieves Aesthetic Quality 0.5437 on CogVideoX-2B under W4A6, matching or exceeding cited W4A8 static baselines.The cited W4A8 values are 0.5332 for both SmoothQuant and ViDiT-Q.
- Quantitative Results: DMPQ reaches Overall Consistency 0.2474 on the 2B model and 0.2492 on the 5B model while protecting volatile layers with INT8.Stable layers are compressed to NVFP4 based on temporal instability.
- Qualitative Results: Qualitative comparisons show that DMPQ with TDC and PDR preserves fine details and global spatiotemporal consistency at W4A6, unlike several PTQ baselines.The baselines exhibit hallucinations, semantic drift, or local geometric distortions.
- Efficiency: TDC with DMPQ treats skipped blocks as 0-bit activations, yielding an average activation bit-width of approximately 6 bits.The reported average is 6.035 bits.
- Efficiency: The framework reduces memory by 3.32× and achieves 1.36× end-to-end speedup after fused CUDA kernels make quantization overhead practically negligible.The broader conclusion reports 1.92× acceleration for the complete method.
- Ablation Study: DMPQ raises VQA-A to 75.5332, while PDR restores Flow Score to 5.5417 and reaches VQA-A 76.1173 after naive TDC-DMPQ causes error drift.The ablation compares DMPQ, TDC, and PDR across visual, semantic, and temporal metrics.
6 Conclusion
6Bit-Diffusion combines dynamic mixed-precision quantization, temporal delta caching, and purified delta refresh to reduce Video DiT inference costs. On CogVideoX, it reports 1.92× acceleration and 3.32× lower memory usage while maintaining original video quality.
- Conclusion: 6Bit-Diffusion dynamically assigns NVFP4 to stable layers, INT8 to sensitive layers, skips redundant block computations, and refreshes cached updates to limit accumulated quantization errors.Its components are DMPQ, TDC, and PDR.
- Conclusion: 1.92x inference acceleration and 3.32x memory reduction are reported on CogVideoX while maintaining the original video quality.These are the paper's headline efficiency results.