Source-linked AI summary
Q-ARVD: Quantizing Autoregressive Video Diffusion Models
Siao Tang, Xinyin Ma, Gongfan Fang, Xingyi Yang, Xinchao Wang
TL;DR
ARVD quantization remains challenging because autoregressive error accumulation creates highly uneven frame sensitivity and heterogeneous weight outliers. Q-ARVD addresses these issues with final-quality frame weighting and adaptive dual-scale quantization. Experiments report near-lossless visual quality, a 1.97× model-size reduction, and a 1.30× latency speedup for INT8 deployment.
Problem
ARVD quantization is underexplored, while existing methods for bidirectional diffusion transformers often yield suboptimal performance.
Method
Q-ARVD combines final-quality guided frame weighting with outlier-aware adaptive dual-scale quantization.
Results
1.97× model-size reduction and 1.30× latency speedup are achieved by the practical INT8 model, with near-lossless visual quality.
Takeaways & Limitations
Q-ARVD provides a quantization framework tailored to autoregressive video diffusion models and maintains high video quality across the temporal span.
Takeaways & Limitations
The identified exponential-like frame sensitivity is currently used only during quantization reconstruction, while manually optimized CUDA kernels may provide further efficiency gains.
Abstract
from arXiv · showhide
Autoregressive video diffusion models (ARVDs) have emerged as a promising architecture for streaming video generation, paving the way for real-time interactive video generation and world modeling. Despite their potential, the substantial inference cost of ARVDs remains a major obstacle to practical deployment, making model quantization a natural direction for improving efficiency. However, quantization for ARVDs remains largely unexplored. Our empirical analysis shows that directly applying existing quantization schemes developed for standard diffusion transformers to ARVDs leads to suboptimal performance, revealing quantization behaviors that differ from those observed in bidirectional diffusion models. In this paper, we identify two critical challenges in quantizing ARVDs: (C1) Highly unbalanced frame-wise quantization sensitivity. Error accumulation during autoregressive generation can induce severely skewed quantization sensitivity across frames, following an exponential-like decay pattern. (C2) Prominent and heterogeneous outlier patterns in weights. Weight distributions exhibit pronounced outlier channels, whose patterns vary substantially across layer types and block depths. To address these issues, we propose Q-ARVD, a novel framework for accurate ARVD quantization. (S1) To tackle the highly unbalanced frame-wise sensitivity, Q-ARVD incorporates a final-quality aware frame-weighting mechanism into the quantization objective. (S2) To prevent heterogeneous outliers from degrading performance, Q-ARVD introduces an outlier-aware adaptive dual-scale quantization, which automatically detects the presence and quantity of outlier channels for an arbitrary layer, and isolates them to protect normal channels. Extensive experiments demonstrate the superiority of Q-ARVD.
1 Introduction
ARVD quantization is important for efficient streaming deployment but differs from bidirectional diffusion quantization. Q-ARVD addresses skewed frame sensitivity and heterogeneous weight outliers with specialized mechanisms.
- Existing quantization methods often perform suboptimally on ARVDs because autoregressive generation differs from bidirectional diffusion.
- Autoregressive error accumulation makes early frames disproportionately important, with frame-wise sensitivity following an exponential-like temporal decay.
- Q-ARVD uses final-quality guided frame weighting to reflect how quantizing each frame affects the overall generated video.
- Q-ARVD combines adaptive dual-scale quantization with frame weighting to address heterogeneous outliers and sensitivity discrepancies.
- 1.97× model-size reduction and 1.30× latency speedup are reported for the practical INT8 deployment.
2 Related Works
This section reviews the shift toward autoregressive video diffusion for streaming generation and outlines model quantization as a general approach to improving inference efficiency.
- Autoregressive video diffusion: Autoregressive video diffusion models generate frames or chunks sequentially, conditioning each new output on previously generated frames.Their generation process parallels causal decoding in large language models.
- Autoregressive video diffusion: Early autoregressive video diffusion models use multi-step denoising and therefore face high inference latency.Later work improves efficiency and quality through few-step distillation, exposure-bias mitigation, and teacher-student architecture alignment.
- Autoregressive video diffusion: Advances in efficiency, quality, and long-horizon generation make autoregressive video diffusion suitable for streaming video generation, real-time interaction, and world modeling.Recent research extends fixed-length autoregressive models to long-horizon generation.
- Model quantization: Quantization improves inference efficiency by representing model weights and input activations with low-precision formats such as INT4 or INT8.The process uses scaling factors and zero-points to map values into a low-precision representation.
- Model quantization: Calibration-based reconstruction optimizes quantization parameters to better preserve model performance.The reconstruction uses activations and weights from a calibration dataset, with learnable parameters including scaling factors and rounding schemes.
- Quantization for diffusion models: Quantization research has progressed from diffusion-model UNets, with temporal-denoising adaptations, toward diffusion transformers.The cited works reflect this architectural transition in diffusion-model quantization.
3 Method: Q-ARVD
Q-ARVD addresses ARVD quantization through final-quality guided frame weighting and adaptive dual-scale handling of heterogeneous weight outliers. The framework measures frame sensitivity by final video-quality degradation and detects outlier channels automatically before separating their quantization treatment.
- Q-ARVD combines final-quality guided frame weighting with outlier-aware adaptive dual-scale quantization.These mechanisms target unbalanced frame-wise sensitivity and heterogeneous outlier patterns, respectively.
- Final-quality Guided Frame-weighting: Earlier frames have greater impact on overall video quality because autoregressive quantization errors propagate through subsequent frame generation.The generation of each frame is conditioned on previous frames, so errors in earlier frames can undermine later frames.
- Final-quality Guided Frame-weighting: Frame sensitivity α_i is quantified by the degradation in final video quality when only the i-th frame is quantized.The method compares the original autoregressive video with a process in which quantization is enabled for the selected frame.
- Final-quality Guided Frame-weighting: Sensitivity varies sharply across chunks, with self-forcing W8A8 scoring 0.70 for chunk 1 and less than 0.01 for the last chunk.The measured pattern exhibits an exponential-like decay across the temporal axis.
- Outlier-aware Adaptive Dual-scale Quantization: Outlier channels form a small subset with much larger magnitudes than normal channels, and their prevalence varies across layer types and block depths.The second FFN layers are prominent, whereas cross-attention value projections have smooth distributions.
4 Experiments
Experiments evaluate Q-ARVD on two autoregressive video diffusion models across multiple quantization settings, metrics, visual comparisons, and ablations. Q-ARVD consistently preserves quality better than baselines, while its modules address different precision regimes and improve deployment efficiency.
- Experimental Setup: Q-ARVD is evaluated on self-forcing and causal-forcing using W8A8, W4A8, and W4A6 quantization settings.Baselines include MinMax, AdaRound, SmoothQuant, PTQ4DiT, and SVDQuant.
- Main Results: Q-ARVD consistently achieves the best FVD-FP and LPIPS-FP scores, with larger gains under the lower-bit W4A8 and W4A6 settings.The experiments treat FVD-FP and LPIPS-FP as primary reference-based criteria.
- Main Results: Q-ARVD maintains video quality across the temporal span, whereas MinMax accumulates errors and SVDQuant introduces semantic changes.The visual comparison covers generated content relative to the original Bfloat16 video.
- Main Results: 1.30× latency speedup and 1.97× model-size reduction are achieved by the W8A8 model on an NVIDIA A6000 GPU.The deployment test uses Triton with batch size 2 and default configurations.
- Metric Analysis: FVD-FP and LPIPS-FP show greater dispersion and bitwidth-order consistency than standard VBench metrics for distinguishing quantized-model quality.The study reports narrow VBench ranges and potentially counterintuitive ordering across some precision settings.
- Ablation Study: Frame weighting helps more at W8A8, while dual-scale quantization helps more at W4A8; combining both modules achieves the best overall performance.Low-precision weights are more vulnerable to degradation from outliers, increasing the relevance of dual-scale quantization.
5 Conclusion
The conclusion presents Q-ARVD as a quantization framework tailored to autoregressive video diffusion models. Its experiments support sensitivity-aware frame weighting and adaptive outlier handling as effective design directions.
- Conclusion: Q-ARVD is presented as the first quantization framework tailored for autoregressive video diffusion models.The framework combines final-quality guided frame weighting with outlier-aware adaptive dual-scale quantization.
- Conclusion: The paper reports quantitative and qualitative evidence supporting the effectiveness and rationale of Q-ARVD’s two design components.The stated directions target unbalanced frame-wise sensitivity and heterogeneous weight outliers.
A Additional Visualization of Outlier Patterns
Additional visualizations show outlier patterns across all layers in blocks 0, 10, and 29, enabling comparison of patterns across block depths.
- Outlier Patterns: Figures 8–10 visualize outlier patterns for all 10 layers in blocks 0, 10, and 29.The figures provide coverage at early, intermediate, and late block depths.
B Additional Samples for Visual Comparison
Additional samples compare self-forcing and causal-forcing models across W4A8, W4A6, and W8A8 settings. The reported qualitative results show Q-ARVD preserving quality while baselines degrade more visibly at low bitwidths.
- Self-Forcing: Figures 11–13 compare self-forcing outputs at W4A8, W4A6, and W8A8.These visualizations extend the self-forcing comparison across the tested bitwidths.
- Causal-Forcing: Figures 14–16 compare causal-forcing outputs at W4A8, W4A6, and W8A8.These visualizations extend the causal-forcing comparison across the tested bitwidths.
- Qualitative Results: Q-ARVD preserves video quality across bitwidths, while baselines show noticeable degradation in the low-bit W4A8 and W4A6 settings.The reported comparison covers both self-forcing and causal-forcing models.
C Additional Implementation Details and Extra Cost Analysis
The implementation preserves selected small modules in BFloat16 and uses Triton kernels to execute INT8 activation quantization, INT8 GEMM, and de-quantization.
- Time embedding, projection modules, and the final prediction head remain in BFloat16 because they are important for performance and small in size.
- Reconstruction uses batch size 8, learning rates of 2e-3 for rounding and 4e-5 for scale, and 2000 training iterations.
- Triton implements separate activation-quantization and INT8 GEMM/de-quantization kernels.The first converts floating-point activations to INT8; the second multiplies INT8 weights and activations and de-quantizes using scales.
D Discussion about Related Outlier-handling Methods
The discussion compares scaling-, rotation-, and low-rank-branch approaches for handling quantization outliers, emphasizing trade-offs and deployment constraints.
- Scaling-based methods suppress activation outliers by jointly scaling weights and activations, but transfer quantization difficulty from activations to weights.
- Rotation-based methods smooth weight and activation distributions through orthogonal transformations, but lack a theoretical guarantee of consistently favorable rotated distributions.
- Rotation can amplify certain channels in practice, while offline rotation is incompatible with DiTs and online rotation adds notable overhead.
- Low-rank branch methods are identified as another mainstream paradigm for addressing quantization outliers.
E Ablation of Minimum Constraint Alpha
The minimum outlier threshold α balances false detections against ineffective constraints, with α = 1.20 used in all experiments and evaluated through ablation.
- α imposes a minimum outlier threshold to reduce false outlier detection.
- Setting α too large raises the outlier threshold, whereas setting it too small can make the constraint ineffective and cause false detection.
- α = 1.20 is used in all experiments and reported to work well.The paper additionally evaluates robustness to α in Table 5.
F Limitations
The method currently uses exponential-like frame-wise sensitivity only during quantization reconstruction, leaving broader uses and further kernel optimization open.
- The exponential-like frame-wise sensitivity is currently leveraged only during quantization reconstruction.
- The sensitivity could potentially extend to techniques such as mixed-precision quantization.
- Dedicated manual CUDA-kernel optimizations might provide additional efficiency gains beyond the Triton deployment.