PaperlayerView exact revision on arXiv

Source-linked AI summary

ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation

Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Enshu Liu, Rui Wan, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, Yu Wang

arXiv:2406.02540v3cs.CV

TL;DR

Large DiTs and multi-frame video generation raise memory and compute costs, and existing quantization methods face challenges on text-to-image and video tasks. ViDiT-Q analyzes those challenges and combines specialized quantization techniques; it reports W4A8 with minimal performance degradation and practical GPU-kernel savings and speedups.

  • Problem

    Existing quantization methods face challenges on text-to-image and video generation, while large models and multi-frame processing burden memory and computation.

    Source: §1 · p. 2

  • Method

    ViDiT-Q combines fine-grained dynamic quantization, static-dynamic channel balancing, and metric-decoupled mixed precision for DiTs.

    Source: §1 · p. 2

  • Results

    ViDiT-Q achieves W4A8 with minimal performance degradation for popular video and image generation models, while CUDA kernels provide 2-2.5× memory savings and 1.4-1.7× latency speedup.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The reported results show practical memory savings and latency speedup alongside minimal degradation at W4A8 for the evaluated image and video models.

    Source: §6 · p. 10

Abstract

from arXiv · show

Diffusion transformers have demonstrated remarkable performance in visual generation tasks, such as generating realistic images or videos based on textual instructions. However, larger model sizes and multi-frame processing for video generation lead to increased computational and memory costs, posing challenges for practical deployment on edge devices. Post-Training Quantization (PTQ) is an effective method for reducing memory costs and computational complexity. When quantizing diffusion transformers, we find that existing quantization methods face challenges when applied to text-to-image and video tasks. To address these challenges, we begin by systematically analyzing the source of quantization error and conclude with the unique challenges posed by DiT quantization. Accordingly, we design an improved quantization scheme: ViDiT-Q (Video & Image Diffusion Transformer Quantization), tailored specifically for DiT models. We validate the effectiveness of ViDiT-Q across a variety of text-to-image and video models, achieving W8A8 and W4A8 with negligible degradation in visual quality and metrics. Additionally, we implement efficient GPU kernels to achieve practical 2-2.5x memory saving and a 1.4-1.7x end-to-end latency speedup.

1 INTRODUCTION

Large DiT models and multi-frame video generation strain memory and computation, while existing quantization methods face challenges on text-to-image and video tasks. The paper analyzes these challenges and introduces ViDiT-Q for image and video DiTs.

1 INTRODUCTION

  • An OpenSORA HPC-AI video with 16 frames at 512×512 uses over 10 GB of GPU memory and takes about 50 seconds on an Nvidia A100.The introduction presents this as an example of the deployment burden of video generation.Source: §1 · p. 2
  • The analysis identifies high data variation within quantization groups as a major error source and finds that reducing quantization error alone may not preserve textual alignment or temporal consistency.The paper also notes that generation quality is multi-faceted.Source: §1 · p. 2
  • ViDiT-Q combines fine-grained dynamic quantization, static-dynamic channel balancing, and metric-decoupled mixed precision to address distinct DiT and generation-quality challenges.The mixed-precision design separates quantization effects across different quality dimensions.Source: §1 · p. 2
  • ViDiT-Q targets DiT quantization challenges and is validated across image and video models, with efficient GPU kernels for practical savings and acceleration.The cited introduction describes the validation and kernel implementation without giving the numerical results reported in the conclusion.Source: §1 · p. 2

2 RELATED WORKS

Related work covers diffusion transformers for image and video generation, metrics for evaluating their outputs, and quantization methods for diffusion models and transformers. The paper positions DiT quantization for text-to-image and video as an unresolved challenge for existing approaches.

2.1 DIFFUSION TRANSFORMERS FOR IMAGE AND VIDEO GENERATION

  • Transformer diffusion backbones have been used for image generation and text-to-video, while high-resolution images and multi-frame videos add hardware costs.The section names DiT and UViT for image generation, PixArt-α for text-to-image, and Latte as an early transformer text-to-video model.Source: §2.1 · p. 2

2.2 IMAGE AND VIDEO GENERATION EVALUATION METRICS

  • Image and video generation require multiple evaluation metrics because output quality includes fidelity, prompt alignment, human preference, motion, and temporal aspects.The passage lists FID and IS for image features, ClipScore for prompt alignment, and video measures including FVD, CLIPSIM, CLIP-temp, Flow-score, and DOVER.Source: §2.2 · p. 3

2.3 MODEL QUANTIZATION

  • Prior quantization methods address diffusion timesteps and transformer channel imbalance, but applying them to text-to-image and video DiTs causes notable performance degradation.Q-DiT uses channel-specific quantization parameters, while PTQ4DiT uses a fixed channel-balance mask across timesteps.Source: §2.3 · p. 3

3 PRELIMINARY ANALYSIS

The preliminary analysis attributes quantization error to large data variation within quantization groups and identifies several dimensions of variation in DiTs. It also argues that generation quality requires evaluation beyond quantization error alone.

3.1 QUANTIZATION ERROR ANALYSIS

  • The quantization objective is framed as minimizing the difference between quantized and floating-point models, using layer-wise weight and activation error as a surrogate.Quantized weights and activations share parameters within each quantization group.Source: §3.1 · p. 3
  • Large groups can set a wide quantization range from a few extreme values, increasing rounding error for most elements.The analysis describes clamping and rounding errors as a trade-off; minmax quantization avoids clipping, leaving rounding error from large scaling as the principal source in this setting.Source: §3.1 · pp. 3–4

3.2 UNIQUE CHALLENGES FOR DITS AND VISUAL GENERATION

  • DiT data varies across tokens, conditions, timesteps, and channels, with video token variation spanning both spatial and temporal dimensions.Activation channel imbalance also changes over timesteps.Source: §3.2 · p. 4
  • Quantization error is often used as a proxy objective, but visual generation quality includes aspects such as alignment, aesthetics, temporal consistency, and flickering.The passage identifies temporal dimensions as additional considerations for video generation.Source: §3.2 · p. 4

4 VIDIT-Q: QUANTIZATION SCHEME TAILORED FOR DITS

ViDiT-Q addresses DiT quantization challenges with finer, dynamic quantization, static-dynamic channel balancing, and metric-decoupled mixed precision. These techniques target data variation, timestep-varying channel imbalance, and multiple dimensions of generation quality.

4 VIDIT-Q: QUANTIZATION SCHEME TAILORED FOR DITS

  • ViDiT-Q combines fine-grained and dynamic quantization, static-dynamic channel balancing, and metric-decoupled mixed precision to address the identified challenges.The scheme is tailored to DiTs and lower-bitwidth generation.Source: §4 · p. 4

4.1 FINE-GRAINED GROUPING AND DYNAMIC QUANTIZATION

  • Channel-wise weights and token-wise activations use hardware-friendly groupings, reducing activation group size to the number of channels with negligible overhead.The groupings respect the input-channel summation in linear layers; the authors also propose online dynamic parameters that adapt to changing timesteps and conditions.Source: §4.1 · pp. 4–5

4.2 STATIC-DYNAMIC CHANNEL BALANCING

  • A single balancing α is sensitive to timestep-dependent activation imbalance, while different α values would require different weight versions across timesteps.The paper combines scaling and rotation to use their respective strengths against static and dynamic imbalance.Source: §4.2 · pp. 5–6
  • The dynamic channel imbalance arises from feature modulation: pretrained scale-shift effects form a static component, while timestep embeddings introduce a dynamic component.The method applies scaling to address static imbalance and rotation to address the varying distribution.Source: §4.2 · pp. 5–6

4.3 METRIC DECOUPLED MIXED PRECISION DESIGN

  • Low quantization error alone may not preserve generation quality because highly sensitive layers can bottleneck performance across different quality dimensions.The paper reports that MSE can overemphasize content changes at the expense of visual quality and temporal consistency.Source: §4.3 · pp. 6–7
  • Metric-decoupled mixed precision assigns higher bit-widths by layer group using VQA, ClipScore, and FlowScore sensitivity measures within four denoising ranges.The allocation iteratively raises the bit-width of the most sensitive layers until the target budget is reached; cross-attention, spatial attention and FFN, and temporal attention affect different quality aspects.Source: §4.3 · p. 7

5 EXPERIMENTS

Experiments evaluate ViDiT-Q on text-to-video and text-to-image generation, then measure hardware savings and ablate its W4A8 techniques. Results highlight baseline failures at low bit-widths and report memory and latency gains from the implemented kernels.

5.1 IMPLEMENTATION DETAILS AND EXPERIMENTAL SETTINGS

  • Video tests use OpenSORA with 100-step DDIM; image tests use PixArt-α on 1,024 COCO prompts with FID, Clipscore, and ImageReward.Video evaluation includes VBench and OpenSORA prompt-set metrics; image generation uses 20-step DPM-solver.Source: §5.1 · pp. 7–8, Table 1 · p. 8

5.2 MAIN RESULTS

  • Video baselines expose W4A8 channel-balance failures, while ViDiT-Q maintains text-to-image performance across bit-widths.The text-to-video evaluation compares generation quality across methods and bit-widths; the text-to-image evaluation uses PixArt-α on COCO.Source: Figure 5 · p. 8, §5.2 · p. 9, Figure 6 · p. 9

5.1 IMPLEMENTATION DETAILS AND EXPERIMENTAL SETTINGS

  • Fused quantized GEMM CUDA kernels are profiled on an A100 at batch size 1 over 20 denoising steps.Quantization and Hadamard operations are fused into preceding operations to reduce overhead.Source: §5.1 · p. 8

5.3 HARDWARE RESOURCE SAVINGS

  • 1.99× peak-memory reduction at W8A8 and 2.42× at W4A8 accompany latency speedups of 1.71× and 1.38×, respectively.The memory and latency comparisons are for ViDiT-Q against the FP16 baseline; the W8A8 latency comparison also includes the naive scheme.Source: §5.3 · pp. 9–10, Figure 7 · p. 10

5.4 ABLATION STUDIES

  • Dynamic token-wise quantization, static-dynamic channel balancing, and metric-decoupled mixed precision each improve W4A8 generation quality in the ablations.Static-dynamic balancing brings quality to a level comparable to FP; MSE-based mixed precision instead degrades performance.Source: §5.4 · p. 10, Table 2 · p. 10, Figure 8 · p. 10

6 CONCLUSION AND LIMITATIONS

ViDiT-Q targets DiT quantization and reports W4A8 generation with minimal quality degradation, alongside practical memory and latency savings. The conclusion notes that mixed precision needs further refinement and lower activation bit-width is needed to fully use 4-bit weight acceleration.

6 CONCLUSION AND LIMITATIONS

  • ViDiT-Q achieves W4A8 with minimal performance degradation and reports 2–2.5× memory savings plus 1.4–1.7× latency speedup.The authors identify mixed-precision refinement and lower activation bit-width as remaining needs for fully using 4-bit-weight acceleration.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

The authors state that their findings are fully reproducible and that anonymized code, figure videos, and experiment details are provided.

REPRODUCIBILITY STATEMENT

  • Anonymized code, figure videos, and experimental details are provided to support reproducing the reported findings.The authors say the experiment details include hyperparameters, training protocols, and evaluation methods.Source: REPRODUCIBILITY STATEMENT · p. 11

A ADDITIONAL EXPERIMENTAL DETAILS

The appendix motivates quantizing linear layers rather than attention computation and records implementation choices for baseline methods.

A.1 MOTIVATION FOR QUANTIZING LINEAR LAYERS ONLY

  • With FlashAttention, attention computation accounts for 14.3% of block latency, so the authors focus quantization on linear layers.The t embedding, y embedding, and final layer remain FP16 because their computation is less than 1/1000 of overall latency.Source: §A.1 · p. 16, Figure 9 · p. 16

A.2 IMPLEMENTION DETAILS FOR BASELINE METHODS

  • Baseline implementations follow each method’s described calibration, grouping, scaling, or rotation procedure, including tensor-wise static activations for PTQ4DiT.The appendix specifies timestep-wise calibration for Q-Diffusion, channel-wise grouping for Q-DiT, grid search for SmoothQuant, and linear-layer rotations for QuaRot.Source: §A.2 · p. 16

B.1 BENCHMARK SUITE

The benchmark suite evaluates video generation across frame quality, temporal quality, and text-prompt semantics, using multiple video metrics. Its analysis also compares ViDiT-Q with naive PTQ on VBench.

B.1 BENCHMARK SUITE

  • VBench covers frame-wise quality, temporal consistency and dynamics, and adherence to the text prompt.Its dimensions include aesthetics and imaging quality; subject, background, and scene consistency; motion smoothness; dynamic degree; and overall consistency.Source: §B.1 · pp. 16–17
  • The evaluation uses three official VBench prompt sets and generates one video per prompt.The sets contain 72 subject-consistency prompts, 93 overall-consistency prompts, and 86 scene prompts.Source: §B.1 · p. 17

B.2 SELECTED METRICS

  • The selected metrics assess video-reference similarity, quantization-related semantic loss, text-video similarity, frame-to-frame semantics, video quality, motion, and temporal flickering.FVD uses 101 UCF101 videos as references; FVD-FP16 instead uses 101 videos generated by the FP16 model and is reported to correlate more strongly with human perception.Source: §B.2 · p. 17

C DETAILED ANALYSIS OF EXPERIMENTAL RESULTS

  • ViDiT-Q outperforms naive PTQ on VBench, with the radar chart comparing normalized results across benchmark dimensions.The chart corresponds to Table 1; its caption states the comparison but provides no metric values.Source: Figure 10 · p. 18

C.1 TEXT-TO-VIDEO PERFORMANCE ON VBENCH

Across VBench dimensions, ViDiT-Q stays near FP16 and outperforms naive PTQ; qualitative comparisons also cover text-to-image generation and ablations. On UCF-101, baseline quantizers degrade notably or fail at W4A8.

C.1 TEXT-TO-VIDEO PERFORMANCE ON VBENCH

  • ViDiT-Q achieves performance similar to FP16 across W8A8, W6A6 mixed precision, and W4A8 mixed precision on VBench, outperforming naive PTQ W8A8.The analysis covers dynamic degree, consistency, and image quality; accompanying figures illustrate these dimensions and results.Source: §C.1 · pp. 18–19, Figure 11 · p. 20, Figure 12 · p. 21

C.2 TEXT-TO-IMAGE GENERATION ON COCO

  • ViDiT-Q images nearly match FP16 in visual quality and text-image alignment, while naive PTQ produces blurred W8A8 images and nearly pure-noise W4A8 images.The comparison is qualitative and appears in Figure 14.Source: §C.2 · p. 23, Figure 14 · p. 23

C.3 VISUALIZATION OF ABLATION STUDIES

  • In the W4A8 ablation, mixed precision reaches quality similar to FP16 after dynamic quantization, channel balancing, and static-dynamic channel balancing improve intermediate outputs.The preceding stages progress from blank images to a background, then reduced color deviation and improved object consistency, but still show degradation before mixed precision.Source: §C.3 · p. 24, Figure 15 · p. 24

D.1 PERFORMANCE OF VIDIT-Q ON UCF-101 DATASET

  • On UCF-101, naive PTQ and Q-Diffusion degrade notably at W8A8 and fail at W4A8, while SmoothQuant still degrades notably at W4A8.The experiments evaluate OpenSORA and Latte; the authors note that FVD is noisy with few videos and propose FVD-FP16 as an alternative for quantization effects.Source: §D.1 · p. 24

D.2 PERFORMANCE OF VIDIT-Q FOR SUPER RESOLUTION TASK.

The additional experiments report ViDiT-Q results for super-resolution, hardware efficiency, lower-bit settings, and comparison or combination with other quantization methods. They also describe its quantization design and linear-layer implementation.

D.2 PERFORMANCE OF VIDIT-Q FOR SUPER RESOLUTION TASK.

  • ViDiT-Q maintains Inf-DiT performance across bitwidths, with W4A8 PSNR and SSIM described as similar to FP16 on the super-resolution task.The evaluation uses bicubic 4× downsampling on DIV2K validation images, with center cropping to satisfy the patch-size constraint.Source: §D.2 · pp. 24–25, Table 4 · p. 25

D.3 EFFICIENCY IMPROVEMENT ON DIFFERENT HARDWARE DEVICES

  • Memory use falls 2× on both RTX3090 and Jetson Orin Nano; latency speedup is 1.6× on RTX3090 and approximately 1.82× on Orin.The authors say platform-specific CUDA tiling could further improve speedup.Source: §D.3 · p. 26, Figure 17 · p. 26

D.4 PERFORMANCE OF VIDIT-Q UNDER LOWER BITWIDTH.

  • ViDiT-Q lower-bit mixed-precision plans perform comparably to or better than Q-DiT W8A8 on Pixart-Sigma, while W4A8 is presented as the conservative setting.The plans average approximately W5A5 for W4A4-MP and W5A8 for W2A8-MP; 2-bit weight quantization may require quantization-aware training.Source: §D.4 · p. 26, Table 5 · p. 27, Figure 18 · p. 27

D.5 COMPARISON WITH GENERAL QUANTIZATION METHODS

  • AdaRound and BRECQ show moderate drops versus FP16, while ViDiT-Q is comparable to FP16; at W4A8, AdaRound and BRECQ fail to produce meaningful content.The authors relate the W4A8 failure to channel-wise variation these general methods are not designed to handle.Source: §D.5 · pp. 26–27, Table 6 · p. 28

D.6 COMBINATION WITH ATTENTION QUANTIZAITON METHOD

  • ViDiT-Q linear-layer quantization and SageAttention QK quantization target different computations and can be combined for additional speedup.The paper reports that linear layers account for more than 80% of model computation and that adding SageAttention can moderately improve latency without notable performance degradation.Source: §D.6 · p. 27, Table 7 · p. 28

E ADDITIONAL ANALYSIS

  • The additional analysis compares ViDiT-Q with baseline quantization designs and mixed-precision analysis schemes.The section also includes a comparison of efficiency when combining with SageAttention.Source: §E.1 · p. 28, Table 7 · p. 28

E.1 COMPARISON WITH BASELINE QUANTIZATION METHODOLOGY DESIGN

  • Static coarse-grained quantization and channel-group methods degrade at W8A8 and fail at W4A8; dynamic quantization avoids offline timestep calibration but adds online overhead.The authors report that efficient CUDA kernels make the overhead acceptable, at 1.74× to 1.71×, while dynamic quantization improves algorithm performance.Source: §E.1 · pp. 28–29

E.2 HARDWARE IMPLEMENTATION OF QUANTIZED LINEAR LAYERS.

  • For quantized linear-layer GEMM, weight rows and activation columns share quantization parameters so integer summation can be separated from scaling.The described implementation uses output-channel-wise weight parameters and channel-wise activation parameters.Source: §E.2 · p. 29, Figure 19 · p. 29

E.3.1 MOTIVATION FOR METRIC DECOUPLE

The analysis connects layer types with particular evaluation metrics and describes metric-decoupled sensitivity analysis. It compares this approach with MSE-based and joint-search alternatives.

E.3.1 MOTIVATION FOR METRIC DECOUPLE

  • Layer types correlate with particular metrics: cross-attention with text-video alignment, temporal attention with FlowScore, and self-attention predominantly with visual quality.Self-attention sensitivity is reported as 0.6232 for visual quality, versus 0.2364 and 0.1404 for the other aspects.Source: §E.3.1 · pp. 29–30, Figure 20 · p. 30

E.3.2 DETAILED PROCESS OF METRIC DECOUPLED ANALYSIS

  • Metric-decoupled analysis selects task-relevant metrics, groups similar layers, examines their metric correlations, then measures relative importance within groups.The authors state that this reduces the search space and avoids directly comparing metrics with different absolute-value scales, helping preserve multiple aspects in mixed-precision plans.Source: §E.3.2 · p. 30

E.3.3 COMPARISON OF DIFFERENT MIXED PRECISION SEARCH METHOD.

  • Under W4A8, MSE-based and joint-metric search perform worse than uniform W4A8, while metric-decoupled analysis compares layers with similar effects.The authors identify cross-metric value balancing as a difficulty for joint search and describe the approach as analogous to controlled-variable analysis.Source: §E.3.3 · pp. 30–31, Table 8 · p. 31

Related papers

Loading 2406.02540v3…