PaperlayerView exact revision on arXiv

Source-linked AI summary

OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

Donghyun Lee, Jitesh Chavan, Duy Nguyen, Sam Huang, Liming Jiang, Priyadarshini Panda, Timo Mertens, Saurabh Shukla

arXiv:2607.02461v1cs.CVcs.AIcs.LG

TL;DR

DiT activation drift makes calibration-based PTQ difficult to transfer across timesteps, prompts, checkpoints, and modalities. OrbitQuant uses a normalized, RPBH-rotated basis with an analytic Lloyd–Max codebook, folding the rotation into weights offline. It achieves state-of-the-art low-bit PTQ across image and video DiTs and produces usable images at W2A4.

  • Problem

    DiT activations shift across timesteps, prompts, and guidance branches, so existing PTQ methods must recollect and refit calibration data for new checkpoints, resolutions, or modalities.

  • Method

    OrbitQuant normalizes and RPBH-rotates activations to a fixed marginal, uses one offline Lloyd–Max codebook per dimension, and folds the same rotation into weight rows.

  • Results

    OrbitQuant achieves state-of-the-art PTQ on GenEval and VBench without calibration data across image and video DiTs, with usable generation at W2A4.

  • Takeaways & Limitations

    A single calibration-free recipe transfers across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX without per-modality tuning.

  • Takeaways & Limitations

    Runtime activation rotation adds a small cost, and non-uniform Lloyd–Max centroids currently require dequantizing codes before BF16 matrix multiplication.

Abstract

from arXiv · show

Diffusion transformers (DiTs) achieve state-of-the-art image and video generation, but their multi-step sampling and growing parameter count make inference expensive. Post-training quantization (PTQ) is the natural remedy, yet DiT activations shift across timesteps, prompts, and guidance branches, forcing prior methods to re-fit calibration data for every new checkpoint or modality. We present OrbitQuant, a data-agnostic weight-activation quantizer that bypasses range estimation by quantizing in a normalized, rotated basis. In this basis, a randomized permuted block-Hadamard (RPBH) rotation concentrates each coordinate around one fixed, known marginal regardless of the input, so a single Lloyd-Max codebook serves all timesteps, prompts, and layers of a given input dimension. We extend the same quantizer to weight rows offline, absorbing the rotation into the weights so that it cancels inside each linear layer and only a forward rotation on the activations remains at runtime. The same recipe transfers from image to video with no per-modality tuning. Across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, it sets the state of the art for PTQ at several low-bit settings. It also pushes PTQ of image diffusion transformers to W2A4 with usable generation quality.

1. Introduction

DiTs make high-quality image and video generation expensive, while activation drift prevents calibration-based PTQ from transferring across inputs. OrbitQuant replaces moving range calibration with a shared rotated-basis codebook and achieves strong low-bit results without calibration data.

  • DiT inference is expensive because denoising repeats across timesteps and remains compute-bound, limiting the benefit of weight-only quantization.
  • Activation statistics shift across timesteps, prompts, and classifier-free-guidance branches, forcing existing methods to recollect calibration data for new checkpoints, resolutions, or modalities.
  • OrbitQuant rotates normalized activations with RPBH so one known distribution supports a single Lloyd–Max codebook across denoising steps.The rotation is folded into weight rows, canceling inside each linear layer while leaving one forward activation rotation at inference.
  • The shared-rotation design quantizes weights and activations in one common basis without calibration data.
  • OrbitQuant achieves state-of-the-art PTQ on GenEval and VBench without calibration data across image and video DiTs.At W2A4, it remains the only evaluated method reported to produce usable images when prior PTQ baselines collapse to noise.

2. Related Work

Prior DiT quantizers mainly rely on calibration, while related rotation methods address outliers or enable data-free vector quantization in other settings. OrbitQuant instead uses an analytic distribution-derived codebook and transfers one recipe across image and video DiTs.

  • LLM rotation-based quantizers fold Hadamard or learned rotations into weights to make activations easier to quantize while preserving the linear-layer output.
  • TurboQuant uses a dense Haar rotation and fixed Beta-marginal codebook for calibration-free vector compression, whereas OrbitQuant applies its distributional codebook to diffusion-transformer quantization.
  • Related calibration-based DiT methods use low-rank branches, block reconstruction, timestep-sensitive scales, mixed precision, calibrated rotations, or channel reordering.
  • DVD-Quant is data-free but tailored to video DiTs with per-model machinery, while ConvRot uses calibration-free regular Hadamard rotations and a uniform grid on FLUX.
  • OrbitQuant uses an analytic codebook requiring no model evaluation at construction and transfers unchanged between image and video DiTs.

3. Preliminaries

The preliminaries establish distributional quantization through normalization, random orthogonal rotation, and offline Lloyd–Max codebooks. OrbitQuant later adapts these ingredients with an efficient block-Hadamard rotation and shared weight-activation treatment.

  • TurboQuant combines a Haar-random orthogonal rotation with a Lloyd–Max scalar codebook designed for the post-rotation coordinate distribution.
  • The DiT objective is to replace weights and activations with low-bit surrogates while preserving Ŵx̂ ≈ Wx at every denoising step and prompt without calibration data.
  • TurboQuant normalizes each vector, preserves its norm, and applies a Haar-random orthogonal rotation before quantization.
  • For d ≥64, rotated coordinates follow a marginal tightly approximated by N(0, 1/d), enabling offline codebooks for each dimension and bit-width.
  • The nearest-centroid codebook uses no scales or zero-points and is shared across layers and rows with the same input dimension.

4. Methodology

OrbitQuant quantizes weights and activations in a shared normalized, rotated basis using fixed distribution-derived codebooks, while RPBH makes the transform efficient and broadly input-agnostic.

  • Data-agnostic quantization: OrbitQuant replaces per-input range calibration with fixed Lloyd–Max codebooks applied after normalization and a shared RPBH rotation.The codebooks are built offline from the post-rotation marginal and reused across dimensions, bit-widths, timesteps, prompts, and layers.
  • Shared rotated basis: Weights are rotated offline by Π_d^T, while incoming activations receive the matching forward rotation Π_d before quantization.The shared basis lets the rotation cancel inside each linear layer, avoiding an inverse rotation at runtime.
  • Offline weight quantization: Weight rows use the same marginal-aware direction quantizer, while row magnitudes remain in BF16 with less than 0.3% storage overhead.Because the rotation is sampled independently of each weight row, its normalized coordinates follow the codebook design marginal.
  • Online activation quantization: Activation quantization normalizes each rotated token into a magnitude and unit direction, quantizes the direction, and rescales it by the token magnitude.The only input-dependent runtime quantity is the per-token scalar s; ε = 10^-10 protects against zero norms on padding tokens.
  • Efficient RPBH rotation: RPBH combines block Walsh–Hadamard transforms, random sign diagonals, and a uniform permutation, requiring O(d log h) computation and compact storage.The block construction works on any dimension d, unlike full randomized Hadamard transforms restricted to power-of-two dimensions.
  • Efficient RPBH rotation: The leading random permutation spreads input coordinates across blocks, keeping rotated coordinates near the fixed marginal and supporting low-bit robustness without data-dependent calibration.The proposition and empirical results connect this variance concentration to a marginal near N(0, 1/d), while removing the permutation degrades low-bit robustness.

5. Experiments

OrbitQuant is evaluated across image and video diffusion transformers at low-bit settings, with comparisons spanning calibration-based and calibration-free baselines. It achieves strong quality, qualitative fidelity, and overhead results, while video memory remains a trade-off.

  • Setup: OrbitQuant is evaluated on three image DiTs at W4A4 and W2A4, and two video DiTs at W4A6 and W4A4.Image models are FLUX.1-schnell, FLUX.1-dev, and Z-Image-Turbo; video models are Wan 2.1-1.3B and CogVideoX-2B.
  • Setup: Image baselines include SVDQuant, AdaTSQ, ViDiT-Q, Q-DiT, QuaRot, and SmoothQuant, while video comparisons include ViDiT-Q, SVDQuant, QuaRot, and SmoothQuant.Baseline numbers are primarily taken from AdaTSQ for image and QVGen for video.
  • Image generation: GenEval: At W4A4, OrbitQuant is essentially lossless and outperforms every PTQ baseline on GenEval across the three image backbones.It exceeds FP16 Overall on FLUX.1-schnell and Z-Image-Turbo and trails FP16 by 0.034 on FLUX.1-dev.
  • Image generation: GenEval: At W2A4, rotation and smoothing baselines collapse to near-zero, while OrbitQuant retains most FLUX quality and remains the only method with meaningful Z-Image-Turbo scores.This is also described as usable image generation where prior PTQ baselines collapse to noise.
  • Video generation: VBench: At W4A6 and W4A4, OrbitQuant ranks first on Overall Consistency for both Wan 2.1-1.3B and CogVideoX-2B.At W4A6 it is the strongest PTQ method on both backbones; at W4A4 it stays closest on most dimensions as baselines lose ground.
  • Qualitative results: In qualitative comparisons, OrbitQuant preserves image structure and color at W3A3 and video scene layout and frame consistency at W4A4, unlike competing methods.The comparisons use BF16 outputs as reference and include FLUX models, Z-Image-Turbo, and Wan 14B.
  • Latency and memory: OrbitQuant has the lowest quantization overhead on image and video, with competitors running 1.09×, 1.28×, and 1.40× slower on image.These fake-quantization measurements assess overhead rather than realized low-bit speedup; OrbitQuant also matches the unquantized image peak memory.
  • Latency and memory: On video, OrbitQuant’s peak memory is 20.3 GB versus 19.3 GB for QuaRot and SmoothQuant and 23.2 GB for ViDiT-Q.The extra memory comes from materializing an index and gather tensor while activations dominate video workloads.

6. Ablations

Ablations show that the random permutation is central to OrbitQuant’s low-bit robustness, while AdaLN weights can remain at INT4 with little quality loss. Structured rotations also provide favorable implementation speed.

  • Comparison between Rotation Matrix: At W3A3 and W2A4, RPBH is strongest among the tested rotations, outperforming dense Haar, permutation-free Block-RHT, and Full RHT.The random permutation spreads clustered outliers across blocks, keeping the rotated marginal compatible with the fixed codebook.
  • Comparison between Rotation Matrix: RPBH adds 0.070 s over Block-RHT, is no slower than Full RHT, and remains constructible at d=1920 for CogVideoX-2B.Structured rotations use a fast Hadamard kernel; dense Haar is reported as running 26× slower.
  • AdaLN bit-width ablation: Quantizing AdaLN weights to INT4 nearly matches BF16 GenEval Overall on all three tested models at fixed W4A4 main-model precision.Lowering AdaLN precision further degrades Overall in a model-dependent way, with FLUX.1-dev and FLUX.1-schnell collapsing at W2.

7. Conclusion

OrbitQuant concludes that calibration-free quantization in a shared rotated and normalized basis transfers across image and video DiTs. It reaches strong low-bit PTQ results and supports usable 2-bit weights where prior methods collapse.

  • 7. Conclusion: OrbitQuant replaces per-timestep range calibration with one distributional codebook for weights and activations in a shared rotated, normalized basis.Offline weight rotation cancels inside each linear layer, leaving a forward RPBH activation rotation at runtime.
  • 7. Conclusion: The same recipe transfers across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX without per-modality tuning and sets the state of the art for low-bit PTQ.The conclusion covers both GenEval and VBench evaluations.
  • 7. Conclusion: OrbitQuant supports usable 2-bit weights for image diffusion transformers where prior PTQ methods collapse.This summarizes the paper’s W2A4 result.

A. Proof Sketch for RPBH Incoherence

The RPBH construction combines blockwise incoherence with random permutation-based mass balancing, yielding rotated coordinates with nearly uniform variance and approximately Gaussian marginals. Experiments apply OrbitQuant across image and video DiTs at low-bit settings, where it remains competitive or functional under challenging quantization.

  • Proof construction: RPBH applies randomized permutations and block-Hadamard transforms to normalized inputs, producing block outputs whose coordinates are analyzed as Rademacher sums.The proof separates per-block incoherence from permutation-based mass balancing.
  • Proof construction: Permutation-based mass balancing equalizes per-coordinate variance near 1/d, a property lost by the no-permutation variant when outliers align poorly with blocks.The argument uses concentration for sampling without replacement and a union bound across blocks and coordinates.
  • Proof construction: When no coordinate dominates the norm, Berry–Esseen bounds show each rotated coordinate is close to N(0, 1/d), not merely matched in variance.The paper reports that Figure 3 empirically confirms this Gaussian approximation.
  • Evaluation setup: OrbitQuant quantizes all transformer linear projections except AdaLN modulation projections, including image- and text-side attention and feed-forward layers.The evaluation uses public image and video checkpoints with their stated sampling configurations.
  • Image results: At W3A3, OrbitQuant achieves the best GenEval Overall score on all three evaluated image models and stays close to FP16.At W2A3, competing rotation and smoothing methods collapse near zero, while OrbitQuant remains usable on FLUX models but degrades sharply on Z-Image-Turbo.
  • Video results: On Wan 14B at W4A4, OrbitQuant is the best PTQ method on seven of eight VBench dimensions and remains near BF16 on most dimensions.The supplied passages also identify comparisons on HunyuanVideo and video-generation tables, but do not provide complete values here.

C.3. Robustness

OrbitQuant’s results are stable across random seeds at most tested settings and remain competitive with fine-tuning-based QAT methods, despite using calibration-free PTQ.

  • Seed robustness: OrbitQuant’s stochastic variation comes from the RPBH rotation and sampling noise, rather than calibration data.The full pipeline was rerun with three random seeds to test whether results depended on one random draw.
  • Seed robustness: At W4A4, GenEval Overall standard deviation is at most 0.005 on every image model across three random seeds.The FLUX models remain within 0.013 on Overall at W2A4, while Z-Image-Turbo shows larger variance at that setting.
  • Comparison with QAT: Although generally below the strongest QAT baseline QVGen, OrbitQuant stays close across most VBench dimensions and leads QAT methods on several Wan 2.1-1.3B dimensions.It leads on Subject Consistency, Scene, and Overall Consistency without gradient steps or per-model design.

D. Limitations and Future Work

OrbitQuant’s runtime path retains an online activation rotation and currently lacks native low-bit execution for its non-uniform Lloyd–Max codebook.

  • Runtime overhead: OrbitQuant applies the RPBH rotation to activations at every forward pass, adding a runtime cost alongside memory savings.The reported cost is 0.451 s per image on a single H100 at 1024^2, and the authors characterize it as small but implementation-related.
  • Runtime overhead: Because Lloyd–Max centroids are non-uniform, the current implementation dequantizes codes and performs matrix multiplication in BF16 instead of using integer tensor-core GEMMs.The authors identify lookup-table GEMM kernels and fused centroid gathering as future work.
Loading 2607.02461v1…