PaperlayerView exact revision on arXiv

Source-linked AI summary

OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

Donghyun Lee, Jitesh Chavan, Duy Nguyen, Sam Huang, Liming Jiang, Priyadarshini Panda, Timo Mertens, Saurabh Shukla

arXiv:2607.02461v1cs.CVcs.AIcs.LG

TL;DR

Diffusion transformers are expensive to run, while existing PTQ methods [24] [52] [53] require recalibration as activations shift. OrbitQuant instead quantizes weights and activations in a shared, normalized, rotated basis using dimension-specific codebooks. It achieves state-of-the-art image and video PTQ on GenEval and VBench, retaining usable W2A4 images where prior PTQ baselines collapse to noise.

  • Problem

    Compute-intensive DiT inference motivates PTQ, but shifting activations require existing methods [24] [52] [53] to recalibrate for each checkpoint, resolution, or modality.

    Source: §1 · p. 2

  • Method

    OrbitQuant quantizes weights and activations in a shared, rotated, normalized basis using one offline distributional codebook per dimension, eliminating input-dependent range calibration.

    Source: §4.1 · p. 3, §4.5 · p. 5

  • Results

    OrbitQuant achieves state-of-the-art image and video PTQ on GenEval and VBench, retaining usable W2A4 images where prior PTQ baselines collapse to noise.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The same quantization recipe transfers from image to video without per-modality tuning.

    Source: §7 · p. 8

    20.3 GB peak video memory exceeds QuaRot and SmoothQuant’s 19.3 GB, with the increase attributed to lookup-materialized index and gather tensors.

    Source: §5.5 · p. 8

Abstract

from arXiv · show

Diffusion transformers (DiTs) achieve state-of-the-art image and video generation, but their multi-step sampling and growing parameter count make inference expensive. Post-training quantization (PTQ) is the natural remedy, yet DiT activations shift across timesteps, prompts, and guidance branches, forcing prior methods to re-fit calibration data for every new checkpoint or modality. We present OrbitQuant, a data-agnostic weight-activation quantizer that bypasses range estimation by quantizing in a normalized, rotated basis. In this basis, a randomized permuted block-Hadamard (RPBH) rotation concentrates each coordinate around one fixed, known marginal regardless of the input, so a single Lloyd-Max codebook serves all timesteps, prompts, and layers of a given input dimension. We extend the same quantizer to weight rows offline, absorbing the rotation into the weights so that it cancels inside each linear layer and only a forward rotation on the activations remains at runtime. The same recipe transfers from image to video with no per-modality tuning. Across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, it sets the state of the art for PTQ at several low-bit settings. It also pushes PTQ of image diffusion transformers to W2A4 with usable generation quality.

1. Introduction

OrbitQuant quantizes DiT weights and activations in a shared rotated basis, replacing per-timestep range calibration with one offline Lloyd–Max codebook. It achieves state-of-the-art PTQ on GenEval and VBench without calibration data.

1. Introduction

  • Compute-bound DiT inference motivates weight–activation PTQ because weight-only quantization yields no measured speedup [24].Repeated denoising evaluations compound inference cost; quantizing both weights and activations compresses memory and computation without retraining.Source: §1 · p. 2
  • DiT activation outliers [45] and drift across timesteps, prompts, and guidance branches [7] [53] force existing PTQ methods [24] [52] [53] to recalibrate.Calibration sets must be recollected and refit for each new checkpoint, resolution, or modality.Source: §1 · p. 2
  • Random rotation maps normalized activations to a fixed, input-independent coordinate distribution [51], enabling one offline Lloyd–Max codebook across denoising steps.The efficient randomized permuted block-Hadamard (RPBH) rotation uses a uniform random permutation to keep activations well-quantizable at low bit-width without calibration.Source: §1 · p. 2
  • OrbitQuant folds the same rotation into weight rows offline, quantizing weights and activations in one shared basis while leaving only one forward RPBH rotation at inference.The shared rotation cancels inside each linear layer.Source: §1 · p. 2
  • W2A4 retains usable images where prior PTQ baselines collapse to noise; OrbitQuant also achieves state-of-the-art PTQ on GenEval and VBench without calibration data.The evaluation covers both image and video diffusion transformers.Source: §1 · p. 2

2. Related Work

OrbitQuant builds on rotation-based quantization but uses an analytic, calibration-free codebook inside DiT layers, transferring unchanged between image and video models.

LLM Quantization

  • LLM weight-only quantizers suit memory-bound decoding [9] [12] [21], while activation quantization handles outlier channels through scaling [46] or rotations [2] [5] [18] [29] [40] [42].Rotation-based methods fold Hadamard or learned rotations into weights through computational invariance, preserving outputs while making activations easier to quantize.Source: §2 · p. 2
  • PolarQuant and TurboQuant [51] use distribution-derived codebooks for standalone KV-cache quantization; OrbitQuant instead applies shared rotations that cancel inside DiT projections.PolarQuant uses analytically known polar-angle distributions after random preconditioning; TurboQuant [51] uses a dense Haar rotation with a Cartesian Beta-marginal Lloyd–Max codebook. Both rotate back for reconstruction, whereas OrbitQuant replaces dense Haar rotation with efficient RPBH.Source: §2 · p. 2
  • Unlike calibrated permutations in DuQuant [27] and PeRQ [38], OrbitQuant samples permutations uniformly at random with a probabilistic guarantee of well-behaved rotated coordinates.DuQuant [27] orders channels by outlier magnitude; PeRQ [38] balances per-block mass, which its analysis identifies as governing block-Hadamard outlier suppression.Source: §2 · p. 2

Diffusion Quantization

  • Most existing DiT quantizers depend on calibration for outlier handling, channel balancing, scale fitting, precision allocation, rotations, reordering, or calibration-data selection.SVDQuant [24] fits a high-precision low-rank branch; PTQ4DiT [45] combines channel balancing with block reconstruction; AdaTSQ [52] fits scales with timestep-sensitive precision; ViDiT-Q [53] combines per-channel calibration and mixed precision. LRQ-DiT [49] uses calibrated DuQuant-style [27] rotations, PermuQuant [8] calibrates channel reordering, and S2Q-VDiT [11] combines Hessian-aware calibration-data selection with token-level distillation.Source: §2 · p. 2
  • OrbitQuant uses an analytic distribution-derived codebook requiring no model evaluation at construction, and transfers unchanged between image and video DiTs.RPBH-rotated coordinates concentrate around a fixed marginal f_d ≈ N(0, 1/d), enabling one Lloyd–Max codebook per dimension and bitwidth across layers, timesteps, prompts, and modalities. Folding the rotation into weights makes it cancel inside each layer, avoiding calibration despite timestep and CFG-branch drift.Source: §2 · p. 3, Figure 2 · p. 3
  • DVD-Quant [26] is data-free but video-specific with per-model machinery, while ConvRot [19] uses calibration-free group-wise regular Hadamard rotations and a uniform grid on FLUX.DVD-Quant [26] combines rotated quantization with grid refinement and adaptive bit allocation. QVGen [20] and RobuQ [48] instead use quantization-aware training, with RobuQ [48] reaching ternary weights on ImageNet DiTs.Source: §2 · pp. 2–3

3. Preliminaries

This section defines calibration-free weight–activation quantization for DiT linear layers and reviews two ingredients inherited from TurboQuant [51]: Haar-random rotation and a Lloyd–Max scalar codebook.

3.1. Notation

  • The objective is to approximate Wx using quantized weights and activations at b_w and b_a bits per coordinate, across every denoising step and prompt without calibration data.L denotes the target linear layers, and D their distinct input dimensions.Source: §3.1 · p. 3

3.2. TurboQuant

  • TurboQuant [51] normalizes each vector, stores its norm, and applies a Haar-random orthogonal rotation [32], yielding an input-independent coordinate marginal.Originally developed for KV-cache compression, TurboQuant [51] is calibration-free.Source: §3.2 · p. 3, §3 · p. 3
  • Offline Lloyd–Max codebooks minimize MSE for each input-dimension/bit-width pair, providing 2^b centroids.For d ≥64, the post-rotation coordinate marginal is tightly approximated by N(0, 1/d), and distinct coordinates are nearly independent.Source: §3.2 · p. 3
  • Codebooks use no scales or zero-points and are shared across all layers and rows with the same input dimension.Dequantization looks up centroids, applies the inverse rotation, and rescales by the stored norm.Source: §3.2 · p. 3

4. Methodology

OrbitQuant quantizes weights and activations in a shared normalized, rotated basis, replacing input-range calibration with fixed dimension-specific Lloyd–Max codebooks. Absorbing the rotation into offline-quantized weights leaves only a forward activation rotation at inference.

4.1. Overview

  • Shared weight and activation rotations cancel inside each linear product, eliminating inverse rotations at runtime.Weights are quantized offline and activations online. Before quantization, W′x′ = Wx; the quantized operands approximate this product directly.Source: §4.1 · p. 3, §4.3 · p. 4

4.2. Offline Weight Quantization

  • Offline weight quantization rotates each weight matrix, separates row magnitudes from unit directions, quantizes directions with Lloyd–Max, and restores the magnitudes.The marginal-matched codebook is MSE-optimal for directions following fd. Row norms are stored in BF16, adding less than 0.3% overhead relative to quantized-direction storage, and rotated quantized weights replace the originals.Source: §4.2 · p. 4

4.3. Online Activation Quantization

  • Online activation quantization rotates each token, normalizes its direction, applies the fixed Lloyd–Max quantizer, and rescales by its magnitude.The per-token magnitude s is the only input-dependent quantity; the codebook needs no re-fitting. Normalization uses s + ε, with ε = 10^-10 guarding against zero norms on padding tokens.Source: §4.3 · p. 4

4.4. Randomized permuted block-Hadamard

  • O(d log h) RPBH transforms [41] replace dense Haar rotations [51] requiring O(d^2) time per token and storage.RPBH stores signs and a permutation array and supports any dimension, unlike full Randomized Hadamard Transforms [2] [42]. Block size h is the largest power of two dividing d, ranging from 128 to 4096 across evaluated models.Source: §4.4 · p. 4
  • A uniformly random permutation spreads coordinates across Hadamard blocks without data-dependent calibration, preventing block-local mixing from trapping concentrated input mass.The variance bound keeps rotated marginals close to N(0, 1/d) and the codebook near-optimal unless one coordinate carries an outsized norm share. Removing the permutation degrades low-bit robustness.Source: §4.4 · pp. 4–5

4.5. Data-agnostic Codebook

  • One dimension-specific Lloyd–Max codebook serves every timestep, prompt, layer, and weight row of that dimension without collecting input statistics.The codebook is computed offline from the dimension-dependent marginal fd; quantization maps normalized, rotated coordinates to their nearest centroids.Source: §4.5 · p. 5
  • OrbitQuant adapts TurboQuant’s [51] fixed-marginal codebook to direct linear-layer computation, replacing its dense rotation and quantize-dequantize reconstruction with RPBH and weight-absorbed rotations.TurboQuant [51] targets KV-cache and vector-database compression. At FLUX.1-schnell attention and feed-forward projections, RPBH-rotated weights and activations match fd approximately as closely as dense Haar rotations, supporting shared-codebook reuse.Source: §4.5 · p. 5, Figure 3 · p. 5

5. Experiments

OrbitQuant leads the reported W4A4 image and W4A6/W4A4 video PTQ comparisons, transferring the image recipe unchanged to video. Fake-quantization tests show the lowest quantization overhead, although its video lookup tensors increase peak memory relative to some baselines.

5.1. Setup

  • OrbitQuant quantizes transformer-block projections across three image and two video DiTs, keeping AdaLN modulation projections, where present, at INT4 RTN [24].Image models are FLUX.1-schnell, FLUX.1-dev, and Z-Image-Turbo at W4A4/W2A4; video models are Wan 2.1-1.3B and CogVideoX-2B at W4A6/W4A4. All implemented methods share the AdaLN treatment, while Wan has no AdaLN modulation.Source: §5.1 · p. 5

5.2. Image generation: GenEval

  • OrbitQuant leads GenEval PTQ at W4A4 and remains functional at W2A4, where rotation and smoothing baselines collapse to near-zero.At W4A4, Overall exceeds FP16 on FLUX.1-schnell and Z-Image-Turbo and trails by 0.034 on FLUX.1-dev. At W2A4, it retains most FLUX quality and uniquely produces meaningful Z-Image-Turbo scores.Source: Table 1 · p. 6, §5.2 · p. 6

5.3. Video generation: VBench

  • OrbitQuant ranks first among PTQ methods on VBench Overall Consistency for both video backbones at W4A6 and W4A4 using the unchanged image recipe.The backbones are Wan 2.1-1.3B [43] and CogVideoX-2B [50]; OrbitQuant also leads or stays closest on most individual dimensions.Source: §5.3 · p. 6, Table 2 · p. 7

5.4. Qualitative Comparison

  • OrbitQuant preserves image structure and color at W3A3 and Wan 14B video layout and frame consistency at W4A4, unlike QuaRot [2] and ViDiT-Q [53].Images remain close to BF16 on FLUX.1-dev, FLUX.1-schnell, and Z-Image-Turbo; competing methods collapse to noise on Z-Image-Turbo and drift in video color and structure.Source: §5.4 · p. 6, Figure 4 · p. 7

5.5. Latency and Memory Analysis

  • 1.09×, 1.28×, and 1.40× slower image inference than OrbitQuant is reported for SmoothQuant [46], QuaRot [2], and ViDiT-Q [53], respectively.OrbitQuant has the lowest weight-and-activation quantization overhead on both image and video, using fixed shared-codebook nearest-centroid lookup instead of dynamic per-token quantization or additional smoothing. These BF16 fake-quantization measurements assess overhead, not realized low-bit speedup.Source: §5.5 · pp. 7–8, Figure 5 · p. 8
  • 20.3 GB video peak memory exceeds QuaRot [2] and SmoothQuant [46] at 19.3 GB but remains below ViDiT-Q [53] at 23.2 GB.Lookup-generated index and gather tensors cause the video memory increase. On images, OrbitQuant has the lowest peak memory, matching the unquantized footprint.Source: §5.5 · pp. 7–8

6. Ablations

On FLUX.1-schnell, RPBH is strongest among the tested rotations at W3A3 and W2A4. INT4 AdaLN weights nearly match BF16 GenEval Overall across three models while retaining 4× FLUX compression.

6.1. Comparison between Rotation Matrix

  • W3A3 and W2A4 give RPBH the highest GenEval Overall among four tested rotations on FLUX.1-schnell; W4A4 differences are within noise.Any orthogonal rotation preserves the forward identity, but compatibility with the fixed codebook requires matching the marginal fd. RPBH’s permutation spreads clustered outliers across blocks, keeping that marginal close to fd.Source: §6.1 · p. 8
  • 26× faster activation rotation than dense Haar is reported for structured rotations; RPBH adds 0.070 s over Block-RHT without exceeding Full RHT latency.RPBH is constructible for every studied dimension, including CogVideoX-2B’s d=1920, where no fast size-d Hadamard kernel exists.Source: §6.1 · p. 8

6.2. AdaLN bit-width

  • INT4 AdaLN weights nearly match BF16 GenEval Overall across three models with the main model at W4A4 and AdaLN activations in BF16.W3 holds across all three models, whereas W2 collapses FLUX.1-dev and FLUX.1-schnell but leaves Z-Image-Turbo robust. Low-bit AdaLN weights corrupt modulation read by every downstream layer.Source: Figure 6 · p. 8, §6.2 · p. 8
  • 27% of weights belong to AdaLN projections, so leaving them in BF16 reduces FLUX compression from 4× to 2.21×.Their timestep-dependent scale-and-shift cannot be folded into neighboring weights. OrbitQuant therefore fixes these projections at INT4 weight RTN regardless of the main bit-width.Source: §6.2 · p. 8, Figure 6 · p. 8

7. Conclusion

OrbitQuant replaces per-timestep range calibration with a single distributional codebook in a shared, rotated, normalized basis. The calibration-free weight-activation quantizer transfers from image to video without per-modality tuning and achieves state-of-the-art low-bit PTQ results on GenEval and VBench.

7. Conclusion

  • OrbitQuant sets the low-bit PTQ state of the art on GenEval and VBench across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX without per-modality tuning.The same recipe transfers from image to video and supports usable 2-bit weights where prior PTQ methods collapse.Source: §7 · p. 8
  • OrbitQuant replaces per-timestep range calibration with a single distributional codebook applied in a shared, rotated, normalized basis for weight-activation quantization.The random permutation in RPBH keeps the rotated marginal well-behaved at low bit-width.Source: §7 · p. 8
  • Absorbing the rotation into weights offline makes it cancel inside each linear layer, leaving only a forward RPBH rotation on activations at runtime.Source: §7 · p. 8

Supplementary Material … C.2. Video Generation on Huge Model

The supplement provides RPBH’s incoherence argument, generation settings, layer-quantization rules, and additional image and large-video experiments. OrbitQuant leads the W3A3 image comparisons and most Wan 14B W4A4 dimensions, but degrades sharply on Z-Image-Turbo at W2A3.

A. Proof Sketch for RPBH Incoherence

  • RPBH combines per-block incoherence with permutation-driven mass balancing, concentrating conditional coordinate variances around 1/d with probability at least 1 - δ.Random signs make coordinates mean-zero. Permutation balances block masses regardless of outlier placement, whereas per-block incoherence alone also holds without permutation.Source: A. Proof Sketch for RPBH Incoherence · p. 12
  • When no coordinate carries an outsized share of the norm, Berry–Esseen bounds support approximating every rotated coordinate by N(0, 1/d).Conditional on the permutation, each coordinate sums independent bounded terms with variance M_j/h; Figure 3 provides empirical confirmation of the Gaussian approximation.Source: A. Proof Sketch for RPBH Incoherence · p. 12

B.1. Generation settings

  • Generation uses public-checkpoint samplers and step counts for images, specified frame and guidance settings for videos, and NVIDIA H100 GPUs.FLUX.1-schnell uses 4 steps and guidance 0.0; FLUX.1-dev uses 50 steps and guidance 3.5; Z-Image-Turbo uses 10 steps and guidance 0.0. Wan 2.1-1.3B uses 81 frames, 480×832, 50 steps, CFG 5.0; CogVideoX-2B uses 49 frames, 480×720, 50 steps, CFG 6.0.Source: §B.1 · p. 12

B.2. Quantized and skipped layers

  • OrbitQuant quantizes attention and feed-forward projections, including text-conditioning K/V, but treats AdaLN modulation separately and leaves several peripheral components in BF16.AdaLN uses INT4 RTN weights at group size 64 with BF16 activations: its dynamic scale-and-shift prevents shared-rotation cancellation, and its single conditioning token offers no activation-compute savings. Embeddings, the timestep MLP, final un-patchify head, and text encoder remain BF16.Source: §B.2 · p. 12

C.1. Lowest bit-widths: W3A3 and W2A3

  • W3A3 OrbitQuant achieves the best GenEval Overall on all three image models while staying close to FP16.AdaTSQ [52] leads on some individual dimensions, but OrbitQuant is more consistent; SVDQuant [24] collapses on Z-Image-Turbo.Source: Table 4 · p. 13, §C.1 · p. 13
  • W2A3 OrbitQuant alone remains functional and usable on FLUX models, while rotation and smoothing baselines approach zero and OrbitQuant degrades sharply on Z-Image-Turbo.The passage attributes baseline collapse to a 3-bit uniform grid failing to place levels where rotated activations are dense.Source: Table 4 · p. 13, §C.1 · p. 13

C.2. Video Generation on Huge Model

  • 7 of 8 VBench dimensions favor OrbitQuant as the best PTQ method on Wan 14B at W4A4.OrbitQuant stays within noise of BF16 on most dimensions. Rotation and smoothing baselines drop sharply on motion and scene, where activation outliers are largest.Source: Table 5 · p. 13, §C.2 · p. 14
  • OrbitQuant transfers calibration-free to HunyuanVideo, competing with DVD-Quant [26] and exceeding it on imaging quality, motion smoothness, and scene.The comparison uses W4A4 and one recipe across backbones, without per-model design; baseline and DVD-Quant numbers come from the DVD-Quant paper.Source: §C.2 · p. 14

C.3. Robustness · C.4. Comparison with QAT

OrbitQuant generally trails the strongest QAT baseline, QVGen [20], but stays close across most VBench dimensions and leads several on Wan 2.1-1.3B without fine-tuning. Three-seed image tests show consistent results, with larger variability for Z-Image-Turbo at W2A4.

C.3. Robustness

  • OrbitQuant’s calibration-free pipeline has only two stochastic components: the RPBH rotation and sampling noise.The seed-based evaluation checks whether results depend on a particular random draw.Source: §C.3 · p. 14
  • 0.005 or less GenEval Overall standard deviation is observed for every image model at W4A4, supporting single-seed representativeness.The reported variability comes from three full-pipeline runs with different random seeds.Source: §C.3 · p. 14
  • 0.013 or less GenEval Overall standard deviation holds for the FLUX models at W2A4, while Z-Image-Turbo shows larger variance at W2A4.Z-Image-Turbo at W2A4 is the reported exception to otherwise consistent results across seeds.Source: §C.3 · p. 14
  • 3 random seeds are used to rerun the full pipeline and report GenEval means and standard deviations.The evaluation covers three image models at both W4A4 and W2A4.Source: §C.3 · p. 14

C.4. Comparison with QAT

  • OrbitQuant generally trails the strongest QAT baseline, QVGen [20], while staying close across most VBench dimensions.QVGen [20] fine-tunes the quantized model with an objective directly targeting video quality.Source: §C.4 · p. 14
  • OrbitQuant surpasses every compared QAT method on Subject Consistency, Scene, and Overall Consistency on Wan 2.1-1.3B.It achieves these results without gradient steps or per-model design.Source: §C.4 · pp. 14–15

D. Limitations and Future Work

OrbitQuant adds a small runtime cost for activation rotations, while its current implementation dequantizes codes for BF16 matrix multiplication. Future work targets fused kernels for native low-bit computation with non-uniform centroids.

D. Limitations and Future Work

  • 0.451 s per image is reported for the fused activation-rotation implementation on a single H100, with rotation required at every forward pass.The implementation combines a fused fast Hadamard transform kernel with random permutation; this small runtime cost accompanies the memory savings.Source: D. Limitations and Future Work · p. 15
  • Non-uniform Lloyd–Max centroids lack an off-the-shelf codebook GEMM kernel, so OrbitQuant currently dequantizes codes and performs matrix multiplication in BF16.Integer tensor cores operate on uniform grids; all baselines also use BF16 matrix multiplication under fake quantization.Source: D. Limitations and Future Work · p. 15
  • Lookup-table GEMM kernels for non-uniform weight quantization [14] motivate a planned fused kernel that gathers centroids and computes in a native low-bit format.Centroid gathering would occur in the GEMM prologue, addressing an implementation limitation rather than a limitation of the method.Source: D. Limitations and Future Work · p. 15

Related papers

Loading 2607.02461v1…