PaperlayerView exact revision on arXiv

Source-linked AI summary

H-DDCM: Hadamard Codebooks for Fast Generative Image and Video Compression

Or Ronai, Vladimir Kulikov, Tomer Michaeli

arXiv:2610.18125v1cs.CVcs.AIcs.LG

TL;DR

Training-free generative compression avoids task-specific retraining, but dense codebooks and combinatorial encoding are costly at modern latent dimensions. H-DDCM uses implicit randomized Hadamard dictionaries and table-free coding, and extends the framework to streaming video; it reports competitive image and video performance with reduced overhead.

  • Problem

    Dense Gaussian codebooks and their residual-matching operations impose substantial memory and computational costs at the large latent dimensions of modern image and video generators.

    Source: §1 · p. 2

  • Method

    H-DDCM replaces Gaussian codebooks with implicit randomized Hadamard dictionaries, uses table-free lexicographic coding, and adapts the framework to autoregressive video with re-encoded KV-cache prefixes.

    Source: §1 · p. 2

  • Results

    Across images and video, H-DDCM reports competitive rate-distortion-perception performance; at FLUX scale, it reduces round trip time from 66 s for Turbo-DDCM (K = d) to 44 s.

    Source: §1 · p. 2, §5 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    H-DDCM matches complete-codebook Turbo-DDCM fidelity on FLUX without materializing the codebook and achieves the highest measured throughput among evaluated methods sharing the Causal-Forcing backbone.

    Source: §1 · p. 2, §7 · p. 10

    The approach inherits ethical risks from pretrained generative models, including possible societal biases and harmful or malicious repurposing.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Training-free diffusion-based compression methods repurpose the generative priors of a diffusion or a flow model to compress data without retraining them as task-specific codecs. Recent codebook-based approaches such as Turbo-DDCM achieve strong perceptual quality at ultra-low bitrates, but their reliance on dense Gaussian codebooks remains costly in both memory and computation, and scales poorly with the large latent dimensions of modern image and video generators. Their combinatorial bitstream encoding further becomes computationally heavy at this scale. We introduce H-DDCM, a scalable generative compression framework that replaces Gaussian codebooks with implicit randomized Hadamard dictionaries. The fast Walsh-Hadamard transform allows for implicitly synthesizing full-dimensional dictionaries with $O(d \log d)$ compute for projection and reconstruction and $O(d)$ memory, without materializing the full-dimensional dictionary. We further introduce an efficient table-free lexicographic coding scheme that makes combinatorial encoding practical at larger dimensions. Beyond images, we adapt the framework to an autoregressive video flow model and develop a streaming compression scheme that periodically re-encodes recently reconstructed frames into a fresh KV-cache prefix, preserving temporal context across segments at no additional transmitted bits. This enables long-form, close to real-time generative video compression on a single GPU. Across image and video benchmarks, H-DDCM achieves competitive rate-distortion-perception performance while substantially reducing the memory and computational overhead of codebook-based training-free generative compression.

1 INTRODUCTION

Training-free generative compression avoids retraining a generator, but dense Gaussian codebooks and combinatorial encoding become difficult to scale to modern image and video latents. H-DDCM replaces these codebooks with implicit randomized Hadamard dictionaries, adds table-free coding, and extends the framework to streaming video.

1 INTRODUCTION

  • Dense Gaussian codebooks and their residual-matching operations scale poorly to large image and video latents, creating substantial memory and computational bottlenecks.A complete codebook is estimated to require approximately 128 GiB for FLUX image latents and 167 GiB for Causal-Forcing video latents.Source: §1 · p. 2
  • Randomized Hadamard dictionaries replace dense Gaussian codebooks with implicit projections and reconstructions, reducing computation to O(d log d) and memory to O(d).The Hadamard basis is exactly orthogonal, and the construction also extends to overcomplete dictionaries formed from multiple independently randomized bases.Source: §1 · p. 2
  • A table-free lexicographic encoder and decoder with grouping preserves combinatorial bit savings while making encoding practical at larger latent dimensions.The approach addresses the dense precomputed binomial-coefficient table used by Turbo-DDCM, whose construction and storage become impractical at higher dimensions.Source: §1 · p. 2
  • H-DDCM adapts Causal-Forcing to streaming video by re-encoding recent reconstructed frames into a fresh KV-cache prefix without additional transmitted bits.On images, the structured dictionary matches complete-codebook Turbo-DDCM fidelity without materializing that codebook; on video, the codec has the highest measured throughput among evaluated methods sharing the Causal-Forcing backbone.Source: §1 · p. 2

2 RELATED WORK

Prior generative compression methods include learned codecs, training-free diffusion approaches, and generative video codecs. H-DDCM belongs to the training-free setting and replaces explicit Gaussian codebooks with an implicit structured dictionary.

Neural and generative compression

  • Learned generative codecs use GAN or diffusion priors but require compression-specific training or fine-tuning, tying the codec to a backbone and generally requiring retraining when it changes.Earlier learned codecs targeted low distortion but tended to produce over-smoothed reconstructions at low bitrates.Source: §2 · p. 2

Training-free diffusion-based image compression

  • Training-free image methods leave the generator frozen and encode information through its sampling trajectory; DDCM uses reproducible codebook atoms, and Turbo-DDCM combines atoms in closed form.Turbo-DDCM reduces expensive denoising evaluations by over 90%.Source: §2 · pp. 2–3

Training-free generative video compression

  • Training-free video methods include ZeroGVC, which combines image-codec I-frames with Turbo-DDCM-steered autoregressive P-frames, and GVCC, which converts a deterministic rectified-flow ODE into an equivalent marginal-preserving SDE.Both retain explicit Gaussian codebooks, whose costs are especially high for video because latent chunks are larger than image latents.Source: §2 · p. 3

Autoregressive and streaming video diffusion

  • Causal-Forcing distills from an autoregressive teacher to address structural limits in prior causal-video distillation and supports practical real-time streaming generation on a single GPU.The cited structural condition is frame-level injectivity at the ODE-initialization stage, which bidirectional teachers violate.Source: §2 · p. 3

3 BACKGROUND

The background reviews how Turbo-DDCM encodes information through diffusion sampling and why its codebook and bitstream operations become costly at large dimensions. It then introduces randomized Hadamard transforms as a fast implicit alternative.

3.1 DENOISING DIFFUSION MODELS

  • Diffusion sampling reverses a gradual noising process, and its stochastic transition variable provides the channel that DDCM replaces with information-carrying noise maps.The paper uses DDPM sampling for images and a few-step consistency formulation for its causal video backbone.Source: §3.1 · p. 3

3.2 TURBO-DDCM

  • DDCM selects reproducible codebook noise, while Turbo-DDCM approximates each residual with a sparse combination of atoms to transmit more information per step and reduce denoiser evaluations.For orthonormal dictionaries, thresholding the largest residual correlations solves the sparse objective exactly; Turbo-DDCM applies this rule to approximately orthogonal Gaussian atoms.Source: §3.2 · pp. 3–4
  • Turbo-DDCM encodes the selected unordered subset by its lexicographic rank, avoiding permutation redundancy, but dense codebooks and precomputed binomial tables become costly as latent dimensions grow.Restricting the codebook reduces computation but also limits available residual directions.Source: §3.2 · p. 4

3.3 RANDOMIZED HADAMARD TRANSFORM

  • The normalized Hadamard matrix is an orthonormal basis whose correlations can be computed by FWHT in O(d log d) computation and O(d) memory without materializing the basis.Random sign modulation preserves exact orthogonality while allowing the randomized basis to be regenerated from a shared seed and applied with one FWHT.Source: §3.3 · p. 4

4 METHOD

H-DDCM replaces dense codebook operations and precomputed combinatorial tables with implicit Hadamard transforms and table-free grouped coding, then adds multiple bases to expand low-rate candidate directions.

4 METHOD

  • The method addresses both scaling bottlenecks: dense Gaussian codebook operations and costly combinatorial bitstream computation.It also extends the structured dictionary with multiple randomized Hadamard bases for the low-rate regime.Source: §4 · p. 5

4.1 STRUCTURED HADAMARD DICTIONARY

  • An orthonormal randomized Hadamard basis makes residual thresholding exact, with encoding and decoding in O(d log d) time and O(d) memory without storing a dictionary.The encoder uses one FWHT and top-M selection; the decoder uses one FWHT for reconstruction.Source: §4.1 · p. 5

4.2 BITSTREAM OPTIMIZATION

  • Table-free lexicographic coding preserves combinatorial index savings, while grouping trades a small rate overhead for smaller, independently parallelizable operations.The proposed encoders use incremental Pascal-rule updates or a hockey-stick formulation with binary-search unranking.Source: §4.2 · pp. 5–6

4.3 EXTENSION: OVERCOMPLETE HADAMARD DICTIONARY

  • The overcomplete dictionary concatenates B randomized orthogonal bases, giving the encoder Bd candidate directions instead of d.Because atoms across bases are not mutually orthogonal, global top-M selection is no longer exact; the richer dictionary also adds transform and index costs, so fewer atoms fit a fixed bit budget.Source: §4.3 · p. 6

5 APPLICATION TO IMAGE COMPRESSION

On SD 2.1 and FLUX image-compression evaluations, H-DDCM is competitive in rate-distortion-perception performance and reduces round-trip time at FLUX scale while avoiding explicit codebook materialization.

Evaluation

  • Evaluations use SD 2.1 at 512 × 512 and FLUX at 1024 × 1024, measuring distortion by PSNR and perception by FID.The comparisons include training-free methods sharing the backbone, plus trained compression codecs and BPG for SD 2.1; FLUX comparisons use training-free methods.Source: §5 · pp. 6–7

Results

  • H-DDCM matches complete-codebook Turbo-DDCM and DiffC in PSNR/FID; FLUX round-trip time is 44 s versus 66 s for complete-codebook Turbo-DDCM.On FLUX, capping Turbo-DDCM’s codebook below the latent dimension reduces quality, while the complete codebook increases memory and runtime.Source: Figure 1 · p. 7, §5 · pp. 7–8
  • At matched bitrate on SD 2.1, H-DDCM and Turbo-DDCM (K = d) each take 1.3 s round trip, versus 5.5 s for DiffC and 61 s for DDCM.The reported time includes compression and decompression.Source: §5 · p. 8

6 APPLICATION TO STREAMING VIDEO COMPRESSION

H-DDCM adapts its framework to the CF autoregressive video backbone, using cache refresh to preserve context across segments and support long-form, close-to-real-time compression.

6 APPLICATION TO STREAMING VIDEO COMPRESSION

  • The video evaluation uses CF at 832×480, applying residual coding to each causal latent chunk with four consistency steps and an overcomplete dictionary of B = 8.The evaluation uses no deterministic tail (N = 0) and compares encoding and decoding throughput on a single NVIDIA H200 GPU.Source: §6 · pp. 8–9
  • Re-encoding the last 9 reconstructed pixel frames at each segment boundary creates a fresh KV-cache prefix without additional transmitted bits.This preserves recent context while keeping the cache within CF’s training regime; the first newly coded chunk uses a larger selection budget before returning to M indices per step.Source: §6 · pp. 8–9
  • The evaluation covers UVG and MCL-JCV, with HEVC Classes B/C/D additionally reported in Appendix C, and compares classical, neural, and CF-based training-free methods.The comparison includes HEVC, VVC, DCVC-UF, ZeroGVC, and Turbo-DDCM with capped and complete codebooks.Source: §6 · pp. 9–10
  • H-DDCM achieves competitive rate-distortion-perception performance and the highest measured encoding and decoding throughput among evaluated CF-based methods.Figure 3 compares rate, PSNR, DOVER, FVMD, and throughput; Figure 4 illustrates visual and temporal behavior, including discontinuities at ZeroGVC segment boundaries.Source: Figure 3 · p. 9, Figure 4 · p. 9, §6 · p. 10

7 CONCLUSION

H-DDCM combines implicit Hadamard dictionaries and scalable table-free coding for generative compression, extending the framework to streaming video. It reports competitive rate-distortion-perception performance and the highest measured throughput among evaluated methods sharing the CF backbone.

7 CONCLUSION

  • H-DDCM achieves competitive rate-distortion-perception performance and the highest measured throughput among evaluated methods sharing the CF backbone.On FLUX, it matches the fidelity of a complete Turbo-DDCM codebook without materializing it while retaining the runtime efficiency of a capped codebook.Source: §7 · p. 10
  • H-DDCM replaces dense Gaussian codebooks with implicit randomized Hadamard dictionaries and precomputed lexicographic tables with scalable table-free, group-factorized coding.These changes enable efficient compression at large latent dimensions.Source: §7 · p. 10
  • For video, an overcomplete Hadamard dictionary and re-encoded-tail KV-cache refresh support long-form, close-to-real-time streaming compression on the CF autoregressive backbone.The refresh preserves context across segment boundaries.Source: §7 · p. 10
  • The approach inherits ethical considerations from pretrained generative models, whose outputs may reflect societal biases or be misused in sensitive applications.The authors state that their contributions are intended solely to advance research in generative modeling.Source: ETHICS STATEMENT · p. 10

A EXPERIMENT IMAGE COMPRESSION CONFIGURATIONS

The image experiments specify configurations for Stable Diffusion 2.1 and FLUX, including their datasets, method sets, and rate sweeps.

A EXPERIMENT IMAGE COMPRESSION CONFIGURATIONS

  • The image configurations report exact experiment settings and use pretrained backbones for training-free methods, while trained baselines use public checkpoints without additional training or fine-tuning.The section specifies configurations for SD 2.1 and FLUX image experiments.Source: A EXPERIMENT IMAGE COMPRESSION… · p. 15

A.1 SD 2.1 (KODAK24, DIV2K, CLIC2020)

  • SD 2.1 experiments use the same method set and per-method hyperparameters across three datasets; H-DDCM and full-codebook Turbo-DDCM each use T = 30 and a 22-point M grid.DDCM uses T = 1000 under the stated codebook configurations; trained baselines use released checkpoints, and BPG is evaluated at five QP values.Source: §A.1 · p. 15

A.2 FLUX (DIV2K)

  • At FLUX scale (d = 218), the comparison includes H-DDCM, capped and complete Turbo-DDCM, and DiffC.H-DDCM uses 14 rate points; Turbo-DDCM variants use matched rate targets, while materializing the full Gaussian codebook would require approximately 128 GiB in FP16, so it is processed in blocks.Source: §A.2 · pp. 15–16

B EXPERIMENT VIDEO COMPRESSION CONFIGURATIONS

The experiments compare H-DDCM across image and video settings, using matched generative backbones and multiple datasets. They report competitive rate-distortion-perception performance, examine qualitative results and design ablations, and detail the configurations used.

B EXPERIMENT VIDEO COMPRESSION CONFIGURATIONS

  • Each continuation generates 18 new latent frames after a three-frame prefix re-encoded from the preceding segment’s last nine reconstructed pixel frames; the prefix requires no transmitted bits.The first segment generates 21 latent frames, and the first newly coded chunk after refresh uses a larger bit budget until the steady-state budget reaches its floor.Source: B EXPERIMENT VIDEO COMPRESSION… · p. 16
  • Video experiments use the CF backbone at its native 832 × 480 resolution and compare rate-matched H-DDCM, Turbo-DDCM, ZeroGVC, trained, and classical codec configurations.Experiments use the same configurations across video datasets; ZeroGVC uses independent 32-pixel-frame GOPs, while HEVC and VVC use two-pass bitrate targeting.Source: B EXPERIMENT VIDEO COMPRESSION… · p. 16

C.1 IMAGE COMPRESSION QUANTITATIVE COMPARISON; C.3 VIDEO COMPRESSION QUANTITATIVE COMPARISON

  • H-DDCM maintains competitive rate-distortion-perception performance across image and video comparisons, with low runtime or highest measured CF-backbone throughput among evaluated training-free methods.The image comparisons include complementary distortion and timing measures; the video comparisons cover HEVC Classes B, C, and D.Source: Figure S1 · p. 17, §C.1 · p. 17, Figure S2 · p. 18

C.1 IMAGE COMPRESSION QUANTITATIVE COMPARISON

  • Additional image comparisons report SSIM and LPIPS alongside separate encoding, decoding, and round-trip times; PSC is excluded from timing and memory comparisons because of its slow runtime.Figure S2 covers Kodak24, DIV2K, and CLIC2020 on SD2.1 and DIV2K on FLUX.Source: §C.1 · p. 17, Figure S2 · p. 18

C.2 IMAGE COMPRESSION QUALITATIVE COMPARISON

  • Qualitative image comparisons show training-free diffusion-based methods on Kodak24, DIV2K, CLIC2020, and FLUX/DIV2K at the stated image resolutions.The SD2.1 comparisons use 512 × 512 images; the FLUX comparison uses 1024 × 1024 images.Source: §C.2 · p. 19, Figure S3 · p. 19, Figure S4 · p. 19

C.4 VIDEO COMPRESSION QUALITATIVE COMPARISON

  • Video examples pair spatial frames with spatiotemporal slices to show reconstruction fidelity and temporal continuity across MCL-JCV, HEVC, and UVG examples.The captions report comparable rate-distortion-perception tradeoffs for H-DDCM and temporal discontinuities at ZeroGVC’s I-frame boundaries.Source: §C.4 · p. 23, Figure S9 · p. 23, Figure S10 · p. 23

C.5 ABLATION OF THE OVERCOMPLETE HADAMARD DICTIONARY

  • Across MCL-JCV bitrates, overcomplete variants improve distortion and perceptual metrics, with B = 8 strongest among tested values; at 0.01 BPP it is more consistent and visually realistic.The comparison tests B ∈ {2, 4, 8} randomized bases against the single-basis dictionary.Source: §C.5 · p. 25, Figure S14 · p. 25, Figure S15 · p. 25

C.6 ABLATION OF TRANSPOSE AND PSEUDO-INVERSE SYNTHESIS

  • On UVG, direct synthesis and pseudo-inverse synthesis achieve nearly identical rate-distortion-perception performance, so the method uses direct synthesis to avoid an additional solve.The selected noise vectors remain highly aligned; in an orthonormal dictionary, the pseudo-inverse expression reduces to direct combination.Source: §C.6 · p. 26, Figure S16 · p. 26

C.7 ABLATION OF CACHE REFRESH AND FIRST-STEP INITIALIZATION

The ablations test cache refresh and first-step initialization, while the sampling appendix specifies the image and video updates. Refresh improves quantitative results over cache alternatives, whereas removing initialization trades alignment for some better aggregate metrics.

C.7 ABLATION OF CACHE REFRESH AND FIRST-STEP INITIALIZATION

  • On MCL-JCV, re-encoded-tail refresh improves over hard-reset and continuous-cache alternatives; at 0.01 BPP, hard resets cause segment discontinuities and continuous cache causes severe artifacts beyond the training context regime.The comparison uses H-DDCM OC with B = 8.Source: §C.7 · p. 27, Figure S17 · p. 27, Figure S18 · p. 28
  • Removing first-step initialization can improve some aggregate metrics, but the resulting video struggles to remain temporally aligned with the input.The qualitative ablation uses BPP = 0.01.Source: §C.7 · p. 27, Figure S17 · p. 27

D SAMPLING FORMULATIONS

  • For image sampling, the final N + 1 trajectory steps are deterministic and bit-free, using the σt = 0 DDIM update instead of the stochastic reverse transition.The deterministic update recombines the current state with the network’s estimate of the clean image.Source: D SAMPLING FORMULATIONS · p. 29
  • Video consistency sampling rescales the predicted clean chunk toward the next step and injects selected noise; the first step uses a source-chunk spatial-mean proxy residual before later steps use network predictions.Unlike images, all four video consistency steps select and transmit noise.Source: D SAMPLING FORMULATIONS · p. 29

E BITSTREAM PROTOCOL

The protocol ranks selected atom supports rather than transmitting their indices independently, and uses table-free methods to make ranking practical at scale. The explicit-codebook analysis quantifies the memory burden and the fidelity tradeoff of capping its size.

E BITSTREAM PROTOCOL

  • Ranking the sorted M-index support as one lexicographic rank avoids independently transmitting indices and their M! redundant permutations.The rank uniquely determines the selected support.Source: E BITSTREAM PROTOCOL · p. 30
  • Grouped coding first ranks the group-count vector, then each nonempty group’s local support; the decoder recovers group identity from the local rank’s position.Setting G = 1 gives a flat rank, while G = B groups atoms by randomized basis.Source: E BITSTREAM PROTOCOL · p. 30
  • Table-free recurrence avoids impractical precomputed tables while producing bit-identical ranks and inverses with O(K) updates and O(1) auxiliary memory.The tested procedure matched the direct formula at every tested (K, M) pair.Source: E BITSTREAM PROTOCOL · p. 31
  • Hockey-stick coding suits low-M video, while recurrence coding is used for image sweeps where hockey-stick becomes less competitive.The recurrence has O(K) cost; hockey-stick encoding uses O(M) binomial evaluations and decoding uses O(M log K).Source: E BITSTREAM PROTOCOL · pp. 31–32
  • The recurrence avoids the table’s one-time build cost and is comparable to or faster per call once that table exists.At FLUX scale, the table build takes 1386.8 seconds at K = 2^18 and occupies 8.10 GiB.Source: E BITSTREAM PROTOCOL · pp. 31–32, Table S1 · p. 32

F COST OF AN EXPLICIT CODEBOOK

  • An explicit codebook occupies 128 GiB for FLUX and 167 GiB for a CF video chunk, and larger codebooks must be regenerated in blocks, increasing runtime.Capping K reduces cost but excludes residual components outside the codebook span, even when more atoms are selected.Source: F COST OF AN EXPLICIT CODEBOOK · p. 33

G OVERCOMPLETE HADAMARD DICTIONARY

The overcomplete extension concatenates randomized Hadamard bases and selects atoms across them, with grouped rate matching for comparison against a single basis. The shared trajectory algorithms describe compression and decompression, while streaming segmentation and cache refresh are specified separately.

G OVERCOMPLETE HADAMARD DICTIONARY

  • Each overcomplete basis operates in a padded Hadamard domain, then crops to the true latent coordinates; the resulting dictionary is overcomplete.The padded basis is orthogonal, but cropping means its effective dictionary has orthonormal rows rather than orthonormal columns.Source: G OVERCOMPLETE HADAMARD DICTIONARY · p. 33
  • Keeping each basis’s local top M scores reduces the global search to BM candidates before selecting the overall top M.The implementation batches reconstruction across active bases.Source: G OVERCOMPLETE HADAMARD DICTIONARY · pp. 33–34
  • Rate matching compares overcomplete and single-basis dictionaries at similar slot lengths, and the overcomplete version uses strictly fewer atoms because each index costs more to name.The overcomplete configuration groups each randomized basis separately and estimates slot length using balanced per-basis counts.Source: G OVERCOMPLETE HADAMARD DICTIONARY · p. 34, Table S3 · p. 34

H H-DDCM PSEUDO-CODE

  • The shared single-trajectory algorithms encode selected coefficients and decode them using the shared seed; streaming segmentation and KV-cache refresh are handled separately.The implicit dictionary is applied through the FWHT rather than materialized.Source: H H-DDCM PSEUDO-CODE · p. 35

Related papers

Loading 2610.18125v1…