PaperlayerView exact revision on arXiv

Source-linked AI summary

CoD: A Diffusion Foundation Model for Image Compression

Zhaoyang Jia, Zihan Zheng, Naifu Xue, Jiahao Li, Bin Li, Zongyu Guo, Xiaoyi Zhang, Houqiang Li, Yan Lu

arXiv:2511.18706v3cs.CV

TL;DR

Text conditions describe fine image details inefficiently, and discrete text vocabularies prevent joint encoder–decoder optimization for rate–distortion. CoD is a compression-oriented diffusion foundation model that uses native image tokens and an information bottleneck to jointly optimize compression and diffusion generation. CoD substantially improves diffusion-based compression, particularly at ultra-low bitrates, and CoD-based DiffC matches VTM in PSNR while outperforming MS-ILLM in FID.

  • Problem

    Text conditions describe fine image details inefficiently, and discrete text vocabularies prevent joint encoder–decoder optimization for rate–distortion.

    Source: §1 · p. 2

  • Method

    CoD is a compression-oriented diffusion foundation model that uses native image tokens and an information bottleneck to jointly optimize compression and diffusion generation.

    Source: §1 · p. 2

  • Results

    CoD substantially improves diffusion-based compression, particularly at ultra-low bitrates, and CoD-based DiffC matches VTM in PSNR while outperforming MS-ILLM in FID.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    CoD is self-supervised, trained on open image datasets in around 20 A100 GPU days (0.3% cost), versus Stable Diffusion v1.5’s about 6,250 A100 GPU days [8].

    Source: §1 · p. 2

    CoD is not yet trained for high-resolution inputs, and like other diffusion codecs it does not meet real-time coding requirements.

    Source: §6 · p. 9

Abstract

from arXiv · show

Existing diffusion codecs typically build on text-to-image diffusion foundation models like Stable Diffusion. However, text conditioning is suboptimal from a compression perspective, hindering the potential of downstream diffusion codecs, particularly at ultra-low bitrates. To address it, we introduce \textbf{CoD}, the first \textbf{Co}mpression-oriented \textbf{D}iffusion foundation model, trained from scratch to enable end-to-end optimization of both compression and generation. CoD is not a fixed codec but a general foundation model designed for various diffusion-based codecs. It offers several advantages: \textbf{High compression efficiency}, replacing Stable Diffusion with CoD in downstream codecs like DiffC achieves SOTA results, especially at ultra-low bitrates (e.g., 0.0039 bpp); \textbf{Low-cost and reproducible training}, 300$\times$ faster training than Stable Diffusion ($\sim$ 20 vs. $\sim$ 6,250 A100 GPU days) on entirely open image-only datasets; \textbf{Providing new insights}, e.g., We find pixel-space diffusion can achieve VTM-level PSNR with high perceptual quality and can outperform GAN-based codecs using fewer parameters. We hope CoD lays the foundation for future diffusion codec research. Codes are released at https://github.com/microsoft/GenCodec/tree/main/CoD.

1. Introduction

Text-conditioned diffusion models are not naturally suited to compression: their text conditions describe fine details inefficiently and prevent joint rate–distortion optimization. CoD replaces text conditioning with native image tokens in a compression-oriented diffusion foundation model, improving downstream compression and offering lower-cost, reproducible training.

1. Introduction

  • Text conditions describe fine-grained spatial and texture details inefficiently, while discrete text vocabularies prevent joint encoder–decoder optimization for rate–distortion.Prior findings also report that text conditions can be detrimental to compression, especially at low bitrates.Source: §1 · p. 2
  • CoD replaces text conditions with native image tokens and an information bottleneck, enabling joint optimization of compression and diffusion generation at bitrates as low as 0.0039 bpp.The bottleneck transmits essential semantic information while leaving the diffusion model flexibility to generate realistic details.Source: §1 · p. 2
  • CoD is a general foundation model, not a fixed codec, and can replace Stable Diffusion in downstream diffusion-compression algorithms.Unlike Stable Diffusion's generation priors, CoD additionally learns compression priors that boost downstream performance, particularly at low bitrates.Source: §1 · p. 2
  • CoD takes about 20 A100 GPU days versus Stable Diffusion v1.5's 6,250, and uses open image datasets rather than text-image pairs or proprietary data.Its self-supervised image-only training is described as easier to scale and reproduce.Source: §1 · p. 2
  • The analysis finds that pixel-space diffusion can deliver high perceptual quality and high PSNR across a wide bitrate range, reaching about 47 dB at 4 bpp.The paper also reports that diffusion codecs can surpass GAN-based codecs with significantly fewer parameters.Source: §1 · pp. 2–3
  • CoD-based DiffC significantly outperforms Stable-Diffusion-based DiffC, especially at low bitrates, and matches VTM in PSNR while outperforming MS-ILLM in FID.The comparison highlights performance at ultra-low bitrates and pixel-space reconstruction fidelity.Source: §1 · p. 2

2. Compression-oriented Diffusion

CoD combines a compact image-token condition with diffusion reconstruction, and trains the model to balance perceptual generation with reconstruction fidelity. It uses unified rectified-flow training and image-only datasets.

2.1. Pipeline

  • CoD encodes images into compact tokens, applies an entropy bottleneck, decodes them into diffusion conditions, and reconstructs images by denoising Gaussian noise.The encoder outputs at 1/32 resolution; the condition decoder produces a 1/16-resolution condition for denoising.Source: §2.1 · p. 3

2.2. Diffusion Training

  • CoD uses rectified flow to predict a velocity field, whose loss is theoretically linked to distribution alignment and perceptual quality.The paper relates minimizing the rectified-flow loss to reducing KL divergence between reconstructed and original distributions.Source: §2.2 · p. 4
  • Unified training adds one-step reconstruction distortion to rectified-flow training, addressing its tendency to preserve structure more than color.A randomly selected α% of samples use t ∈ [0, 1], while the rest use t = 0; the t = 0 objective directly minimizes one-step distortion.Source: §2.2 · p. 4

2.3. Implementation Details

  • CoD trains on 22M publicly available images and requires about 20 A100 GPU days, around 0.3% of Stable Diffusion v1.5’s training cost.Training combines rectified-flow, representation-alignment, codebook-commitment, and auxiliary losses, with progressive training from 256 × 256 to 512 × 512.Source: §2.3 · pp. 4–5

3. Analysis on Foundation Model

The analysis evaluates CoD against Stable-Diffusion-based codecs, examines model scaling and sampling-step control, and compares latent- and pixel-space diffusion. It reports scaling gains, controllable distortion–perception trade-offs, and broader quality and bitrate ranges for pixel-space CoD.

3.1. Evaluation

  • On Kodak at 512 × 512, latent-space CoD achieves superior reconstruction quality to Stable-Diffusion-based codecs at its native 0.0039 bpp.The comparison uses the same SD-VAE for latent CoD, DiffC, DDCM, PerCo (SD), and OSCAR.Source: §3.1 · p. 5, Figure 6 · p. 5

3.2. Scaling Law

  • On Kodak, pixel-space CoD models trained at 256×256 for 400k steps show consistently better compression as parameters increase; 49M-parameter CoD achieves notably better reconstruction quality than 181M-parameter MS-ILLM.Source: §3.2 · p. 5

3.3. Zero-Shot Distortion-Perception Control

  • At 0.0039 bpp, pixel-space CoD’s PSNR rises by 3.4 dB, from 16.2 dB to 19.6 dB, when sampling steps fall from 25 to one.Twenty-five steps give the best perceptual quality; intermediate step counts smoothly interpolate, unlike PerCo (SD), which does not show this controllable improvement.Source: §3.3 · pp. 5–6

3.4. Revisiting Pixel-Space Diffusion

  • Pixel-space CoD exceeds 1.0 bpp while improving reconstruction quality, surpasses latent diffusion models in LPIPS around 0.3 bpp, and maintains higher PSNR across the range.Latent-space CoD remains bounded by VAE quality even up to 0.60 bpp.Source: §3.4 · p. 6

4. Downstream Compression Performance

In downstream comparisons, pixel-space CoD combines VTM-comparable distortion with better perceptual quality, while latent-space CoD performs best among compared latent codecs at lower bitrates. Visual comparisons also report sharper details from pixel-space CoD at high bitrates.

Pixel-space comparison

  • Across pixel-codec bitrates, CoD achieves superior PSNR, DISTS, and FID, though it may not achieve the best LPIPS at certain bitrates.HiFiC, MS-ILLM, CDC, and TACO are optimized with LPIPS, whereas CoD is not.Source: §4 · p. 6, Figure 8 · p. 7
  • −2.1% BD-Rate using VTM as anchor: CoD achieves comparable BPP–PSNR performance to VTM while delivering significantly better perceptual quality.Figure 8 compares pixel-space codecs; CoD is not optimized using LPIPS, unlike HiFiC, MS-ILLM, CDC, and TACO.Source: §4 · p. 6, Figure 8 · p. 7

Latent-space comparison

  • At bitrates below 0.02 bpp, DiffC with latent-space CoD achieves the best reconstruction quality among the compared latent-space codecs.At higher bitrates, the performance gap narrows as results approach the SD-VAE limit; CoD is slightly worse than DiffC on some points, including FID at 0.02 bpp on CLIC.Source: §4 · p. 7

Visualization

  • At high bitrates, pixel-space CoD reconstructs sharp, accurate details, while SD-VAE-based latent codecs distort small characters and MS-ILLM struggles to recover clear text.At low bitrates, CoD has higher fidelity than SD-based codecs and more realistic details than prior pixel-space methods.Source: §4 · p. 7

5. Related Works

The related-work discussion situates CoD among generative image codecs and diffusion codecs. It contrasts CoD’s compression-oriented foundation-model approach with prior GAN-based and pretrained-diffusion approaches.

5.1. Generative Image Compression

  • Generative image compression seeks both low distortion and perceptual quality; GAN-based codecs improve realism over MSE-optimized codecs but can increase reconstruction distortion.The discussion says TACO uses textual information to mitigate MSE degradation but still does not achieve an ideal distortion–perception balance.Source: §5.1 · p. 7

5.2. Diffusion Image Compression

  • Most diffusion codecs rely on pretrained foundation models such as Stable Diffusion, while CoD is presented as a compression-oriented alternative backbone.Unlike CDC, CoD emphasizes training at approximately 0.0039 bpp using diffusion loss rather than traditional perceptual losses, and analyzes scaling behavior, pixel versus latent space, and conditioning modalities.Source: §5.2 · p. 8

6. Conclusion and Limitation

CoD is presented as a compression-oriented diffusion foundation model jointly optimized for compression and generation, with potential for strong rate–distortion–perception trade-offs. The authors also identify limits in input resolution and real-time coding.

6. Conclusion and Limitation

  • CoD is a compression-oriented diffusion foundation model trained from scratch for joint optimization of compression and diffusion generation.The authors report that it substantially improves diffusion-based compression, particularly at ultra-low bitrates.Source: §6 · p. 8
  • Pixel-space diffusion demonstrates potential for strong rate–distortion–perception trade-offs across a wide bitrate range.The authors say CoD establishes a foundation for future advances in diffusion-based compression.Source: §6 · p. 9
  • CoD is not yet trained for high-resolution inputs and, like other diffusion codecs, does not meet real-time coding requirements.Extending to higher resolutions such as 2K is described as technically feasible but substantially more computationally costly.Source: §6 · p. 9

Change Log

The change log records an initial submission, a revision with updated algorithms and analyses, and a CVPR camera-ready update.

Change Log

  • The initial submission was released on 2025-11-24.Source: Change Log · p. 9
  • The 2025-12-10 revision updated one-step codec results, perceptual supervision, downstream-codec analyses, and X-prediction.Source: Change Log · p. 9
  • The 2026-03-16 update marked the CVPR camera-ready version and updated the GitHub code link.Source: Change Log · p. 9

Appendices

The supplementary material adds training details and results, then examines CoD characteristics and further codec applications. Its conclusions highlight extreme compression, one-step coding, perceptual loss, and X-prediction.

Appendices

  • The supplementary material extends the paper with training details, results, CoD analyses, and additional downstream codec applications.Source: Appendices · p. 11
  • At 64 bits, CoD preserves semantics; finetuned one-step CoD is reported to achieve real-time, SOTA performance comparable to StableCodec [58] and OneDC [53].Source: Appendices · p. 11
  • CoD-based perceptual loss significantly improves MS-ILLM [36].Source: Appendices · p. 11
  • For pixel-space CoD, X-prediction performs better than V-prediction.Source: Appendices · p. 11

A. Towards 64-Bit Image Compression

The appendices test CoD at 64 bits and develop one-step and higher-bitrate variants. The results report semantic preservation at extreme compression, improved downstream DiffC performance, and gains from restored perceptual supervision at higher bitrates.

A. Towards 64-Bit Image Compression

B.1. Preliminary

  • One-step codecs train end-to-end; DMD distillation matches real and fake sample distributions using a pretrained multi-step model for real scores and a learned fake-score model.Source: §B.1 · p. 12

B.2. Distilling a CoD

  • The one-step objective combines pixel, perceptual, representation-alignment, and codebook-commitment losses, alternating one CoD update per 10 steps with nine fake-score updates.Training took approximately 96 A100 GPU hours, versus 1,664 hours for DMD-based Stable Diffusion distillation.Source: §B.2 · p. 13

B.3. Finetuning to Higher Bitrates

  • Fine-tuned one-step CoD significantly improves performance at higher bitrates after perceptual supervision is restored.The tested variants use 0.0312 bpp and 0.125 bpp; fine-tuning follows initialization and joint fine-tuning stages.Source: §B.3 · p. 13

B.4. Evaluation

One-step CoD combines competitive ultra-low-bitrate reconstruction with fast inference, without relying on pretrained Stable Diffusion components.

B.4. Evaluation

  • 25.2 ms per 512 × 512 image: one-step CoD runs faster than a single Stable Diffusion step used in OneDC and StableCodec.The authors attribute this efficiency partly to removing redundant AdaLN-Zero modules when the timestep is fixed at 0.Source: §B.4 · p. 13
  • One-step CoD achieves competitive reconstruction quality against OneDC and StableCodec, particularly at ultra-low bitrates.Visual comparisons show higher fidelity than both multi-step CoD and OneDC.Source: §B.4 · p. 13
  • The method reaches this performance without pretrained Stable Diffusion components, including one-step models or SD-VAEs.Fixed-length codebook-index coding is used for simplicity; entropy modeling or latent compression could provide further gains.Source: §B.4 · p. 13

C. Towards Perceptual Supervision via CoD

CoD supports several downstream uses, including perceptual supervision and codec variants, with different quality-speed trade-offs. The analyses also identify optimization biases and prediction-target limitations that affect downstream performance.

C. Towards Perceptual Supervision via CoD

  • After 2K decoder-finetuning steps, CoD-based DMD supervision substantially improves MS-ILLM’s perceptual quality across all metrics.The encoder remains fixed, so information encoded under MSE-only optimization stays heavily distorted; native image conditions align more closely with the original image and avoid captioning during optimization.Source: C. Towards Perceptual Supervision via… · p. 14

D.1. Comparison of Downstream Codecs

  • Finetuned one-step CoD attains the best perceptual scores, especially DISTS and FID, while zero-shot DiffC has the highest PSNR and strong perceptual quality.CoD-based MS-ILLM offers a lightweight, substantially faster coding alternative, with slightly worse performance than zero-shot DiffC.Source: §D.1 · p. 14

D.2. Finetuning Multi-step CoD at Higher Bitrates

  • At higher bitrates, jointly optimizing a condition under explicit diffusion loss can bias it toward score-norm maximization, producing oversharpened, oversaturated, lower-fidelity reconstructions.A strong information bottleneck at ultra-low bitrates makes reducing diffusion loss harder and limits this tendency; unified training and auxiliary losses also help counteract it.Source: §D.2 · pp. 14–15

E. Exploring X-Prediction Pixel Diffusion

  • X-prediction improves pixel-space CoD’s perceptual metrics over V-prediction, but does not consistently improve downstream DiffC.Converting predicted x to velocity requires denominator clamping for stability, which can impair likelihood estimation near t = 1 and degrade DiffC metrics, especially at very high bitrates.Source: E. Exploring X-Prediction Pixel… · p. 15

F. Additional Details and Results

  • CoD’s full training process takes 464 A100 GPU hours, approximately 20 days.The training-stage configuration is summarized in Table 3.Source: Table 3 · p. 16

F.1.1 Datasets

  • CoD is trained on 22.1M images from ImageNet-21K, OpenImages V4, and SA-1B, with image resolution as the only filtering criterion.The authors describe this as a relatively modest data scale and expect higher-quality, larger-scale training to improve performance.Source: §F.1.1 · p. 16

F.1.2 Unified Training

Unified post-training and auxiliary supervision address complementary weaknesses in CoD training. The section also reports inference costs, additional evaluations, and visual comparisons across bitrates.

F.1.2 Unified Training

  • Unified post-training (ID = D) achieves the best overall results among the reported configurations.Unified training improves reconstruction fidelity; auxiliary supervision alone yields competitive FID but lower PSNR and LPIPS, while ωα = 3 gives additional gains over the default ωα = 1.Source: §F.1.2 · pp. 16–17, Figure 18 · p. 16

F.1.3 Multi-Stage Training

  • The 256 × 256 and 512 × 512 settings use the same 1024-bit bottleneck, allocated at 0.0156 bpp and 0.0039 bpp, respectively.The encoder downsample ratio changes from 16× to 32× while the codebook size stays unchanged.Source: §F.1.3 · p. 17

F.2. Complexity Analysis

  • One-step pixel-space CoD processes a 512 × 512 image in 25.2 ms and offers faster inference with a smaller parameter footprint than multi-step codecs.Pixel-space CoD avoids expensive VAE decoding, while its decoupled pixel head makes it slightly slower than latent-space CoD.Source: §F.2 · p. 17

F.3. Additional Evaluation Metrics

  • Around 0.004 bpp on Kodak, CoD outperforms existing codecs across the reported subjective and semantic dimensions.The evaluation includes 20 participants’ MOS ratings, CLIP-based similarity, caption similarity, and a VQA benchmark.Source: §F.3 · p. 17, Table 6 · p. 17

F.4. Visual Comparison

  • Across a wide bitrate range, CoD-based DiffC presents higher perceptual quality than other codecs.Figure 19 provides additional visual comparison examples.Source: §F.4 · p. 17, Figure 19 · p. 18

Related papers

Loading 2511.18706v3…