Source-linked AI summary
One-Step Diffusion-Based Image Compression with Semantic Distillation
Naifu Xue, Zhaoyang Jia, Jiahao Li, Bin Li, Yuan Zhang, Yan Lu
TL;DR
Diffusion-based codecs can improve perceptual realism, but iterative sampling adds decoding overhead and reconstructions may lose fidelity. OneDC combines latent compression with one-step diffusion, hyperprior semantic guidance, semantic distillation, and hybrid-domain optimization. It reports over 39% bitrate reduction and 20× faster decoding than existing multi-step diffusion codecs.
Problem
Diffusion codecs can improve perceptual realism but may compromise fidelity and decode slowly because of iterative sampling; text guidance can also miss fine-grained or localized semantics.
Source: §1 · p. 1Method
OneDC combines a latent compression module with a one-step diffusion generator, using hyperprior features distilled from a pretrained tokenizer for semantic guidance and hybrid pixel- and latent-domain training.
Source: §1 · pp. 2–3Results
Over 39% bitrate reduction and 20× faster decoding than existing multi-step diffusion codecs.
Source: §1 · p. 3Takeaways & Limitations
Experiments demonstrate the potential of one-step diffusion for generative image compression.
Source: §1 · p. 3OneDC decoding speed does not meet real-time requirements.
Source: §5 · p. 10
Abstract
from arXiv · showhide
While recent diffusion-based generative image codecs have shown impressive performance, their iterative sampling process introduces unpleasing latency. In this work, we revisit the design of a diffusion-based codec and argue that multi-step sampling is not necessary for generative compression. Based on this insight, we propose OneDC, a One-step Diffusion-based generative image Codec -- that integrates a latent compression module with a one-step diffusion generator. Recognizing the critical role of semantic guidance in one-step diffusion, we propose using the hyperprior as a semantic signal, overcoming the limitations of text prompts in representing complex visual content. To further enhance the semantic capability of the hyperprior, we introduce a semantic distillation mechanism that transfers knowledge from a pretrained generative tokenizer to the hyperprior codec. Additionally, we adopt a hybrid pixel- and latent-domain optimization to jointly enhance both reconstruction fidelity and perceptual realism. Extensive experiments demonstrate that OneDC achieves SOTA perceptual quality even with one-step generation, offering over 39% bitrate reduction and 20x faster decoding compared to prior multi-step diffusion-based codecs. Project: https://onedc-codec.github.io/
1 Introduction
Diffusion codecs improve perceptual realism but can compromise fidelity and incur substantial decoding overhead; OneDC investigates whether one-step sampling can retain quality while reducing latency. It combines latent compression, semantic hyperprior guidance, and hybrid-domain training, with semantic distillation to strengthen the guidance.
1 Introduction
- Diffusion codecs improve perceptual realism but may deviate from the input and decode substantially slower than conventional VAE-based codecs.Source: §1 · p. 1
- Compressed latents retain coarse structures, motivating the hypothesis that a carefully designed one-step decoder can reconstruct high-frequency content without multi-step sampling.Source: §1 · p. 2
- OneDC combines compact latent coding with one-step diffusion to synthesize high-frequency details, using pixel-domain perceptual loss and latent-domain diffusion distillation to balance fidelity, realism, and efficiency.Source: §1 · pp. 2–3
- Text guidance can miss fine-grained or localized semantics and require costly captioning, so OneDC uses hyperprior features and distills semantic knowledge from a pretrained tokenizer.Source: §1 · pp. 2–3
- OneDC achieves SOTA performance, with over 39% bitrate reduction and 20× faster decoding than existing multi-step diffusion codecs.Source: §1 · p. 3
2 Related Work
Prior learned codecs improve rate-distortion performance but can lose perceptual detail at low bitrates, while generative approaches pursue more realistic reconstructions. Diffusion codecs and one-step generators offer relevant tools, but one-step diffusion for generative compression remains underexplored.
VAE-based Learned Image Compression
- Learned VAE codecs advanced rate-distortion performance beyond VVC-Intra, but distortion-oriented objectives can over-smooth textures at low bitrates.Source: §2 · p. 3
VAE-based Generative Image Compression
- GAN-based codecs such as Agustsson et al. [2] and HiFiC [41] use perceptual and adversarial training to synthesize high-frequency details.Source: §2 · p. 3
- Other VAE-based methods use local likelihoods, semantic feedback, or generative tokenizers, yet extremely low-bitrate quality remains limited by model capacity and training scale.Source: §2 · p. 4
Diffusion-based Generative Image Compression
- Diffusion codecs improve perceptual realism at ultra-low bitrates, but often struggle with fidelity and incur substantial overhead from multi-step sampling.Source: §2 · p. 4
3 Methodology
OneDC combines compact latent compression with a one-step diffusion generator, using hyperprior semantics and hybrid pixel- and latent-domain training to balance reconstruction fidelity and perceptual realism.
3.1 Framework Overview
- OneDC encodes images into compact latents, derives semantic guidance from the quantized hyperprior, and refines the initial latent with a one-step diffusion generator before decoding the reconstruction.The hyperprior semantic decoder supplies context to the generator’s cross-attention layers.Source: §3.1 · p. 5
3.2.1 From Text to Hyperprior
- OneDC replaces text prompts with a categorical hyperprior as diffusion guidance, providing spatially aligned semantics that can be optimized jointly with the diffusion model.Finite scalar quantization compresses the hyperprior; its semantic context replaces textual embeddings in cross-attention.Source: §3.2.1 · pp. 5–6
3.2.2 Semantic Distillation for Hyperprior
- Semantic distillation transfers knowledge from a pretrained generative tokenizer to the hyperprior codec through an auxiliary token-prediction task, improving semantic encoding and reconstruction quality.The predictor and tokenizer encoder are used only during training, so they add no inference overhead.Source: §3.2.2 · pp. 6–7
3.3 Training Strategy
- Training proceeds in two stages: Stage I learns compression, semantic distillation, and detail synthesis; Stage II fine-tunes diffusion with pixel- and latent-domain objectives while fixing the compression module.The hybrid strategy combines pixel-domain perceptual supervision with latent-domain diffusion distillation and adversarial alignment.Source: §3.3 · p. 7
4 Experiment
OneDC is evaluated against learned and diffusion-based codecs across datasets, resolution settings, and reconstruction criteria. It delivers strong perceptual quality with one-step decoding, while ablations show the roles of semantic guidance and hybrid-domain training.
4.1 Implementation and Evaluation Settings
- OneDC is evaluated against VAE and diffusion codecs on Kodak, CLIC2020, and MS-COCO 30K using perceptual, traditional distortion, realism, and bitrate-saving metrics.The generator uses a pretrained one-step model; training includes 512- or 1024-pixel crops, and FID uses dataset-specific patch or full-image protocols.Source: §4.1 · pp. 8–9
4.2 Main Results
- 55.27% bitrate saving on Kodak and 54.60% on CLIC2020 in LPIPS versus DiffC; OneDC also leads LPIPS and DISTS across datasets.It achieves lower FID than multi-step diffusion baselines on CLIC2020 and 39.55% bitrate saving over PerCo (SD) on MS-COCO 30K.Source: §4.2 · p. 9
- At the lowest bitrate, OneDC produces the most natural and faithful reconstructions, while MS-ILLM is blurry and multi-step methods show structural distortions.Similar visual trends are reported on MS-COCO 30K.Source: §4.2 · p. 9
4.3 Ablation Studies and Efficiency Evaluation
- Removing semantic guidance degrades reconstruction quality by up to 40%; hyperprior guidance outperforms text guidance on high-resolution CLIC2020, and semantic distillation adds gains.The ablation evaluates BD-Rate on CLIC2020; the reported table summarizes BD-Rate (%).Source: §4.3 · pp. 9–10, Table 1 · p. 9
- Pixel-only training produces gridlike artifacts, while latent-only training degrades all metrics and causes color shifts, supporting the complementary roles of both domains.Figure 9 shows artifacts without latent-domain training and color shift without pixel-domain supervision.Source: §4.3 · p. 10, Figure 9 · p. 10
- OneDC decodes over 20× faster than multi-step diffusion methods, but is slightly slower than the pure VAE-based MS-ILLM.The comparison reports coding time and BD-Rate; the paper attributes OneDC’s added capacity to preserving perceptual quality under extreme compression.Source: Table 2 · p. 9, §4.3 · p. 10
5 Conclusion
OneDC combines latent compression with a one-step diffusion generator, using hyperprior guidance and two-stage training to target efficient, high-quality ultra-low-bitrate compression. Its decoding speed improves on multi-step diffusion methods but does not meet real-time requirements.
5 Conclusion
- OneDC combines a latent compression module with a one-step diffusion generator for efficient coding and fast reconstruction under ultra-low bitrates.The hyperprior supplies semantic guidance, enhanced through distillation; Stage I uses pixel-domain supervision and Stage II hybrid-domain perceptual learning.Source: §5 · p. 10
- OneDC achieves state-of-the-art perceptual quality with fast decoding, supporting the potential of one-step diffusion for generative image compression.Source: §5 · p. 10
- Despite speedups over multi-step diffusion methods, OneDC’s decoding speed does not meet real-time requirements.The authors identify model distillation and architectural optimization as future efficiency work.Source: §5 · p. 10
A Experiment
Appendix evaluations report strong OneDC performance across additional datasets, metrics, and evaluation settings. They also examine pixel-level fidelity, semantic distillation, bitrate allocation, and computational costs.
A Experiment
- The comparison uses official checkpoints or implementations where available; some results are taken from published papers, and multi-step codecs use their default sampling settings.Source: A Experiment · p. 16
- For full-resolution CLIC2020 and DIV2K, FID uses overlapping 256 × 256 patches; for MS-COCO 30K it uses full images, while resized settings use dataset-specific patch sizes.The authors also report results using DiffEIC’s 256 × 256 FID patch protocol for resized CLIC2020.Source: A Experiment · p. 16
- Across datasets, metrics, and evaluation settings, OneDC consistently achieves state-of-the-art performance, including additional DIV2K and CLIC2020 evaluations.The CLIC2020 results also include a resize-and-center-crop evaluation following DiffEIC’s 256-FID protocol.Source: A Experiment · p. 16
- OneDC’s PSNR is slightly lower than MS-ILLM, but it matches MS-ILLM on MS-SSIM and outperforms other diffusion methods on that metric.The paper notes that PSNR optimization at extremely low bitrates can suppress high-frequency detail and produce blur.Source: A Experiment · p. 16
- Across Kodak, CLIC2020, and MS-COCO 30K visual comparisons, OneDC is reported to deliver superior quality at the lowest bitrate cost.Source: A Experiment · p. 16
- Additional COCO2017 reconstructions support that semantic distillation transfers richer semantic information to the hyperprior-conditioned model.The evaluation replaces textual conditioning with hyperprior features and reconstructs using only the hyperprior signal.Source: A Experiment · p. 16
- With the hyperprior budget fixed at 0.0035 bpp, increasing latent bitrate improves fidelity and perceptual quality; LPIPS drops from 0.290 to 0.119.At zero latent bits, the codec still produces coherent reconstructions, indicating that the hyperprior supplies semantic information.Source: A Experiment · p. 16, Table 4 · p. 17
- Under resize-and-center-crop on 512 × 512 Kodak images, PerCo (SD) uses about 22,220 MB, DDCM 4,186 MB, and OneDC 8,038 MB; OneDC decodes over 20× faster than other diffusion-based codecs.The passage attributes PerCo’s substantially higher computational burden to incorporating a large language model; OneDC uses more memory than DDCM, while its single-step design reduces inference cost versus both multi-step PerCo and DDCM.Source: A Experiment · p. 17
B Training Details
Training proceeds in two stages: Stage I trains the compression module and fine-tunes the one-step diffusion model, while Stage II further aligns generator outputs with real-image distributions. The stages combine reconstruction, rate, auxiliary prediction, distillation, and adversarial objectives.
Stage I training
- Stage I minimizes pixel and perceptual reconstruction losses, a rate penalty, and an auxiliary code-prediction loss.The reconstruction term combines L1 and LPIPS; the auxiliary loss is weighted by α = 0.001.Source: B Training Details · p. 17
Training setup
- Stage I runs for 800,000 steps and about 6 days; Stage II runs for 1,000,000 steps and about 12 days on 4×A100 GPUs.Stage I uses AdamW with a three-stage learning-rate schedule; Stage II fixes the generator, fake-network, and discriminator learning rate at 1e-6.Source: B Training Details · pp. 17–18
Stage II training
- Stage II fine-tunes the one-step generator with diffusion-distillation, reconstruction, and latent-space adversarial losses.The objective weights reconstruction by β = 0.625 and adversarial loss by γ = 0.001.Source: B Training Details · pp. 17–18
- Diffusion distillation minimizes expected KL divergence between time-dependent target and generator distributions, using a gradient based on their score-function difference.The fake score network is trained with a denoising loss to track the one-step generator’s evolving distribution.Source: B Training Details · p. 18
- The fake branch and discriminator each receive 10 updates per generator update, with σ = 0.01 for adversarial optimization.The discriminator distinguishes features extracted from real and generated images.Source: B Training Details · p. 18
C Model Details
The model combines diffusion-aligned image features and semantic features in its compression architecture, then adapts the pretrained diffusion U-Net and uses a Swin-based auxiliary predictor.
Model architecture
- The analysis transform extracts pretrained diffusion-model VAE features, incorporates original-input embeddings, and introduces a lightweight multi-scale U-Net to learn compressible latents and capture semantics, integrating semantic features into the hyperprior branch.Source: C Model Details · p. 19
Parameter-efficient adaptation
- LoRA layers of rank 64 adapt the pretrained diffusion U-Net; encoder and decoder components total 394M parameters, while the adapted U-Net adds 928M.The additional U-Net parameters comprise about 860M for SD1.5 and 68M for LoRA.Source: C Model Details · p. 19
Auxiliary code predictor
- The paper says further details of the semantic distillation mechanism appear in Fig. 15, and adopts a Swin Transformer [37] to improve adaptability across different image resolutions during training.Source: C Model Details · p. 19
D Social Impact
The paper frames generative compression as a potential efficiency benefit for visual media, while noting that synthesized content can diverge from the input. Supplementary examples compare reconstruction quality, bitrate, and semantic guidance.
Positive Aspects
- Generative codecs can reduce storage and transmission demands for high-resolution images, with potential benefits in bandwidth-limited settings and social communication.The paper also identifies personal entertainment as a possible application.Source: D Social Impact · p. 19
Model details
- The supplementary diagrams identify an SD 1.5 pretrained VAE encoder and a Swin auxiliary predictor with window size 16 and head dimension 64.The predictor diagram labels normal and shifted windows as W and SW.Source: Figure 14 · p. 20, Figure 15 · p. 20
Negative Aspects
- Synthesized reconstructions may deviate from the original, raising concerns about fidelity and authenticity.The passage points to multi-realism codecs as a possible way to control the realism–fidelity trade-off.Source: D Social Impact · p. 20
Visual comparisons
- On the Kodak examples, DiffC requires at least 1.4× bitrate; on the CLIC2020 768 × 768 examples, it requires ≥1.5 higher bitrate while producing slightly lower reconstruction quality than OneDC.MS-COCO examples report that PerCo requires ≥1.4 bitrate over OneDC but still has suboptimal fidelity.Source: Figure 16 · p. 21, Figure 18 · p. 23, Figure 19 · p. 24
Semantic guidance
- The semantic-guidance comparison reports that hyperprior guidance is more faithful than text prompts, and semantic distillation further improves object-level accuracy.The text-prompt example shows severe distortions; the comparison uses a pretrained one-step diffusion model [66].Source: Figure 20 · p. 25