PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards image compression with perfect realism at ultra-low bitrates

Marlène Careil, Matthew J. Muckley, Jakob Verbeek, Stéphane Lathuilière

arXiv:2310.10325v2cs.CVeess.IV

TL;DR

Distortion-focused codecs can produce perceptibly unrealistic images at very low bitrates. PerCo decodes with iterative diffusion conditioned on quantized image features and a global description. It reports realistic reconstructions at 0.003 bits per pixel, with FID and KID more stable across bitrates than competing methods.

  • Problem

    At ultra-low bitrates, existing codecs can introduce severe perceptible artifacts, while distortion-focused optimization can compromise realism.

    Source: §1 · p. 1

  • Method

    PerCo uses an iterative diffusion decoder conditioned on a vector-quantized image representation and a global textual image description.

    Source: §1 · pp. 1–2

  • Results

    At 0.003 bits per pixel, PerCo produces realistic reconstructions; its FID and KID are improved at low rates and more stable across bitrates.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    PerCo's reconstructions show that realistic image compression can be maintained at ultra-low rates, with visual quality less dependent on bitrate than in other methods.

    Source: §1 · p. 2, §5 · p. 9

    Evaluation focuses on images up to 768×512, and PerCo has somewhat poorer MS-SSIM, PSNR, and LPIPS performance at rates above 0.01 bpp.

    Source: §5 · p. 9

Abstract

from arXiv · show

Image codecs are typically optimized to trade-off bitrate \vs distortion metrics. At low bitrates, this leads to compression artefacts which are easily perceptible, even when training with perceptual or adversarial losses. To improve image quality and remove dependency on the bitrate, we propose to decode with iterative diffusion models. We condition the decoding process on a vector-quantized image representation, as well as a global image description to provide additional context. We dub our model PerCo for 'perceptual compression', and compare it to state-of-the-art codecs at rates from 0.1 down to 0.003 bits per pixel. The latter rate is more than an order of magnitude smaller than those considered in most prior work, compressing a 512x768 Kodak image with less than 153 bytes. Despite this ultra-low bitrate, our approach maintains the ability to reconstruct realistic images. We find that our model leads to reconstructions with state-of-the-art visual quality as measured by FID and KID. As predicted by rate-distortion-perception theory, visual quality is less dependent on the bitrate than previous methods.

1 Introduction

At very low bitrates, distortion-focused compression can produce perceptibly unrealistic images, while prior diffusion compression work faced limits in image size. PerCo uses a diffusion decoder conditioned on quantized image features and a global description, and reports realistic reconstructions with more rate-stable FID and KID.

1 Introduction

  • Rate-distortion optimization can compromise realism, and prior diffusion compression work by Theis et al. (2022) could not scale beyond 64×64 images because of computational constraints.Perfect realism is theoretically possible with zero f-divergence and FID across rates, while increasing mean-squared error by no more than twofold relative to a rate-distortion-optimal codec.Source: §1 · p. 1
  • At bitrates below 0.04 bpp, PerCo has lower FID and KID than other approaches, with flatter metric curves across bitrates.The authors also report consistent CLIP and mIoU improvements, especially at low rates.Source: §1 · p. 2
  • PerCo uses a diffusion decoder conditioned on a vector-quantized image representation and a textual image description.The description supplies global semantic context alongside the quantized image representation.Source: §1 · p. 2
  • At 0.003 bits per pixel, PerCo produces realistic reconstructions, and reports state-of-the-art FID and KID on MS-COCO 30k with no significant FID degradation as bitrate falls.Figure 1 compares Kodak reconstructions at each method's lowest available bitrate.Source: §1 · p. 2, Figure 1 · p. 2

2 Related work

Related work spans neural codecs optimized for rate-distortion, methods adding perceptual or adversarial objectives, and diffusion-based compression. PerCo is especially related to low-rate text-conditioned diffusion approaches, but uses captioning and learned vector-quantized image features as conditioning.

2 Related work

  • Neural codecs progressed from outperforming JPEG on rate-distortion to hyperprior and latent-conditioning improvements (Ballé et al., 2017; Ballé et al., 2018; Minnen et al., 2018; Minnen & Singh, 2020; Cheng et al., 2020; He et al., 2022a).Rate-distortion optimization alone can create distributional mismatch, while adversarial and perceptual losses have been used to improve human-perceptual quality (Blau & Michaeli, 2018, 2019; Agustsson et al., 2019; Mentzer et al., 2020).Source: §2 · p. 2
  • Global textual or visual decoder conditioning can function as side information in conditional perceptual quality (Xu et al., 2023).The paper relates this framing to its global representation.Source: §2 · p. 3
  • Diffusion compression work explored reverse channel coding, residual diffusion, refinement without additional bits, and jointly trained diffusion decoders across varied image settings (Ho et al., 2020; Theis et al., 2022; Ghouse et al., 2023; Hoogeboom et al., 2023; Yang & Mandt, 2023).Reported settings include 32×32 CIFAR-10, 64×64 ImageNet, and higher-quality codec evaluations.Source: §2 · p. 3
  • For ultra-low-rate text-conditioned compression, Lei et al. (2023) use prompt inversion and binary contour maps, while PerCo uses fast captioning and learned vector-quantized image features.The paper characterizes contour maps as carrying little appearance information.Source: §2 · p. 3

3 Perceptual compression with a diffusion decoder

PerCo pairs a quantized local image representation with global image context and uses a conditioned diffusion model to reconstruct images in latent space. Its framework fixes the coding rate during training and uses a diffusion-based objective in place of conventional distortion.

3.1 Overall framework

  • PerCo fixes its rate through uniform coding, allowing the rate term to be dropped while a diffusion-based formulation replaces the conventional distortion term.The rate is controlled by the number of quantized elements and codebook size.Source: §3.1 · p. 3
  • The diffusion model iteratively denoises in latent space, and training minimizes an estimate of the previous diffusion step from the current latent and image code.The loss is averaged over diffusion steps and uses v prediction; an image-level loss such as LPIPS may be added.Source: §3.1 · p. 4

3.2 Encoding local and global context

  • At the target bitrate, a hyper encoder reduces LDM features to an h×w tensor, vector-quantizes it into local codes, and upsamples the codes for denoiser conditioning.The encoder projects LDM features to a lower spatial resolution before quantization.Source: §3.2 · pp. 4–5
  • Global context comes from a fixed image-captioning model; its caption is losslessly compressed, while global image features are also explored as an alternative.User-generated captions can also be used.Source: §3.2 · p. 5
  • The denoiser receives local codes by concatenation and text through cross-attention; inference-time classifier-free guidance contrasts local-only with local-and-text conditioning.Training drops text conditioning in 10% of iterations to enable classifier-free guidance.Source: §3.2 · p. 5

4 Experiments

PerCo is evaluated against image codecs on Kodak and MS-COCO, with the main results showing stronger realism and semantic metrics at low bitrates. Ablations examine conditioning choices, quantization, and denoising steps, revealing trade-offs between realism and distortion.

4.1 Experimental setup

  • Evaluation uses Kodak and MS-COCO at 256×256 and 512×512, with FID/KID, distortion, semantic-alignment metrics, and representative handcrafted, neural, and diffusion codec baselines.Kodak reports only LPIPS and MS-SSIM because its 24 images are insufficient for reliable FID/KID and it lacks semantic annotations.Source: §4.1 · pp. 5–6

4.2 Main results

  • Below 0.04 bpp, PerCo achieves lower FID and KID and flatter realism curves, improves CLIP and mIoU, and yields the best FID at both Figure 4 rates.For LPIPS and MS-SSIM, it improves over Text-Sketch but is worse than some other methods; the authors caution these similarity metrics may be less meaningful at low rates.Source: §4.2 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6
  • Qualitative comparisons show PerCo reconstructions with fewer artifacts, more faithful colors, and realistic details; text-only Text-Sketch lacks spatial alignment.VTM and MS-ILLM outputs are described as blurry or blocky, while Text-Sketch can strongly deviate from the originals in color.Source: §4.2 · pp. 6–7, Figure 5 · p. 7

4.3 Ablations

  • Lower-bitrate reconstructions show more diversity, while denoising-step effects vary: at low rates, more steps improve FID but reverse the MS-SSIM advantage of fewer steps.At higher bitrates, metrics remain nearly constant across denoising steps; the evaluation spans seven bitrates from 0.0032 to 0.1263 bpp.Source: §4.3 · pp. 7–9, Figure 6 · p. 8, Figure 8 · p. 9
  • Spatial-only conditioning performs worse on FID and CLIP than other variants, while its LPIPS and mIoU remain comparable because local features support alignment.Text-only conditioning has worse LPIPS and mIoU than variants with spatial conditioning, while its CLIP scores align with models using both modalities.Source: §4.3 · pp. 7–8
  • Applying classifier-free guidance to textual conditioning only gives the best results, with significantly better FID and mIoU than the other tested guidance forms.The alternatives omit guidance or apply it to both text and spatial feature maps.Source: §4.3 · p. 8
  • PerCo without quantization performs roughly like the LDM autoencoder, indicating that quantization is the main performance constraint and may also cause suboptimal training through noisy gradients.The LDM autoencoder significantly improves all metrics in the comparison.Source: §4.3 · p. 8, Table 1 · p. 9
  • Global image embeddings are competitive at 3.5× lower bitrate than BLIP captions, but slightly worsen high-rate LPIPS and MS-SSIM; BLIP and ground-truth captions yield similar results.IDEFICS captions yield similar FID and somewhat better CLIP than BLIP, but their longer encoding cost accounts largely for worse LPIPS and mIoU.Source: §4.3 · pp. 7–9

5 Conclusion

PerCo pairs a VQ-VAE-like encoder with a diffusion-based decoder and textual image descriptions, producing realistic reconstructions at very low bitrates. The study also notes reconstruction-performance and resolution limits, and cautions against practical deployment without bias analysis.

5 Conclusion

  • PerCo combines a VQ-VAE-like encoder, diffusion-based decoder, and textual image descriptions, with realistic reconstructions reported down to 0.003 bits per pixel.The conclusion also reports improved semantics-driven metrics and more stable FID and KID across bitrates.Source: §5 · p. 9
  • The study evaluates images only up to 768×512 and reports poorer MS-SSIM, PSNR, and LPIPS reconstruction performance than existing approaches in some settings.The authors attribute this at least partly to the LDM autoencoder and the distortion-perception tradeoff, leaving detailed study for future work.Source: §5 · p. 9

Ethics statement

  • The authors advise research-only use until potential biases are extensively analyzed, since model performance may favor data similar to its training data.The concern applies to machine-learning models generally and image compression in particular.Source: Ethics statement · p. 10

A Experimental details

The experiments evaluate Kodak and MS-COCO images, vary spatial and textual bitrate allocations, and use a pretrained latent diffusion model with selected finetuning and denoising settings. The appendix also documents baseline evaluation sources and the runtime tradeoffs of diffusion decoding.

A Experimental details

  • Evaluation includes Kodak images at 512×768 pixels and the MS-COCO 2014 validation set, whose ground-truth segmentation maps and descriptive captions enable semantics-oriented mIoU and CLIP score metrics.Source: A Experimental details · p. 12
  • For MS-ILLM and MS-PatchGAN, additional-low-bitrate checkpoints came from the authors; HFD/DDPM, DIRAC, and Multi-Realism paper results came through personal communication, while HifiC used available GitHub checkpoints.Source: A Experimental details · p. 12
  • Training uses a pretrained 1.4B-parameter latent diffusion model with fixed LDM autoencoder weights and 15% of diffusion weights finetuned.The hyper-encoder is trained for five epochs; above 0.05 bpp, an LPIPS loss is added, requiring a smaller batch size.Source: A Experimental details · p. 12
  • The diffusion model is finetuned on 50 timesteps, then sampled with 20 denoising steps below 0.05 bpp and five steps at higher rates.Inference uses deterministic DDIM sampling with σ_t fixed to zero.Source: A Experimental details · p. 12
  • Target total bitrate is set by accounting for caption bitrate and adjusting spatial grid size and codebook size; codebooks below 64 elements harmed training.For 512×512 COCO images, BLIP captions average 0.00219 bpp and IDEFICS captions 0.00622 bpp; Table 2 lists spatial-code configurations.Source: A Experimental details · p. 13, Table 2 · p. 13
  • PerCo decoding averages 0.67 seconds with five steps and 2.54 seconds with 20 steps per Kodak image; the five-step decoder is about eight times slower than HiFiC and MS-ILLM.The reported speed comparison is in Table 3; encoding complexity is described as similar to those adversarial-based codecs.Source: A Experimental details · p. 13, Table 3 · p. 13

B.1 Quantitative results

The quantitative appendix compares PSNR across codecs, reports an LPIPS-loss ablation, and explains an additional CLIP image-similarity metric used alongside caption-based CLIP scores.

B.1 Quantitative results

  • PerCo’s PSNR is higher than Text-Sketch’s but lower than MS-ILLM’s on Kodak and COCO; the LDM autoencoder is comparable to MS-ILLM at 0.08 bpp and VTM at 0.04 bpp.The appendix uses Figure 9 for Kodak and COCO PSNR comparisons and notes that the LDM autoencoder’s PSNR may explain PerCo’s lower values across bitrates.Source: §B.1 · pp. 13–14, Figure 9 · p. 14
  • Adding LPIPS loss produces small improvements in both LPIPS and FID at higher bitrates.Figure 12 presents this ablation for training the diffusion decoder.Source: §B.1 · p. 14
  • Caption-based CLIP scores saturate at 30.4 on real images, motivating an image-embedding similarity metric whose real-image threshold is 1.Figure 10 reports evaluation using the CLIP image metric.Source: §B.1 · p. 14, Figure 10 · p. 14

B.2 Qualitative results

Qualitative comparisons show that textual conditioning matters more when spatial bitrate is low, while image quality remains fairly stable across tested rates. The appendix also contrasts PerCo’s perceptual appearance with pixelwise metrics and compares it with other codecs at low and high rates.

B.2 Qualitative results

  • At low spatial bitrates, changing text can alter scene semantics and omitting text substantially worsens image quality; at higher spatial bitrate, text has minimal influence.Figure 13 shows examples at 0.0019 and 0.1250 spatial bpp, using the same initial noise for samples at a given rate.Source: §B.2 · p. 14, Figure 13 · p. 17
  • Across targeted spatial rates from 0.0046 to 0.1279 bpp, PerCo image quality is described as quite stable, while similarity to the original increases with bitrate.Figure 14 illustrates reconstructions of two MS-COCO 30k images.Source: §B.2 · pp. 14, 16, Figure 14 · p. 18
  • The LDM autoencoder produces detailed images, but its reconstruction fidelity is lower than MS-ILLM’s at higher bitrates; at similar PSNR, its details can appear sharper.On Kodak, the autoencoder scores 0.0775 LPIPS, 0.9131 MS-SSIM, and 25.6003 PSNR; Figure 11 illustrates the comparison.Source: Figure 11 · p. 15, §B.2 · p. 16
  • At 0.0031 bpp, PerCo appears sharper and more realistic than MS-ILLM at 0.0065 bpp despite worse PSNR, MS-SSIM, and LPIPS values.The example suggests these pixelwise metrics do not align well with human preference at very low bitrates; Figure 15 shows the comparison.Source: §B.2 · p. 16, Figure 15 · p. 19
  • At 0.1264 bpp, PerCo reconstructs textures comparably to HiFiC and MS-ILLM and shows fewer artifacts than VTM.Figure 16 compares reconstructions at the highest bitrate range evaluated.Source: §B.2 · p. 16, Figure 16 · p. 20

Related papers

Loading 2310.10325v2…