PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Latent Coding for Ultra-Low Bitrate Image Compression

Zhaoyang Jia, Jiahao Li, Bin Li, Houqiang Li, Yan Lu

arXiv:2512.20194v1cs.CVeess.IV

TL;DR

Pixel-space distortion may not align with human perception at low bitrates, making high-realism and high-fidelity compression difficult. GLC codes in a generative VQ-VAE latent space, and reports 45% fewer bits than MS-ILLM at the same FID on CLIC 2020.

  • Problem

    At low bitrates, pixel-space distortion may not align with human perception, complicating compression that preserves semantic consistency and texture realism.

    Source: §1 · p. 1

  • Method

    GLC performs transform coding in a generative VQ-VAE latent space, with categorical hyper coding and code-prediction supervision.

    Source: §1 · pp. 1–2

  • Results

    45% bit savings compared to MS-ILLM at the same FID on CLIC 2020; GLC also uses less than 0.04 bpp on natural images and less than 0.01 bpp on CelebAHQ.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The generative latent space supports applications including image restoration and style transfer.

    Source: §7 · p. 8

    GLC's generalization is not always satisfactory on specified-dataset training; it cannot guarantee clear and accurate reconstruction of screen contents.

    Source: §7 · p. 8

Abstract

from arXiv · show

Most existing image compression approaches perform transform coding in the pixel space to reduce its spatial redundancy. However, they encounter difficulties in achieving both high-realism and high-fidelity at low bitrate, as the pixel-space distortion may not align with human perception. To address this issue, we introduce a Generative Latent Coding (GLC) architecture, which performs transform coding in the latent space of a generative vector-quantized variational auto-encoder (VQ-VAE), instead of in the pixel space. The generative latent space is characterized by greater sparsity, richer semantic and better alignment with human perception, rendering it advantageous for achieving high-realism and high-fidelity compression. Additionally, we introduce a categorical hyper module to reduce the bit cost of hyper-information, and a code-prediction-based supervision to enhance the semantic consistency. Experiments demonstrate that our GLC maintains high visual quality with less than 0.04 bpp on natural images and less than 0.01 bpp on facial images. On the CLIC2020 test set, we achieve the same FID as MS-ILLM with 45% fewer bits. Furthermore, the powerful generative latent space enables various applications built on our GLC pipeline, such as image restoration and style transfer. The code is available at https://github.com/jzyustc/GLC.

1. Introduction

GLC addresses the mismatch between pixel-space distortion and human perception at low bitrates by coding in a generative VQ-VAE latent space. It combines latent transform coding with categorical hyper coding and code-prediction supervision, and reports low-rate results and vision applications.

1. Introduction

  • At low bitrates, pixel-space distortion may not align with human perception, while semantic consistency and texture realism are difficult to exploit through pixel-space transforms alone.Source: §1 · p. 1
  • VQ-VAE latents encode semantic visual components, support realistic texture generation, and provide a sparse, distortion-robust representation for compression.Source: §1 · p. 2
  • 45% bit savings compared to MS-ILLM at the same FID on CLIC 2020; GLC also uses less than 0.04 bpp on natural images and less than 0.01 bpp on CelebAHQ.Source: §1 · p. 2
  • GLC supports image restoration and style transfer; its restoration application surpasses cascading a restoration model with a neural codec.Source: §1 · p. 2
  • GLC performs transform coding in a generative VQ-VAE latent space, using categorical hyper coding and code-prediction supervision to reduce hyper-information bits and enhance semantic consistency.Source: §1 · p. 2
  • GLC is trained in three stages: learning a perception-aligned VQ-VAE latent space, compressing it with latent supervision, then jointly fine-tuning with code-prediction-based pixel supervision.Source: Figure 3 · p. 3

2. Related Works

The related work covers learned pixel-space compression and generative approaches to perceptual quality, then turns to VQ-VAE latent-space modeling and its extension to compression.

2.1. Learned Image Compression

  • Learned image compression developed pixel-space transforms alongside improvements to probability models, network structures, optimization, and rate-variable coding.Source: §2.1 · p. 3
  • Generative compression methods pursue perceptual quality, but sketch- and text-conditioned approaches can deviate from the input; later work explored fidelity, realism, and generative coding, including MS-ILLM [46].Agustsson et al. [2] introduced generative compression; Text+Sketch [35] uses captions and sketches with conditional diffusion. HiFiC [44] pursued high-fidelity compression, while subsequent work explored transform coding [16] [22], post-processing [24], and fidelity-realism tradeoffs [3].Source: §2.1 · p. 3

2.2. Latent Space Modeling

  • VQ-VAE latent-space modeling has supported image generation, including transformer-based generation [17] and diffusion-based high-resolution generation [48].Source: §2.2 · p. 3
  • Latent-space modeling has also been applied to facial restoration and conferencing; this paper explores it for generative image compression through a specialized latent transform-coding paradigm.Source: §2.2 · p. 3
  • Figure 4 compares indices-map coding, transform coding with a factorized hyper module, and the proposed transform coding with a categorical hyper module.Source: Figure 4 · p. 4

3. Method

GLC encodes images in a perception-aligned VQ-VAE latent space, then transform-codes those latents for entropy coding. Its design combines a sparse semantic representation with categorical hyper-information and rate-variable compression.

3.1. Overview

  • Images pass through a latent encoder, analysis transform, scalar quantization, synthesis transform, and latent decoder to produce the reconstruction.The analysis transform produces code y; after quantization, the synthesis transform recovers the latent representation for decoding.Source: §3.1 · p. 4

3.2. Generative Latent Auto-Encoder

  • The generative VQ-VAE maps images to visual semantic elements and provides generative decoding, while its discrete codebook bottleneck yields a sparse, robust latent space.The paper links these properties to semantic consistency and texture realism.Source: §3.2 · p. 4

3.3.1 Categorical Hyper Module

  • Figure 5 compares factorized and categorical hyper modules; categorical z encodes essential semantic and structural information with much less bits.The categorical module uses a hyper codebook to store basic semantic elements rather than low-level information.Source: Figure 5 · p. 4, §3.3.1 · p. 5

3.3. Transform Coding in Latent Space

  • GLC replaces VQ-indices-map coding with transform coding to reduce latent redundancy, then entropy-codes quantized codes using estimated probabilities.The probability estimate uses a categorical hyper module and a quadtree-partition-based spatial context module.Source: §3.3 · p. 4

3.3.2 Rate-Variable Transformation

  • Transform coding supports rate-variable compression by converting latents to a unified Gaussian distribution whose parameters can vary with the rate.GLC incorporates a scaler q to achieve rate-variable compression.Source: §3.3.2 · p. 5

4. Progressive Training

GLC trains a perception-aligned latent space, learns latent transform coding with code-prediction supervision, and then fine-tunes the full network. Code prediction is auxiliary during training rather than part of inference.

4. Progressive Training

  • GLC uses three training stages: learn a perception-aligned latent space, train transform coding on it, then fine-tune the full network with stage-specific losses.The stages target perceptual quality, low bitrate, and improved compression performance, respectively.Source: §4 · p. 5

4.1. Stage I : Auto-Encoder Learning

  • Stage I trains a generative VQ-VAE initialized as encoder E and decoder D, using reconstruction, perceptual, adversarial, and codebook losses.An auxiliary codebook performs nearest vector quantization to promote latent sparsity.Source: §4.1 · p. 5

4.2. Stage II : Transform Coding Learning

  • Stage II fixes the trained auto-encoder and learns latent transform coding with code-prediction supervision and a rate-distortion objective.The code-prediction loss combines index cross-entropy with latent reconstruction error; the objective trades estimated rate against code distortion.Source: §4.2 · p. 5

4.3. Stage III : Joint Training

  • Stage III fine-tunes the full network with pixel-space reconstruction, perceptual, adversarial, and code-prediction losses alongside rate-distortion supervision.The pixel-space code-prediction loss compares input and reconstructed-image latents encoded by the stage-I VQ encoder.Source: §4.3 · pp. 5–6

4.4. Discussion of Code-Prediction-Based Loss

  • GLC uses code prediction only as auxiliary training supervision, not during inference, to avoid a codebook bottleneck and leave the decoder flexible for fine-tuning.The authors report that this supervision enhances semantic consistency in reconstructions.Source: §4.4 · p. 6

5. Experiments

Experiments evaluate natural and facial image compression against traditional, neural, and generative codecs, then assess proposed components through ablations. GLC shows strong benchmark and qualitative results at ultra-low bitrates.

5.1. Implementation Details

  • Natural-image training uses ImageNet, then OpenImage, while facial-image training uses FFHQ; evaluation uses CLIC 2020 and CelebAHQ, respectively.Models train on random 256 × 256 natural-image patches and 512 × 512 facial images, with varying λ for rate-variable compression.Source: §5.1 · p. 6

5.2. Main Results

  • 45% fewer bits than MS-ILLM at equivalent FID on CLIC 2020; on CelebAHQ, GLC outperforms all methods across all metrics.Figure 8 compares methods on both datasets.Source: §5.2 · p. 6, Figure 8 · p. 7
  • Qualitatively, GLC yields high-fidelity, high-realism results where Text+Sketch is low-fidelity, TCM/HiFiC/MS-ILLM blur, and MS-ILLM struggles with facial attributes.These comparisons cover natural and facial image compression at ultra-low bitrates.Source: Figure 9 · p. 7, §5.2 · p. 7

5.3. Ablation Study

  • Ablations report 66.2% loss for indices-map coding, 17.7% improvement from the categorical hyper module, 60.7% drop from inference-time code prediction, and 13.1% gain from auxiliary supervision.The studies evaluate BD-Rate on the FID-BPP curve; Tables 1 and 2 present the ablations.Source: §5.3 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

6. Applications

The GLC pipeline supports restoration and style transfer through alternative encoders and decoders. The restoration application outperforms a cascaded restoration-and-codec scheme without extra model parameters.

Applications

  • A restoration encoder maps distorted images to clean latents, allowing noisy-image compression and clean-image decompression; Table 3 reports better performance than cascading restoration with GLC, without extra model parameters.Figure 10 illustrates the restoration workflow.Source: §6 · p. 8, Table 3 · p. 8, Figure 10 · p. 8
  • A stylization decoder can decode a latent into another style for style transfer.Figure 10 depicts decoding an image into another style.Source: §6 · p. 8, Figure 10 · p. 8

7. Conclusion and Limitation

GLC performs transform coding in a generative VQ-VAE latent domain and reports state-of-the-art performance on several benchmarks. Its generalization is not always satisfactory for codecs trained on specified datasets.

7. Conclusion

  • GLC combines generative VQ-VAE latent-domain transform coding with a categorical hyper module and code-prediction-based supervision for ultra-low-bitrate compression.The paper reports state-of-the-art performance on several benchmarks and develops vision applications on the GLC pipeline.Source: §7 · p. 8

7. Limitation

  • Generalization is not always satisfactory for GLC trained on specified datasets; clear and accurate reconstruction of screen contents is not guaranteed.The authors identify improving generalization through model structure and training strategy as future work.Source: §7 · p. 8

Supplementary Material

The supplementary material describes the GLC network, latent autoencoder, and transform-coding design, with additional results, limitations, and application details. It also reports a perceptual-metric comparison.

Supplementary Material

  • The supplement includes network structure, additional experimental results, discussion of limitations, and application details.Source: Supplementary Material · p. 12

8.1. Generative Latent Auto-Encoder

  • The latent autoencoder uses VQ-VAE designs, with patch attention replacing global attention for high-resolution images; patch attention improves CLIC 2020 performance.Natural images use a VQGAN structure with f = 1/16 and codebook size 16384; facial images use a modified CodeFormer design with f = 1/32 and codebook size 1024.Source: §8.1 · p. 12, Table 4 · p. 12

8.2. Transform Coding in Latent Space

  • Latent transformation converts latent l into code y for quantization, while learned qenc and qdec scalers enable rate-variable compression.The transformation uses cascaded depth-wise blocks and a channel number N = 256.Source: §8.2 · p. 12
  • Figure 11 compares PSNR, MS-SSIM, LPIPS, and DISTS; its example indicates DISTS is a better perceptual reference metric than LPIPS.Source: Figure 11 · p. 12
  • The entropy model estimates quantized-code probabilities using categorical hyper and spatial-context modules; hyper indices are fixed-length coded at log2 Mh bits each.The spatial context module uses the hyper prior and previously decoded parts of the quantized code.Source: §8.2 · p. 12

9. Experiments

The experiments evaluate perceptual quality, rate-variable visual results, and complexity across natural and facial images. They report improved MS-COCO performance and lower BD-DISTS values than comparison methods, alongside metric and evaluation-protocol caveats.

9.1. Perceptual Metrics

  • DISTS, FID, and KID are the primary perceptual metrics because pixel-level metrics can rank a perceptually superior image worse; FID and KID are measured on 256×256 natural-image patches.CelebAHQ FID and KID use all 30,000 512×512 images; Kodak FID and KID are omitted because only 192 patches are generated.Source: §9.1 · pp. 12–13

9.2. Quantitative Results

  • At BPP=0.0289 on Kodak, Text+Sketch is omitted from Figure 18 because its PSNR=11.97dB and MS-SSIM=0.3127 deviate significantly from other curves.Figure 18 reports PSNR and MS-SSIM results for completeness despite their limitations for perceptual evaluation.Source: §9.2 · p. 13
  • Around 0.12 bpp, GLC exhibits significant performance improvement over HFD, PerCo, and MS-ILLM on MS-COCO 30K.The comparison uses selected 2014 validation images as 256×256 patches; Figure 13 presents the results.Source: §9.2 · p. 13, Figure 13 · p. 13

9.3. Visual Results

  • As bitrate increases, GLC improves semantic consistency and produces more intricate textures; visual comparisons report stronger fidelity and realism than other methods.Rate-variable examples are shown in Figure 14; comparisons cover Kodak, CelebAHQ, CLIC2020, and DIV2K.Source: §9.3 · p. 14, Figure 14 · p. 14

9.4. Complexity

  • 0.070 lower BD-DISTS on CelebAHQ with less latency, and 0.047 lower BD-DISTS on Kodak with comparable latency, are reported versus MS-ILLM; Kodak BD-DISTS is 0.140 lower with much less latency versus Text+Sketch.The Text+Sketch caption-generation cost is excluded.Source: §9.4 · p. 14

10. Discussion on Limitations

GLC generalizes strongly on natural and facial images but not consistently across image types; screen images remain a challenging case.

10. Discussion on Limitations

  • GLC’s generalization is not always satisfactory, despite its reported superior performance on natural and facial images.Screen images are identified as a common but significant compression challenge.Source: §10 · p. 14
  • On screen images, GLC produces clearer text than TCM and MS-ILLM but still fails to generate straight background grid lines.The example is shown in Figure 15.Source: §10 · p. 14
  • The authors suggest improving generative latent auto-encoder generalization or using a more suitable GLC training strategy to address the grid-line problem.Source: §10 · p. 14

11. Applications

The GLC pipeline is adapted for restoration and stylization by changing its encoder or decoder. The applications support restoration from noisy inputs and decoding codes into different styles.

11. Applications

  • The section presents restoration and stylization as applications implemented on the GLC pipeline.Source: §11 · p. 14, Figure 16 · p. 16
  • The restoration application encodes distorted images directly into codes and decodes them into restored reconstructions.A restoration encoder maps distorted images to clean latents; the Figure 16 examples use Gaussian noise with σ=20.Source: §11 · pp. 14–15, Figure 16 · p. 16
  • The stylization application replaces the latent decoder with a stylization decoder supervised by content and style losses, enabling codes to decode into different styles.Examples appear in Figure 16; its caption identifies Wikiart and The Scream as style-image sources.Source: §11 · p. 15, Figure 16 · p. 16

Related papers

Loading 2512.20194v1…