Source-linked AI summary
High-Fidelity Generative Image Compression
Fabian Mentzer, George Toderici, Michael Tschannen, Eirikur Agustsson
TL;DR
Lossy compression must preserve visual similarity as bitrate falls, but existing objectives and GAN-based methods expose trade-offs between distortion, perceptual quality, and fidelity to the input. HiFiC combines learned compression with conditional GAN training, studies its architectural and perceptual components, and evaluates the result with diverse metrics and a user study. The method produces high-fidelity reconstructions for high-resolution images and is preferred to previous approaches even when those approaches use more than 2× the bitrate.
Problem
Existing compression methods degrade images as compression increases, while earlier GAN compression preserved high-level semantics but deviated significantly from the input.
Method
The paper combines learned compression with a conditional GAN and studies normalization, network architectures, training strategies, and perceptual losses using multiple evaluation metrics.
Results
HiFiC produces high-fidelity reconstructions for images up to 2000×2000 pixels and is visually preferred to previous approaches even when they use more than 2× the bitrate.
Takeaways & Limitations
FID and KID can guide architecture exploration, but no existing metric perfectly orders arbitrary models according to the user study.
Takeaways & Limitations
Small-scale text can appear typeset in another script, and small faces can exhibit high-frequency noise.
Abstract
from arXiv · showhide
We extensively study how to combine Generative Adversarial Networks and learned compression to obtain a state-of-the-art generative lossy compression system. In particular, we investigate normalization layers, generator and discriminator architectures, training strategies, as well as perceptual losses. In contrast to previous work, i) we obtain visually pleasing reconstructions that are perceptually similar to the input, ii) we operate in a broad range of bitrates, and iii) our approach can be applied to high-resolution images. We bridge the gap between rate-distortion-perception theory and practice by evaluating our approach both quantitatively with various perceptual metrics, and with a user study. The study shows that our method is preferred to previous approaches even if they use more than 2x the bitrate.
1 Introduction
Learned compression can reduce storage while preserving images, but increasing compression often causes artifacts or reconstructions that diverge from the input. This paper proposes a GAN-based method targeting high-fidelity, high-resolution reconstructions and evaluates it with perceptual metrics and a user study.
- Increasing compression degrades images through blocking, banding, checkerboard, poor text, or blurry reconstructions, depending on the algorithm or training metric.Classical methods exhibit algorithm-specific artifacts, while learning-based methods can exploit weaknesses in distortion metrics.
- GAN-based compression can produce perceptually convincing reconstructions at extremely low bitrates, but earlier results preserved mainly high-level semantics rather than input details.The cited prior method operated below 0.08 bpp, with reconstructions deviating significantly from the input.
- Rate-distortion-perception theory formalizes a trade-off in which, at fixed rate, better perceptual quality implies worse distortion, while empirical exploration had remained limited beyond toy datasets.The theory distinguishes pairwise image distortion from distributional perceptual quality.
- The study combines FID, KID, NIQE, LPIPS, PSNR, and MS-SSIM to assess perceptual quality and distortion, while separately examining architecture, training, normalization, and loss choices.The authors report that no single metric predicts the complete user-study ranking, although FID and KID guide exploration.
2 Related work
Prior image compression research progressed from hand-crafted codecs to neural auto-encoders and improved latent probability models. GANs brought advances in realistic image generation and enhancement, motivating their use for perceptual compression.
- JPEG, WebP, JPEG2000, and BPG represent successive hand-crafted approaches, with BPG achieving high PSNR across varying bitrates.
- Neural compression methods use auto-encoders and increasingly accurate latent probability models, including hierarchical priors and autoregressive contexts, to reduce bitrate.
- State-of-the-art neural compression models had surpassed BPG in PSNR before this work.
- GAN progress in high-resolution photorealistic generation was driven by larger data and models, architectural innovations, and normalization techniques that stabilize training.
- Adversarial losses had also advanced image-enhancement tasks such as compression artifact removal and denoising.
3 Method
The method combines learned rate-distortion compression with a conditional GAN, using the quantized latent as conditioning and balancing bitrate, distortion, and adversarial objectives. Its architecture and training design address rate control, quantization consistency, discriminator conditioning, and resolution-related normalization artifacts.
- 3.1 Background: A learned compressor encodes an image into a quantized latent, decodes it into a reconstruction, and entropy-codes the latent using a probability model.
- 3.2 Formulation and Optimization: HiFiC merges learned compression with a conditional GAN, setting y = E(x) and s = y so the discriminator receives the compressed representation as conditioning.
- 3.2 Formulation and Optimization: The training objective combines bitrate, MSE, LPIPS perceptual distortion, and an adversarial term, with λ and β controlling trade-offs among them.The pointwise distortion is d = k_M MSE + k_P d_P, where d_P = LPIPS.
- 3.2 Formulation and Optimization: A rate-target parameter and adaptive λ′ are introduced because MSE, perceptual distortion, and the adversarial term all compete with bitrate.
- 3.2 Formulation and Optimization: The architecture uses a hyperprior probability model, uniform-noise quantization for probability estimation, and rounding with a straight-through estimator when generating reconstructions.
- 3.3 Architecture: The discriminator is single-scale, uses SpectralNorm, and is conditioned on an upscaled version of the latent; ChannelNorm replaces InstanceNorm to avoid resolution-dependent darkening artifacts.
- 4 Evaluation: The user study compares method pairs against the original image using selected 768×768 crops and Elo scores aggregated over 10,000 shuffled comparison orders.
4 Experiments
The experiments evaluate HiFiC and baselines across diverse datasets using distortion and perceptual metrics, alongside a user study comparing methods at matched or higher bitrates. The evaluation also examines metric construction and the effects of GAN training.
- Evaluation setup: HiFiC, M&S Hyperprior, BPG, and a no-GAN baseline are compared across PSNR, LPIPS, MS-SSIM, NIQE, FID, and KID.The user study compares nine methods, including three HiFiC operating points, matched M&S models, two BPG points, and a no-GAN baseline.
- Evaluation setup: FID and KID measure distributional similarity, whereas PSNR, MS-SSIM, and LPIPS measure similarity between individual image pairs.FID uses Inception features and Gaussian Wasserstein distance; KID is unbiased and makes no feature-distribution assumptions.
- Evaluation setup: FID and KID are computed on 256-pixel patches for CLIC2020 and DIV2K, yielding 28 650 and 6 573 patches respectively.These metrics are not reported for Kodak because its 24 images yield only 192 patches.
- User study: At 0.237 bpp, HiFiC is preferred to BPG at 2.1× the bitrate and to MSE-optimized models at 1.7× the bitrate.Figure 3 normalizes human and perceptual scores so lower values are better, and reports average bits per pixel below each method.
- Model variants: The no-GAN baseline keeps HiFiC’s architecture and distortion loss, isolating the effect of GAN training in the experimental comparison.The M&S baseline is an MSE-trained Mean & Scale Hyperprior model using the same probability model and training schedule as HiFiC.
- User study: In the user study, 14 participants ranked nine methods, with BPG evaluated at 1.5× and 2× the bitrate of HiFiCMi.M&S comparisons select reconstructions using at least as many bits as the corresponding HiFiC image.
5 Results
HiFiC achieves strong perceptual quality across bitrates, with user preferences favoring it over competing methods even at substantially higher rates. Quantitative studies show that GANs improve perceptual metrics while worsening distortion, and that architecture and training choices materially affect results.
- User Study: HiFiCLo at 0.120 bpp achieves similar ELO scores to BPG at 0.390 bpp and M&S Hyperprior at 0.405 bpp.These correspond to 3.3× and 3.4× the bitrate, respectively.
- User Study: HiFiCMi at 0.237 bpp is preferred to BPG at 0.504 bpp and to M&S Hyperprior at 1.7× the bitrate.The BPG comparison uses more than twice the bits.
- User Study: No existing perceptual metric reproduces the full human ranking, although FID and KID correctly rank models with fixed architecture and distortion losses.The user-study analysis also finds consistent trends across participants and images, with some close methods changing order.
- Quantitative Results: All perceptual metrics improve when GAN loss is added, while both components of d and total distortion become worse.This comparison fixes the architecture and distortion loss across HiFiC and Baseline (no GAN).
- Quantitative Results: Increasing β improves the FID-based perceptual index but worsens d across rates, with the trade-off weakened at higher rates.The study evaluates β ∈ {0.015, 0.15, 1.5} across three bitrates.
- Studies: A non-conditional discriminator produces less sharp images and worse FID than the conditional discriminator.The authors attribute the difference to altered training dynamics and note that the distortion loss still guides the generator.
- Studies: InstanceNorm causes darkening when the model is evaluated at 512×512 rather than its 256×256 training resolution, while BatchNorm causes unstable training.The architecture study motivates normalization choices for varying image resolutions.
- Studies: Generator metrics saturate around nR=5, whereas nR=1 gives significantly worse FID and PSNR.The default generator uses nR=9 residual blocks.
6 Conclusion
The paper concludes that GAN-optimized neural compression produces perceptually faithful reconstructions and can outperform prior methods at substantially higher bitrates. It also finds that no existing metric fully predicts user preferences, although FID and KID help guide design exploration.
- GAN-optimized neural compression yields reconstructions with high perceptual fidelity that remain visually close to the input.
- HiFiC is preferred to previous methods even when those methods use more than double the bitrate.
- No existing metric perfectly orders arbitrary models according to human preferences, but FID and KID are useful for exploring architectures and design choices.
- The paper identifies perceptual metrics and generative video compression as directions for future work.
Broader Impact
The method could reduce storage and transmission demands through better reconstructions at lower bitrates, but its realistic generator can produce reconstructions substantially different from sensitive inputs.
- Better reconstructions at lower bitrates can reduce picture storage and transmission requirements.
- Because the generator is trained for realistic reconstructions, the method is unsuitable for sensitive contents such as medical images or important documents.
Funding
The supplied material documents implementation details, comparisons, normalization choices, and failure-related hypotheses rather than funding information.
- The supplied material states that the work was done at Google Research and refers readers to method details and visual results in the appendices.
- A.1 Comparing MSE models based on Minnen et al. [32]: The paper compares its M&S Hyperprior model with reported Minnen et al. models and finds the autoregressive model approximately 0.4dB better on average.
- A.2 Losses: Adding a GAN loss improves sharpness, preserves image noise, and avoids gridding artifacts observed in the no-GAN baseline.
- A.3 Comparison to other generative compression methods: The paper compares HiFiC with prior adversarial-loss approaches on Kodak and notes high-frequency artifacts, texture differences, and color shifts in those approaches.
- A.4 ChannelNorm: ChannelNorm normalizes over channels only, unlike BatchNorm, LayerNorm, InstanceNorm, and GroupNorm, which average over spatial dimensions.
- A.4 ChannelNorm: The authors hypothesize that dependence on spatial dimensions H and W causes observed generalization problems.
A.5 User Study: More Results
Additional user-study analyses show that the overall ranking is mostly stable across participants and images, while individual images receive widely varying Elo scores. Per-image rankings also favor a GAN model on nearly every image except one containing small-scale text.
- Global Monte Carlo Elo Scores: Averaging Elo scores across participants and images produces the global box-plot analysis used in the main text.The global tournament aggregates all comparisons across raters and images.
- Per-participant and per-image Elo scores: The overall method order remains unchanged when Elo tournaments are run separately per participant or per image, except among three closely ranked methods.The methods whose order changes are HiFiC at 0.120bpp, BPG at 0.390bpp, and M&S Hyperprior at 0.405bpp.
- Per-participant and per-image Elo scores: Different images obtain a wide range of Elo scores.
- Per-image rankings: On all but one of the 20 study images, a GAN model earns first or second place in the per-image rankings.The exception contains a lot of small-scale text.
- Study interface: The study GUI screenshot and participant-selected crops are provided for inspecting the evaluation interface and inputs.
A.6 Training Details
The appendix documents training procedures, fixed hyper-parameters, evaluation details, and supplementary rate-distortion/perception results across datasets.
- Training procedure: Training alternates one joint E, G, P update with one discriminator update using Adam and a shared learning rate of 1E−4.
- Training procedure: Models are first trained with a higher λ because rate loss can dominate early training, then GAN models are initialized from MSE and LPIPS-trained models.
- Hyper-parameters: The experiments fix the hyper-parameters shown in Fig. A8a while varying λ according to the target rate rt.
- Evaluation: FID and KID use overlapping 256 × 256 crops extracted from each image with a half-crop shift in both dimensions.
- Evaluation: Supplementary curves report rate-distortion and rate-perception results on DIV2K and Kodak, omitting FID and KID for Kodak because its 24 images yield only 192 patches.
A.9 Image Dimensions of the Datasets
The appendix characterizes image-size distributions across the evaluation datasets and identifies their dominant dimensions and high-resolution extent.
- Dataset dimensions: Kodak images are fixed at 768×512 pixels, whereas the other two datasets contain images with varying dimensions.
- Dataset dimensions: Most images cluster around shorter sides of 1400px, with dimensions extending up to 2000px.
- Supplementary figures: Figure A9 provides a screenshot of the user-study graphical interface, while Figures A10 and A11 show rate-distortion and rate-perception curves for DIV2K and Kodak.
- Dataset dimensions: Figure A12 summarizes dataset dimensions using histograms of shorter side, total pixel count, and aspect ratio.
B Further Visual Results
Further visual materials compare HiFiC reconstructions with established compression methods at matched or higher bitrates and provide downloadable image resources.
- Visual comparisons: The visual examples include full reconstructions alongside originals and selected crops comparing HiFiC, M&S Hyperprior, BPG, and JPEG.
- Reference methods: JPEG at Q = 80 is included as an additional reference because that quality factor is commonly used.
- Visual comparisons: HiFiC GAN models produce plausible textures and visually close reconstructions, while BPG at HiFiCLo’s bitrate tends to exhibit block artifacts.
- Visual comparisons: BPG at twice HiFiCMi’s rate begins to look similar to the HiFiC reconstructions.
- Raw images: Large background images are embedded as JPEGs for file size, while crops and packaged method outputs are available as PNGs.
- Raw images: The appendix supplies raw image archives for the user-study CLIC2020 images and complete Kodak, CLIC2020, and DIV2K outputs from three HiFiC models.