Multi-Realism Image Compression with a Conditional Generator

• Rate–distortion compression balances bitrate with input closeness, while realism measures reconstruction-distribution divergence and may increase as distortion rises.The paper uses PSNR for distortion and FID for realism.Source: §1 · pp. 1–2
• Realism-oriented compression can synthesize sharp, textured detail, but its reconstructions may have worse MSE than an average-image reconstruction.Source: §1 · p. 1
• For Kodak kodim20, the high-realism reconstruction reaches 31.3 dB at 0.12 bpp, versus HiFiC’s 29.3 dB at 0.12 bpp.Source: §1 · p. 2
• At high realism, the paper reports sharper reconstructions that more closely match inputs than HiFiC, including readable airplane text and preserved sneaker texture.The figure also notes that HiFiC uses 1.92× and 1.86× the bitrate for similar hair and final-row reconstructions.Source: Figure 2 · p. 2
• A single compressed representation supports low-detail, high-PSNR and detailed, realistic reconstructions, with the receiver choosing between them by changing β.The decoder, rather than the encoder, is conditioned on β.Source: §1 · p. 3
• The method reports a new state of the art in distortion–realism, achieving better distortion at high realism and better realism at low distortion.Source: §1 · p. 3

• Universal representations have been studied theoretically and empirically; Zhang et al. [48] show zero overhead for scalar Gaussian sources and at most a factor-2 distortion increase for general sources under a realism constraint.Their MNIST experiments train further decoders with a frozen encoder; this paper jointly trains an encoder and conditional decoder.Source: §2 · p. 3
• ELIC [13] is described as a practical state-of-the-art neural rate–distortion codec for MSE and MS-SSIM; some stronger methods use serial autoregressive context models and typically decode 10× slower.Source: §2 · p. 3
• Realism and perception refer to divergence between image distributions, while perceptual quality is broader and can also be measured by LPIPS or MS-SSIM.Source: §2 · p. 3
• HiFiC uses a conditional GAN with a hyperprior, while PO-ELIC and Li et al. [21] combine adversarial or perceptual losses with other objectives to improve perceptual quality.Source: §2 · p. 3
• Realism-focused methods often target one distortion–realism point, whereas this method uses one model and representation to target points along the curve.Source: §2 · p. 3
• Gao et al. [9] optimize a latent at encode time for each new trade-off, requiring hundreds or thousands of steps per image.Source: §2 · p. 3
• Iwai et al. interpolate decoder weights after finetuning for GAN loss, targeting extremely low rates below 0.04 bpp on Kodak.Source: §2 · p. 3
• Diffusion with reverse channel coding is reported as promising for 64×64 image compression on ImageNet64.Source: §2 · p. 3

3.1. Background

• The codec maps images to quantized representations, entropy-codes them using a learned symbol model, and balances bitrate against reconstruction distortion.The rate-distortion objective uses λ to control the trade-off.Source: §3.1 · p. 4
• At fixed bitrates, the method’s PSNR–FID/256 points lie closer to the top-left optimum than HiFiC and the SOTA MSE Baseline.The plotted chain of points reflects multiple β-selected reconstructions from one representation; the baselines each show one point.Source: Figure 4 · p. 5

3.2. Our Approach

• A single representation supports multiple reconstructions: the receiver selects β at inference to navigate the distortion–realism trade-off.Training samples β uniformly; the receiver can choose it freely during inference.Source: §3.2 · p. 5
• Training combines a bitrate term with MSE distortion and perceptual and adversarial losses, while λ′ weights bitrate relative to the other terms.The MSE term is scaled by 1/100, and λ′ is set to 100λ.Source: §3.2 · p. 5

3.3. Beta Conditioning

• FourierCond embeds β with Fourier features and an MLP, then projects the resulting features into each convolutional layer; TableCond instead uses β-indexed scaling factors and biases.Both conditioning schemes have the same inference runtime when evaluation uses a finite set of β values.Source: §3.3 · pp. 5–6

4.1. Metrics

• PSNR measures distortion and FID measures realism in the evaluation.Source: §4.1 · p. 6

4.2. Datasets

• Evaluation uses Kodak, CLIC 2020, and MS-COCO; Kodak reports PSNR only, while CLIC and COCO support FID evaluation.CLIC 2020 uses 428 high-resolution test images and patched FID/256; MS-COCO uses 30,000 images.Source: §4.2 · p. 6
• Figure 5 shows an ELIC-based encoder and decoder whose generator residual blocks receive projected features derived from β.Source: Figure 5 · p. 6

4.3. Building Strong Baselines

• The ELIC-based SOTA MSE Baseline approximates unavailable non-generative state-of-the-art models and nearly matches ELIC in PSNR.The reported Kodak PSNR difference is approximately 0.1 dB.Source: §4.3 · p. 6
• The fixed-β GAN baseline outperforms HiFiC in COCO FID, nearly matches it on CLIC 2020, and has significantly stronger PSNR.It is trained end-to-end without HiFiC’s multi-stage training or rate controller.Source: §4.3 · p. 6

4.4. Published Baselines

• HiFiC is the published generative FID baseline on high-resolution CLIC 2020 and MS-COCO; MSE comparisons include Charm, ELIC, BPG, and VTM 17.1.The cited newer methods [14] have limited evaluation and do not provide code for custom datasets.Source: §4.4 · p. 7

4.5. Our Models

• Baselines and ablations train for 2M iterations at batch size 8 on 256px crops; multi-realism models train for 3M steps because the decoder learns high and low realism simultaneously.The models use Adam with learning rate 1e−4 and default settings; the training parameters were not tuned.Source: §4.5 · p. 7

5.1. Main Results

• On Kodak, the SOTA MSE baseline is only 0.0 −0.2dB below ELIC, while the multi-realism model is strong in low-β PSNR and allows high-realism reconstructions.Source: Figure 6 · p. 7, §5.1 · p. 7
• At one rate point, high-β achieves ≈1dB higher PSNR than HiFiC at low FID, while low-β yields about 40% lower FID than non-generative codecs at high PSNR.Across the benchmark comparisons, β=2.56 matches or outperforms HiFiC in FID, and β=0 has significantly better FID than the MSE baseline.Source: §5.1 · p. 7
• The β=0 model has significantly better FID than the MSE baseline, although the MSE baseline has slightly better PSNR.Source: §5.1 · p. 7

5.2.1 Loss Weights and Charm

• On MS-COCO, GAN loss lowers FID until β=2.56, then saturates; LPIPS weight 4.26 remains a good choice, and Charm lowers bitrate.For inference, βmax is set to 2.56 because FID is lowest there.Source: Figure 7 · p. 8, §5.2 · p. 8, §5.2.1 · p. 8

5.1. Main Results

• At β=2.56, reconstructions are closer to the input than HiFiC at the same bitrate or achieve similar visual quality at lower bitrates; decreasing β raises PSNR but loses detail.Source: §5.1 · p. 8

5.2.2 Conditioning

• FourierCond and TableCond give very similar results; FourierCond is selected for the main model because it is simpler to implement.Source: §5.2.2 · p. 8

6. Conclusion

• From one compressed representation, the receiver can decode either a high-realism reconstruction at high β or a high-PSNR reconstruction at low β, and the method can reach a new state of the art in PSNR versus FID.Source: §6 · p. 8
• To the authors’ knowledge, this is the first single-decoder method to move the distortion–realism trade-off to the receiver without changing the bitstream.Users may choose reconstructions closer to the original or with more generated detail, depending on the use case.Source: §6 · p. 8
• High realism is achieved without sacrificing PSNR by more than ≈1−1.5dB; the authors call for further work to push the realism–distortion boundary.Source: §6 · p. 8

A.1. Runtime Benchmarks

• Runtime benchmarks are reported for Tesla V100, NVIDIA 2080 Ti, and NVIDIA 3090 Ti systems.The results are summarized in Tables 1, 2, and 3, respectively.Source: §A.1 · p. 11
• Compression and decompression timings include GPU entropy-parameter prediction and CPU range coding.The benchmark also includes the CPU runtime needed to encode images.Source: §A.1 · p. 11
• Cross-paper runtime comparisons are difficult, and the ELIC implementation omits checkerboard encoding and uneven CHARM grouping, which should improve runtime.The authors say both omitted features should improve entropy-coding and decoding performance.Source: §A.1 · p. 11

A.2. Running Published methods

• HiFiC models from TensorFlow Compression were run with tfci using the hific-lo, hific-mi, and hific-hi models.Source: §A.2 · p. 11
• VTM 17.1 inputs were converted to YCbCr and stored as raw bytes before encoding with EncoderApp and decoding with the saved bitstream.The supplied command specifies 8-bit input, 4:4:4 chroma, and one frame.Source: §A.2 · p. 11
• BPG v0.9.8 was run on PNG inputs with bpgenc using 4:4:4 chroma, x265, and 8-bit depth, then decoded with bpgdec.Source: §A.2 · p. 11

A.3. Comparison to Iwai et al.

• Iwai et al.’s released model was run on CLIC and evaluated in terms of FID and PSNR versus bitrate.Source: §A.3 · p. 11
• The comparison with Iwai et al. is presented in Figure 8.Source: §A.3 · p. 11
• The reported evaluation compares both distortion and realism metrics against bitrate.The metrics named are PSNR and FID.Source: §A.3 · p. 11

A.7. Visual Results

• The appendix provides CLIC 2020 reconstructions for the lowest-rate model at β = 0.0 and β = 2.56.The reconstructions are provided in a zip file.Source: §A.7 · p. 11
• Runtime tables report one-megapixel timings on V100, 2080 Ti, and 3090 Ti GPUs; the 3090 Ti caption notes serial entropy coding and unparallelized autoregressive components.Source: Table 1 · p. 12, Table 2 · p. 12, Table 3 · p. 12

A.4. Comparing to ELIC Reconstructions

• Figure 8’s comparison shows Iwai et al. target lower bitrates but have significantly worse PSNR and FID.Source: Figure 8 · p. 12
• At β = 2.56, our method has higher PSNR than HiFiC at comparable perceptual quality; Figure 9 also compares reconstructions with ELIC.The two reconstructions from the method use the same representation and vary only the generator’s realism weight.Source: §A.4 · p. 11, Figure 9 · p. 13

A.5. More Realism rates

• Figure 10 extends Figure 4 with more rate points, including 0.17 and 0.56 bpp for MS-COCO 30K and 0.092 and 0.32 bpp for CLIC 2020.Source: §A.5 · p. 11, Figure 10 · p. 14

A.6. Comparing conditioning schemes

• FourierCond and TableCond give comparable results, and FourierCond is selected for the main model.Both conditioning approaches were trained in the same manner as the main model.Source: §A.6 · p. 11, Figure 11 · p. 14