Lossy Compression with Gaussian Diffusion

1 INTRODUCTION

• Lossy compression with realism constraints seeks low bitrate and distortion while keeping reconstructions approximately distributed like real data.The cited theoretical and empirical work suggests generative compression may reduce bitrate at similar perceived quality to distortion-only approaches.Source: §1 · p. 1
• DiffC communicates pixels corrupted by isotropic Gaussian noise and uses an unconditionally trained diffusion model, without an encoder transform.Varying the noise level supports arbitrary bitrates, and reconstructions can be generated from incomplete bitstreams.Source: §1 · pp. 1–2
• DiffC uses one model where typical compression schemes use encoder and decoder transforms plus an entropy model.The paper contrasts this with generative approaches that use several models and train sets targeting different bitrates.Source: §1 · p. 2
• At high bitrates, probability-flow reconstruction halves distortion compared with ancestral sampling.The analysis also finds isotropic Gaussian noise close to, but generally not, optimal for Gaussian data under MSE.Source: §1 · p. 2

2 RELATED WORK

• DiffC communicates Gaussian samples using random coding, paralleling Agustsson and Theis (2020), who used dithered quantization to communicate uniform samples.The paper notes that diffusion models can be viewed as hierarchical VAEs with a fixed encoder.Source: §2 · p. 2
• Unlike Ho et al. (2020), DiffC studies distortion under a perfect realism constraint and provides theoretical and empirical evidence that the approach works.It communicates exact samples rather than approximate samples from minimal random coding, and uses an upper bound as a proxy for coding cost.Source: §2 · p. 2
• Like early pixel-level dithering, DiffC applies noise directly to pixels, but adds a generative model for entropy coding and denoising.Roberts (1962) found dithering perceptually more pleasing than quantization banding artefacts.Source: §2 · p. 2
• The paper distinguishes its diffusion process from anisotropic diffusion: the former increases individual-image high-frequency content while smoothing the image distribution.Source: §2 · p. 2
• Under a perfect realism constraint, DiffC can exceed the claimed R(D/2) limit by up to 3 dB in signal-to-noise ratio at the same rate.The authors attribute the discrepancy to Yan et al. (2021) considering deterministic codes, whereas DiffC allows random codes with shared randomness.Source: §2 · pp. 2–3
• Neural compression methods targeting realism generally combine transform coding with VGG-based distortion and adversarial losses, unlike DiffC's single unconditional diffusion model.The paper reports bitrate savings over lossless compression even when allowing imperceptible distortions.Source: §2 · p. 3

3 LOSSY COMPRESSION WITH DIFFUSION

• DiffC first communicates a corrupted data sample and then uses a diffusion model to generate a reconstruction.The communicated Z_t is treated as the result of a Gaussian diffusion process; the model learns reverse conditional distributions to invert that process.Source: §3 · p. 3
• Random coding can communicate an exact sample of Z_t with an upper-bounded cost based on I[X, Z_t], which decreases as noise increases.The cited bound is I[X, Z_t] + log(I[X, Z_t] + 1) + 5 bits.Source: §3 · pp. 3–4
• The coding-cost upper bound is C_t + log(C_t + 1) + 5, where C_t is the expected KL divergence between q(z_t | X) and p_t(z_t).In practice, coding cost can be significantly closer to C_t than the upper bound.Source: §3 · p. 4
• Reverse-channel coding selects a candidate sample and communicates its index, with sender and receiver able to generate the shared candidate sequence.The selected candidate is distributed as q; the index entropy is bounded by the coding-cost result.Source: §3 · p. 4
• Because diffusion models lack tractable marginals p_t, one option communicates successive conditional samples, while another communicates a joint sample q(z_T:t | X).Successive transmission incurs the overhead of coding each conditional sample.Source: §3 · p. 4
• Reverse-channel coding has unresolved practical challenges, including the computational expense of the Li and El Gamal (2018) scheme.The section mostly sets computational complexity aside to assess whether adding isotropic Gaussian noise directly to data is useful.Source: §3 · p. 4
• DiffC-A uses ancestral sampling for reconstruction, whereas DiffC-F uses deterministic probability-flow reconstruction.The probability-flow ODE has the same trajectory of marginal distributions as the Gaussian diffusion process and uses the same score model.Source: §3 · p. 4

4 A RATE-DISTORTION ANALYSIS

• The analysis measures distortion by squared error and assumes a perfectly learned data distribution for its theoretical results.The section treats Gaussian cases analytically and gives bounds for general distributions.Source: §4 · p. 5

4.1 STANDARD NORMAL DISTRIBUTION

• For a standard normal source, ancestral sampling matches an optimal deterministic code, but flow-based reconstruction sets the reconstruction to Z and yields lower distortion.The analysis notes that Z already has the target standard normal distribution.Source: §4.1 · p. 5

4.2 MULTIVARIATE GAUSSIAN

• For multivariate Gaussian data, a direction-specific noise schedule can match the best deterministic encoder, while isotropic noise performs nearly as well as optimal noise in the illustrated example.The schedule uses different noise levels along principal directions; the figure uses 16x16 ImageNet patches.Source: §4.2 · p. 6, Figure 2 · p. 6

4.3 GENERAL DATA DISTRIBUTIONS

• For approximately Gaussian sources, DiffC-A* operates near the best deterministic encoder’s R(D/2) rate; this also holds at high bitrates when differential entropy is finite.The analysis assumes the diffusion model perfectly learned the data distribution and uses mutual information as a proxy for rate.Source: §4 · p. 5, §4.3 · p. 7
• At high bitrates, flow-based reconstruction has half the error of ancestral sampling under a perfect model, consistent with an approximately 3 dB advantage in the Gaussian example.The figure reports up to 3 dB better SNR for flow-based reconstruction.Source: Figure 2 · p. 6, §4.3 · p. 7
• For sources with smooth density and independent components after an orthogonal transformation, flow reconstruction is optimal among reconstructions conditionally independent of X given the noisy input and satisfying perfect realism.The stated source form is X = QS, with Q orthogonal and the components of S independent.Source: §4.3 · p. 8

5.1 DATASET, METRICS, AND BASELINES

• The evaluation uses ImageNet 64x64, removes 4,952 validation duplicates and 744 training-overlap images, and measures a negative ELBO of 3.48 bits per dimension on the remaining subset.The authors remove these images to make evaluation more meaningful, especially for comparisons with non-neural baselines.Source: §5.1 · p. 8
• The study compares against BPG, a strong non-neural HEVC-based codec, and HiFiC, a generative compression baseline evaluated with FID and PSNR.The HiFiC architecture was slightly adapted for ImageNet 64x64, and five models were trained for different bitrates.Source: §5.1 · pp. 8–9

5.2 RESULTS

• DiffC-F significantly outperforms HiFiC in FID, while DiffC-A’s FID is only slightly worse.At around 0.2 bits per pixel, DiffC-F reconstructions are described as perceptually pleasing; HiFiC shows typical GAN artefacts at very low bitrates, which were not observed with DiffC.Source: §5.2 · p. 9
• At high bitrates, DiffC-F’s PSNR is roughly 2.4 dB higher than DiffC-A’s, and both exceed HiFiC and BPG.The authors relate the 2.4 dB gap to their theoretical 3 dB prediction, noting that the diffusion model only approximates the true distribution.Source: §5.2 · p. 9

6 DISCUSSION

• DiffC replaces a typical multi-model compression pipeline with one unconditionally trained diffusion model, while further perceptual-quality gains may come from applying it in latent space.The authors also note that lower-dimensional transform spaces could reduce computational cost, but current image generation requires many diffusion steps.Source: §6 · p. 9

A REVERSE CHANNEL CODING

• Reverse channel coding after each diffusion step is described as more practical and still significantly outperforming HiFiC.It lowers computational cost relative to coding the joint distribution across steps, which requires more score-function evaluations.Source: §6 · pp. 9–10

6 DISCUSSION

• The initial theory shows DiffC-A* performs well when data are close to Gaussian or bitrate is high, while DiffC-F can achieve 3 dB better SNR than DiffC-A at high bitrates.The authors identify broader optimality questions as future work.Source: §6 · p. 10

B NORMAL DISTRIBUTION WITH NON-UNIT VARIANCE

• For a one-dimensional Gaussian source with varying variance, the analysis reduces to communicating a Gaussian source through a Gaussian channel and relates squared error to information in Z.The cited analysis uses the standard-normal result to derive the corresponding varying-variance case.Source: B NORMAL DISTRIBUTION WITH NON-UNIT… · p. 14

C MULTIVARIATE GAUSSIAN

• For multivariate Gaussian data, the rate at fixed distortion is generally suboptimal when using the stated per-coordinate distortion allocation rather than reverse water-filling.The reverse water-filling solution minimizes rate for fixed distortion.Source: C MULTIVARIATE GAUSSIAN · p. 15

D OPTIMAL NOISE SCHEDULE FOR FLOW-BASED RECONSTRUCTION

• The reconstruction achieves the minimal rate at distortion D among reconstructions satisfying the realism constraint X̂ ∼ X.The argument relaxes the realism constraint during optimization, then shows the resulting solution also satisfies the stronger marginal constraint.Source: D OPTIMAL NOISE SCHEDULE FOR… · pp. 15–17

E PROOF OF THEOREM 1

• The proof benchmarks X against a Gaussian with matching first and second-order moments, relating their rate-distortion functions.The lemma assumes X has finite differential entropy.Source: E PROOF OF THEOREM 1 · p. 17
• For the Gaussian comparison, the proof uses the Shannon lower bound and the fact that the Gaussian log density is quadratic.The argument invokes moment matching to connect the log-density expression to the comparison.Source: E PROOF OF THEOREM 1 · p. 17
• The proof applies water-filling with D_i = min(θ, λ_i) and Σ_i D_i = D to obtain the bound.Here λ_i are covariance eigenvalues, and θ is chosen to meet the distortion constraint.Source: E PROOF OF THEOREM 1 · p. 17

F PROOF OF THEOREM 2

• For ImageNet 64x64, the authors use a trained diffusion model to obtain estimates of G_t; these estimates are shown in Fig. 5.Source: F PROOF OF THEOREM 2 · p. 19
• Diffusion increases distributional smoothness, with ˜G_t ≤ G_0.The proof introduces G_t and ˜G_t as smoothness measures for the diffusion marginals.Source: F PROOF OF THEOREM 2 · pp. 19–20
• Three lemmas connect the errors of ancestral and flow reconstructions to source smoothness measured by G_0 and G_t.The proof then decomposes flow reconstruction error around the conditional expectation, with the cross term equal to zero.Source: F PROOF OF THEOREM 2 · pp. 20–22
• Theorem 2 compares ancestral reconstruction ˆX_A with the probability-flow reconstruction ˆX_F under a smooth-density, finite-G assumption.ˆX_F is defined as the solution of the ODE initialized at Z_t; the supplied theorem excerpt does not show its concluding inequality.Source: F PROOF OF THEOREM 2 · pp. 19, 22, 23

G PROOF OF THEOREM 3

• The flow reconstruction minimizes squared error among reconstructions that are conditionally independent of X given Z_t and achieve perfect realism.The theorem assumes X is an orthogonal transform of a vector with smooth density and independent components.Source: G PROOF OF THEOREM 3 · p. 23
• In one dimension, the proof identifies the flow reconstruction as optimal using the transport solution that maps the MSE reconstruction to the source distribution.The cited transport result assumes the source measure is absolutely continuous with respect to Lebesgue measure.Source: G PROOF OF THEOREM 3 · pp. 24–25
• The proof reduces the analysis to independent coordinates, where the ODE separates into M one-dimensional problems.It uses invariance under orthogonal transformations to assume Q = I.Source: G PROOF OF THEOREM 3 · p. 25

H COMPUTE RESOURCES

• VDM training took about 13 days on 32 TPUv3 cores; each HiFiC model took about 4 days on 2 V100 GPUs.Ten HiFiC models targeted five bitrates, with and without pretrained weights; no VDM hyperparameter search was performed.Source: H COMPUTE RESOURCES · p. 26

• Figure 6 shows communicated messages with estimated bitrates in black and DiffC-F reconstructions with PSNR values in red; its leftmost bitrate is lossless VDM compression.Source: Figure 6 · p. 27
• Figure 7 presents reconstructions from different compression methods, with the uncompressed image in the leftmost column and bitrates in black and PSNR values in red.Source: Figure 7 · p. 28
• Figure 8 reports PSNR from average MSE, unlike Section 5, which averages per-image PSNR scores.Source: Figure 8 · p. 29
• Figure 9 evaluates performance against an upper bound on coding cost when information is progressively sent in chunks of B bits.The coding cost is estimated using Ct.Source: Figure 9 · p. 29
• At 10 bits, PSNR is comparable to the strongest baseline while FID remains significantly lower.Source: I ADDITIONAL FIGURES · p. 29
• For HiFiC trained from scratch for MSE only at one targeted bitrate, PSNR improves slightly while FID gets significantly worse.Source: Figure 10 · p. 29