Rethinking Generative Image Compression at Extremely Low Bitrates

Motivation

• Below conventional operating rates, representative codecs can abruptly deform objects, lose salient entities, and make scene structure implausible instead of gradually losing source-specific detail.The introduction terms this failure semantic collapse.Source: §1 · p. 1, Figure 1 · p. 2

Analysis

• Reconstruction losses agree with one another but are nearly orthogonal to semantic objectives, with negative alignment becoming more frequent as the bitrate decreases.The gradient comparisons use MSE, LPIPS, and semantic similarities from vision foundation models.Source: §1 · p. 1
• Pixel-space and reconstruction-oriented VAE diffusion preserve semantics less effectively as rates fall, motivating diffusion in a representation space.The introduction compares pretrained diffusion transformers across pixel, VAE, and RAE spaces.Source: §1 · p. 2

Contribution

• RAE-CoD directly aligns a compressed condition with the clean source representation in a representation autoencoder space, using a latent codec and conditional decoupled diffusion transformer.DINOv3 supplies the clean representation target; the latent codec fuses representation and pixel features into entropy-coded latents.Source: §1 · p. 2

Evaluation and results

• RAE-CoD maintains recognizable, high-quality reconstructions as source consistency falls smoothly, with semantic coherence reported down to 16 bits.The evaluation combines five vision foundation models with a metadata-blind vision-language model assessing recognizability, quality, and consistency.Source: §1 · p. 2

2.1 WHAT HAPPENS AT EXTREMELY LOW BITRATES?

• Semantic collapse is the abrupt loss of valid semantic structure below the method-specific minimum rates at which most generative codecs are evaluated.The passage distinguishes this from gradual loss of source correspondence: zero-rate generation could remain realistic without instance-level correspondence.Source: §2.1 · p. 3

2.2 ANALYSIS OF SEMANTIC COLLAPSE

• Reconstruction losses are nearly orthogonal to semantic objectives and become negatively aligned more often at lower rates; MSE favors blur, while LPIPS produces structured artifacts.Semantic supervision retains relatively more recognizable content in the extreme decoder-adaptation test.Source: §2.2 · pp. 3–4
• Pixel-DiT offers stronger rate-distortion behavior and a higher-rate ceiling, VAE-DiT favors moderate-rate perceptual fidelity, and RAE-DiT preserves semantic representations most efficiently as rates decrease.RAE-DiT retains coherent entities and scene layouts longer as bitrate falls.Source: §2.2 · p. 4
• The analysis motivates a framework with a frozen representation autoencoder, a deep-compression latent codec, and a codec-conditioned decoupled diffusion transformer, trained in representation space.The framework diagram identifies its three components and the space in which its objectives are applied.Source: Figure 6 · p. 5

3 COMPRESSION-ORIENTED DIFFUSION WITH RAE

• RAE-CoD builds compression-oriented diffusion in RAE space and directly aligns the codec with the clean source representation.The method is designed to replace abrupt semantic collapse with gradual loss of source consistency while retaining realistic, recognizable content.Source: §3 · p. 5

3.1 PIPELINE

• The pipeline uses DINOv3 representations as the clean diffusion target and combines representation and pixel features into entropy-coded latents that condition a decoupled diffusion transformer.The representation decoder maps the generated clean representation back to pixels.Source: §3.1 · p. 5

3.2 DIFFUSION TRAINING STRATEGY

• Training combines flow-matching and representation-alignment losses with a deterministic auxiliary objective that encourages the entropy-constrained codec condition to preserve semantics.The auxiliary objective compares a projection of the condition with the clean target representation; the full objective also includes rate and vector-quantization commitment losses.Source: §3.2 · pp. 5–6
• Progressive rate training first increases the rate penalty while adapting the pretrained DDT with LoRA, then fine-tunes the codec and DDT jointly at fixed target rates.Stage II branches from Stage-I checkpoints and trains all DDT parameters together with the codec.Source: §3.2 · p. 6

• The protocol pairs deterministic VFM features with a blinded VLM judge because pixel distortion and perceptual fidelity do not distinguish the targeted semantic differences at extremely low bitrates.Figure 7 covers VFM-based semantic and quality evaluation, while Figure 8 covers VLM-based semantic and quality evaluation on MSCOCO-30K.Source: §4 · p. 6, Figure 7 · p. 7, Figure 8 · p. 7
• Across five VFMs, the protocol reports average relative feature MSE (RelMSE5), cosine similarity (COS5), and Fréchet Distance (FD) for distributional comparison.The five models are Inception-v3, ConvNeXt-v2, DINOv2, SigLIP2, and CLIP.Source: §4 · p. 6
• The blinded VLM evaluation separates semantic recognizability, semantic quality, and semantic consistency as distinct reconstruction properties.Recognizability concerns identifiable content, quality concerns coherence and naturalness, and consistency measures retention of the source meaning.Source: §4 · p. 6
• For source consistency, the protocol separately inventories source and reconstruction content, matches the inventories, and combines semantic recall and precision with global scene similarity.The score combines the harmonic mean of recall and precision with global scene similarity.Source: §4 · p. 6

5.1 SETTINGS

• The evaluation uses MSCOCO-30K as its primary dataset and Kodak as an additional dataset, with images resized and center-cropped to 256 × 256.The comparison includes five generative codecs and two space-controlled variants; rates are estimated in theoretical bits per pixel.Source: §5.1 · p. 7

5.2 MAIN RESULTS

• At around 0.008 bpp, RAE-CoD reduces RelMSE5 by 25.7% and FDr5 by 69.1% relative to the best competing value.Its advantage over Pixel-CoD and VAE-CoD grows near 0.001 bpp.Source: §5.2 · pp. 7–8
• Across RAE-CoD’s curve, SR stays within 86.7–87.2 and SQ within 69.2–70.7, while SC decreases from 61.5 to 17.2.Reconstructions remain recognizable and semantically well formed while becoming less tied to the source.Source: §5.2 · p. 8
• At similar bitrates, RAE-CoD receives 64.2–90.8% of votes across five Kodak comparisons, and the qualitative examples show greater coherence as rates decrease.The study gathered 120 votes from 10 participants per comparison; competing methods lose source entities or develop malformed structures.Source: §5.2 · p. 8, Figure 10 · p. 8, Figure 9 · p. 8

5.3 DISCUSSION AND ABLATION STUDY

• At 16 bits, the hyper latent biases generation toward broad scene semantics, but noise samples can change object identity or layout; around four additional bits improve stability.The total rate with the DiffC reverse-channel-coding step is 20 bits, or 0.000305 bpp.Source: §5.3 · p. 8, Figure 11 · p. 8
• Semantic alignment alone performs substantially better than adding pixel-level MSE as an auxiliary codec loss.Table 1 compares auxiliary losses using BD-rate over four points from 0.001 to 0.008 bpp, relative to no auxiliary loss.Source: §5.3 · p. 8, Table 1 · p. 9
• Using only the pixel encoder requires at least 37.35% more bits than using only the RAE encoder at matched aggregate quality.Fusing pixel and RAE features reduces bitrate relative to RAE-E alone by 32.10% under RelMSE5, 32.92% under COS5, and 66.34% under FDr5.Source: §5.3 · p. 9, Table 2 · p. 9

6 RELATED WORK

• Diffusion-based codecs use learned priors to improve realism through pretrained-diffusion reverse-channel coding, generative refinement, or conditional diffusion-decoder training; this work examines representative codecs below their operating bitrates.Examples include reverse-channel coding with pretrained diffusion (Theis et al., 2022), generative refinement (Ghouse et al., 2023; Hoogeboom et al., 2023), and conditional diffusion-decoder training (Yang & Mandt, 2023).Source: §6 · p. 9
• Image generation uses vision representations through aligned denoiser features and redesigned generative latents, while this work studies their semantic preservation in compression at extreme bitrates.The passage cites representation-based approaches including Yu et al. (2024), Leng et al. (2025), and Singh et al. (2025).Source: §6 · p. 9

7 CONCLUSION

• RAE-CoD combines diffusion in a pretrained representation space with direct alignment of the compressed condition to the source representation.The paper’s VFM and VLM evaluations show recognizable, naturally structured content down to 16 bits while source consistency decreases gradually.Source: §7 · p. 9
• The experiments are limited to 256 × 256 images and a 0.9B-parameter decoupled diffusion transformer, so they establish no scaling law.The effects of larger representation-space models and higher-resolution images on semantic stability, quality, and bitstream size remain unclear.Source: §7 · p. 9

A.1 MODEL CONFIGURATIONS AND EVALUATION

• Five frozen vision encoders are evaluated with model-specific preprocessing, using final-layer global tokens for transformers and averaged final feature maps for convolutional encoders.RelMSE5 and COS5 average image-level results equally across models; FDr5 averages separately normalized feature-space distances.Source: §A.1 · p. 14

A.2 LINEAR PROBING COMPARISON

• All five evaluation encoders provide substantially more linearly accessible category information than the reconstruction-oriented references.DINOv2, SigLIP2, and CLIP are contrasted with supervised-recognition ConvNeXt-v2 and Inception-v3; the probe uses ImageNet-1K features and a trained linear layer.Source: §A.2 · p. 14

B.1 DETAILED EVALUATION PROTOCOL

• The VLM protocol separates source-blind questions about recognizability and image formation from the source-aware question of whether the input meaning is preserved.Table 5 outlines the three-stage evaluation and its semantic-unit inventory.Source: §B.1 · p. 15, Table 5 · p. 15
• The protocol defines SR through identity confidence and SQ through the integrity of recognized content, rather than fidelity to the source or aesthetic preference.Both scores are zero when no semantic unit is recognizable.Source: §B.1 · p. 15
• SC uses source-to-reconstruction and reconstruction-to-source matches, combining importance-weighted semantic recall and precision with global scene similarity.Recall penalizes missing source content, while precision penalizes invented or substituted content.Source: §B.1 · p. 15
• For SR and SQ, image-matched clean-source and low-quality JPEG controls provide identity and lower anchors for normalization.The lower anchor uses baseline JPEG at quality 1 with 4:2:0 chroma subsampling.Source: §B.1 · p. 15
• The frozen Qwen3.5-9B evaluator receives no codec identity, bitrate, or competing outputs, and its model-based judgments are treated as diagnostics rather than a replacement for human evaluation.A qualitative example reports that RAE-CoD preserves room layout and principal objects while its 16-bit reconstruction replaces much source-specific furniture; AEIC-ME produces a degraded, source-unrelated scene.Source: §B.1 · p. 16, Figure 12 · p. 16

B.2 QUALITATIVE EXAMPLES

• At 0.008260 bpp, RAE-CoD retains the living-room type, principal furniture, and overall layout, with SR = 95.872 and SQ = 100.000.Some colors and smaller attributes change; source consistency is SC = 70.191.Source: §B.2 · p. 17
• At 16 bits, RAE-CoD remains recognizable with high source-blind scores, while most source-specific content is replaced and source consistency falls to 26.539.The example reports SR = 95.153 and SQ = 100.000 alongside SC = 26.539.Source: §B.2 · p. 17
• AEIC-ME collapses despite using over fourteen times the rate of 16-bit RAE-CoD, producing low recognizability, quality, and source consistency.Its reported scores are SR = 52.985, SQ = 24.414, and SC = 5.021.Source: §B.2 · p. 17

C.1 ALGORITHM

• DiffC converts a pretrained diffusion model’s variational structure into a progressive lossy code, transmitting progressively refined states toward the original datum.For lossy compression it stops at t* and completes the trajectory with deterministic probability-flow denoising; earlier stopping uses fewer bits and delegates more decisions to the generative prior.Source: §C.1 · pp. 17–18
• DiffC uses reverse-channel coding to communicate a posterior draw, with the expected description length close to D_KL(Q||P) plus logarithmic overhead.The encoder selects a candidate index and the decoder regenerates the shared candidate sequence to recover it.Source: §C.1 · p. 18

C.2 DIFFC ON DIFFERENT DIFFUSION MODELS

• The comparison wraps three pretrained class-conditioned generators in the same DiffC pipeline and uses a balanced 5,000-image ImageNet validation subset.The source class is shared and charged as a fixed 10-bit message; the reported rate includes the arithmetic-coded PFR payload and class label.Source: §C.2 · p. 18
• The three diffusion spaces use source pixels, a seeded frozen-VAE posterior sample, or normalized DINOv3 representations, with model-specific samplers retained.These controls isolate diffusion space while preserving each model’s sampler for valid generation.Source: §C.2 · p. 18
• Across all five VFMs, RAE-DiT has the best feature MSE, cosine similarity, and Fréchet Distance up to approximately 0.056 bpp; Pixel-DiT overtakes it at higher rates.The unaggregated per-representation results are shown in Figure 13.Source: §C.2 · p. 18, Figure 13 · p. 19

C.3 ONE-STEP DIFFC FOR 16-BIT RAE-COD

• The one-step experiment adds a target-dependent transition near pure noise to reduce ambiguity in 16-bit RAE-CoD without changing the codec.The encoder and decoder share seeded noise; chunked PFR selects a source-compatible noisy state before the original RAE-CoD sampler resumes.Source: §C.3 · p. 19
• The approximately four extra bits are an ideal KL-rate analysis, not a measured deployable-file size, because serialization and container overhead are uncounted.The diagnostic verifies recovered states but passes winner indices and KL metadata in memory.Source: §C.3 · p. 19

D.1 MODEL DETAILS

• The 256 × 256 architecture combines DINOv3 representation features and pixel features into coded latents, then decodes a condition for the RAEv2 DDT.The codec has approximately 57.2M parameters and the DDT approximately 875.4M; the RAEv2 decoder remains frozen.Source: §D.1 · pp. 19–20

D.2 TRAINING DETAILS

• Training uses 23.2M ImageNet-21K and CC12M images with the RAEv2 encoder and decoder frozen and the codec learned from scratch.Stage I progressively increases the rate weight while LoRA adapts the pretrained DDT; Stage II branches from target-rate checkpoints and jointly fine-tunes the full DDT and codec.Source: §D.2 · p. 20

D.3 INFERENCE DETAILS

• Inference quantizes the hyper latent to a 4-bit codebook entry and rounds the main latent using predicted means before decoding the condition.The decoded condition is reused across 100 shifted-Euler diffusion steps, with internal guidance, before the frozen RAE decoder maps the generated representation to pixels.Source: §D.3 · p. 20

E ADDITIONAL COMPARISON ON KODAK

• On Kodak, RAE-CoD has lower feature MSE and Fréchet Distance and higher cosine similarity than alternatives at comparable rates, with its advantage persisting toward 16 bits.Figure 14 presents the VFM-based semantic evaluation at 256 × 256.Source: E ADDITIONAL COMPARISON ON KODAK · p. 21, Figure 14 · p. 21

F ADDITIONAL VISUALIZATION

• As rates fall from roughly 0.007–0.008 bpp to 16 bits, fine appearance and source-specific attributes disappear before object identity and spatial arrangement change.Figure 15 shows examples from approximately 512 bits to 16 bits (0.000244 bpp).Source: F ADDITIONAL VISUALIZATION · p. 21, Figure 15 · p. 22
• The visual progression is presented across additional RAE-CoD examples on MSCOCO-30K at 256 × 256.The figure covers rates from approximately 512 bits to 16 bits.Source: Figure 15 · p. 22
• Even when correspondence to the source becomes weak, the outputs remain recognizable and naturally structured rather than collapsing abruptly into malformed content.The repeated transition across examples supports a gradual shift toward unconditional generation.Source: F ADDITIONAL VISUALIZATION · p. 22