PaperlayerView exact revision on arXiv

Source-linked AI summary

ResARC: Residual-Aware AutoRegressive Coding for Ultra-Low Bitrate Image Compression

Qin Yan, Ruixiao Dong, Yutao Xie, Li Li, Ying Chen, Kai Li, Daowen Li, Houqiang Li

arXiv:2609.39451v1cs.CV

TL;DR

Autoregressive generative codecs lose reconstruction fidelity through tokenization and imperfect suffix generation. ResARC generates the quantization residual from decoder context and compresses the generation residual for decoder-side correction; it improves distributional fidelity while maintaining strong perceptual similarity in the ultra-low bitrate regime.

  • Problem

    Autoregressive codecs lose information during discrete tokenization and incur further reconstruction error when generated suffix tokens differ from their ground truth.

    Source: §1 · p. 2

  • Method

    ResARC generates the quantization residual with a decoder-context-conditioned diffusion transformer without extra side information, and compresses the encoder-computed generation residual for decoder-side correction.

    Source: §1 · p. 2

  • Results

    In the ultra-low bitrate regime, ResARC achieves strong perceptual similarity and substantially improved distributional fidelity over leading autoregressive and diffusion-based generative codecs.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings support explicit modeling and compensation of codec residuals as an effective design principle for generative compression.

    Source: §6 · p. 12

Abstract

from arXiv · show

Progressive autoregressive image codecs provide an appealing paradigm for generative compression by quantizing continuous latents into discrete tokens, transmitting coarse-to-fine prefix tokens and generating the remaining suffix tokens at the decoder. However, their reconstruction quality is fundamentally limited by two residuals introduced along this pipeline: the quantization residual, arising from information loss during discrete tokenization, and the generation residual, resulting from imperfect autoregressive generation of the suffix tokens. To address these limitations, we introduce ResARC, a residual-aware autoregressive codec that explicitly compensates for both residuals at the decoder. Specifically, we generate the quantization residual with a diffusion transformer conditioned on the autoregressive decoding context, while requiring no additional side information. In parallel, we compute the generation residual at the encoder and employ a learned Generation Residual Codec to efficiently compress and transmit it for decoder-side correction. The recovered residuals are then integrated with the reconstructed latent representation and decoded through an adapted VAE decoder. Extensive experiments demonstrate that ResARC achieves competitive perceptual similarity while substantially improving distributional fidelity over leading generative codecs across the ultra-low bitrate regime. Code and models will be released soon.

1 INTRODUCTION

Progressive autoregressive codecs lose reconstruction fidelity through quantization and imperfect suffix generation. ResARC compensates for both residuals and improves perceptual similarity and distributional fidelity in the ultra-low bitrate regime.

1 INTRODUCTION

  • Progressive autoregressive coding transmits a coarse-to-fine token prefix and generates the remaining suffix, with prefix depth controlling bitrate in a single model.The design builds on visual-token models that generate across multiple scales.Source: §1 · p. 2
  • Discrete tokenization loses latent information, while imperfect autoregressive suffix synthesis creates a second discrepancy that degrades reconstruction fidelity.The paper names these the quantization residual and generation residual, respectively.Source: §1 · p. 2
  • ResARC generates quantization residuals from decoder context without extra side information, while compressing encoder-computed generation residuals for decoder-side correction.The recovered residuals are fused with the reconstructed latent and decoded through an adapted VAE decoder, preserving progressive bitrate control.Source: §1 · p. 2
  • On DIV2K and CLIC2020, ResARC improves perceptual similarity and distributional fidelity over leading generative codecs in the ultra-low bitrate regime; ablations validate both residual branches.Qualitative comparisons also report better preservation of fine-grained textures and image structures at comparable or lower bitrates.Source: §1 · pp. 2–3

2 RELATED WORK

Generative codecs use learned priors to improve perceptual reconstruction, while autoregressive codecs share a prior for entropy modeling and token generation. ResARC addresses residuals that remain in progressive autoregressive coding.

2 RELATED WORK

  • Generative codecs leverage learned priors to improve perceptual reconstruction, but existing approaches generally lack native progressive bitrate control and tight integration of entropy modeling with the generative prior.The related-work overview spans GAN-based, diffusion-based, and other generative codec approaches.Source: §2 · p. 3
  • Autoregressive codecs share a prior for entropy modeling and sequential generation; ARPC sends coarse-to-fine prefixes and generates omitted suffix tokens.Varying prefix depth controls bitrate within one model.Source: §2 · p. 3
  • Existing autoregressive codecs retain quantization residuals from discrete tokenization and generation residuals from imperfect suffix prediction, which ResARC explicitly compensates.The cited work frames these residuals as distinct sources of reconstruction-fidelity loss.Source: §2 · p. 3

3 BACKGROUND: MULTI-SCALE AUTOREGRESSIVE IMAGE COMPRESSION

The background describes a progressive autoregressive codec that quantizes VAE latents into multi-scale token maps, transmits a prefix, and synthesizes the remaining suffix. Prefix depth controls bitrate, while quantization and synthesis introduce residuals motivating ResARC.

3 BACKGROUND: MULTI-SCALE AUTOREGRESSIVE IMAGE COMPRESSION

  • A VAE encoder maps the image to a continuous latent, which a multi-scale residual quantizer converts into token maps progressively refining coarse structures into fine details.The maps are aggregated by multi-scale summation into the quantized latent.Source: §3 · p. 4
  • For prefix depth k, the codec transmits token scales T≤k and uses the autoregressive model for both prefix entropy coding and omitted-suffix generation.The same model supplies predicted probabilities for coding the prefix and samples the remaining scales after entropy decoding.Source: §3 · p. 4
  • The transmitted prefix and generated suffix are combined into a complete multi-scale token sequence, aggregated, and passed to the decoder for reconstruction.The sequence contains the original prefix scales and autoregressively sampled suffix scales.Source: §3 · p. 4
  • Changing prefix depth changes the number of transmitted scales and bitrate, while reconstruction from quantized tokens and generated suffixes introduces the two residuals motivating ResARC.The coding framework supports progressive bitrate control within a single model.Source: §3 · p. 4

4 RESARC: RESIDUAL-AWARE AUTOREGRESSIVE CODEC

ResARC models autoregressive reconstruction error as quantization and generation residuals, then compensates for each with a distinct decoder-side strategy. It generates the quantization residual from decoder context and transmits a compressed generation residual for correction.

4.1 RESIDUAL-AWARE FORMULATION

  • Discrete tokenization creates a quantization residual, while imperfect suffix generation creates a generation residual; together they explain the latent discrepancy ResARC targets.The quantization residual captures information lost from the continuous latent, and the generation residual captures deviations between synthesized and ground-truth suffixes.Source: §4.1 · pp. 4–5

4.2 QUANTIZATION RESIDUAL GENERATION

  • A conditional diffusion transformer generates the quantization residual from Gaussian noise using autoregressive decoding features, without transmitting extra payload.The generator uses flow matching, and its autoregressive conditioning features are already available at the decoder.Source: §4.2 · pp. 5–6

4.3 GENERATION RESIDUAL CODING

  • At the encoder, ResARC replays suffix generation, computes its mismatch, and entropy-codes it with prefix-dependent scaling and four-pass conditional coding.The codec adapts to prefix depth because generation-residual statistics vary with the number of omitted suffix scales.Source: §4.3 · p. 6

4.4 RESIDUAL FUSION AND RECONSTRUCTION

  • The recovered residuals are integrated into a compensated latent for adapted VAE decoding; only the transmitted generation residual adds residual-branch bits.The quantization residual is synthesized from decoder-available information, whereas the generation residual is entropy encoded and transmitted.Source: §4.4 · pp. 6–7

4.5 TRAINING PIPELINE

  • Training fine-tunes the backbone, learns the two residual branches, then freezes those branches while adapting the VAE decoder to compensated latents.Backbone adaptation uses quantized and continuous latents with equal probability; the final stage adapts the decoder to the compensated-latent distribution.Source: §4.5 · pp. 7–8

5 EXPERIMENTS

Across DIV2K and CLIC2020, ResARC is compared with leading generative codecs and evaluated through ablations of its residual branches and training choices. It achieves a favorable rate-perception trade-off, with the full design benefiting from both residual branches and residual-specific compensation.

5.2 QUANTITATIVE COMPARISONS

  • ResARC achieves a favorable rate-perception trade-off across both datasets, outperforming ARPC on all six metrics and leading overall on DISTS and four distributional fidelity metrics.It remains competitive on LPIPS. The evaluation compares eleven generative codecs at their released bitrate points.Source: §5.2 · p. 8, §5.1 · p. 8

5.3 QUALITATIVE COMPARISONS

  • At similar ultra-low bitrates, ResARC better preserves fine details and image structures than ARPC and diffusion-based codecs.ARPC may alter structures, while StableCodec and ResULIC tend to smooth local details.Source: §5.3 · p. 9, Figure 5 · p. 9

5.4 ABLATION STUDIES

  • Combining both residual branches yields the best overall ablation performance; either branch alone improves all four metrics over the autoregressive baseline.With only the generation branch, DISTS and FID BD-rates fall from +9.80% and +26.57% to +1.90% and +8.03%, respectively.Source: §5.4 · pp. 9–10, Table 1 · p. 10, Figure 6 · p. 10
  • Fine-tuning the autoregressive prior reduces DIV2K BD-rates by 6.42% under DISTS and 11.29% under FID.The authors report that this adaptation provides a stronger foundation for training the residual branches.Source: §5.4 · p. 10, Table 2 · p. 11
  • Residual-aware VAE decoder adaptation reduces DIV2K BD-rates by 4.31% under LPIPS and 7.19% under DISTS.The adaptation improves all four evaluated metrics.Source: §5.4 · p. 10, Table 3 · p. 11
  • Generating both residuals raises BD-rates by 10.46% under DISTS and 11.11% under FID versus ResARC, while transmitting both raises BD-rate across all four metrics.The comparison supports generating the quantization residual from decoder context and explicitly transmitting the generation residual.Source: §5.4 · pp. 10–11, Table 4 · p. 11, Figure 8 · p. 11
  • The tested objectives favor LPIPS loss for the quantization residual generator and DISTS loss for the generation residual codec.Adding DISTS supervision to the generator improves its DISTS BD-rate by 0.65% but increases LPIPS, FID, and KID BD-rates by 1.70%, 1.27%, and 12.10%, respectively.Source: §5.4 · p. 11, Table 5 · p. 12
  • Mixed-latent VAE decoder fine-tuning achieves the best overall BD-rate across all four metrics, with p = 0.5 for sampling the continuous latent.Using only quantized latents incurs 7.42% and 4.03% higher BD-rates under DISTS and FID, respectively, than mixed fine-tuning.Source: §5.4 · p. 11, Table 6 · p. 12

6 CONCLUSION

ResARC compensates for quantization and generation residuals through a residual generator and a transmitted residual codec, supporting strong rate-perception performance among leading generative codecs.

6 CONCLUSION

  • ResARC models quantization and generation residuals separately, generating the former with a DiT and compressing the latter for decoder-side correction.The conclusion presents explicit residual modeling and compensation as an effective design principle for ultra-low bitrate generative image compression.Source: §6 · p. 12

A PSEUDO-CODE OF RESARC

  • Algorithms 1 and 2 summarize the encoding-decoding pipeline and quantization-residual sampling procedure.The encoder-decoder algorithm requires shared models and fixed sampling settings.Source: A PSEUDO-CODE OF RESARC · p. 17, §A.1 · p. 17

A.1 ENCODING AND DECODING

  • The decoder reuses the autoregressive context to synthesize the quantization residual, decodes the transmitted generation residual, and combines both with the reconstructed latent.Matching suffix-sampling realizations and causal symbol order keeps encoder and decoder states aligned.Source: §A.1 · p. 17

A.2 QUANTIZATION RESIDUAL GENERATION

  • The quantization-residual sampler starts from Gaussian noise and integrates a context-conditioned DiT velocity field from t = 1 to t = 0 using Heun updates, except for an Euler final step.The final sample is the predicted quantization residual.Source: §A.2 · p. 18

B GENERATION RESIDUAL CODEC

The Generation Residual Codec uses prefix-depth and autoregressive context to compress generation residuals, with a four-pass entropy model and staged training for latent and perceptual reconstruction.

B GENERATION RESIDUAL CODEC

  • The codec conditions its analysis, entropy, and synthesis transforms on prefix depth and a multi-scale context pyramid aggregated from the reconstructed latent and autoregressive hidden features.The context pyramid is injected into corresponding codec stages.Source: B GENERATION RESIDUAL CODEC · p. 18
  • The four-pass channel-spatial entropy model partitions channels into four groups and assigns spatial positions by row-column parity masks, enabling parallel processing within each pass.Each latent element is assigned to exactly one pass.Source: B GENERATION RESIDUAL CODEC · p. 18
  • The codec quantizes its gain-modulated analysis latent using a predicted quantization-step map and conditional mean, then entropy-codes the resulting integer tensor.The conditional mean and scale are updated using previously reconstructed latent values in later passes.Source: B GENERATION RESIDUAL CODEC · pp. 18–19
  • The entropy model assigns symbols discretized zero-mean Laplace probabilities and factorizes their conditional distribution according to the four-pass causal coding order.Each pass’s parameters depend on the context, prefix embedding, and symbols reconstructed in earlier passes.Source: B GENERATION RESIDUAL CODEC · p. 19
  • Training proceeds through latent-space rate-distortion training, image-space perceptual refinement, and joint fine-tuning; the decoder is adapted to the compensated latent distribution.The perceptual refinement freezes the analysis transform and entropy model while refining synthesis; the final codec stage jointly trains all components.Source: B GENERATION RESIDUAL CODEC · pp. 19–20

C TRAINING DETAILS

ResARC training proceeds through backbone and residual-branch optimization, followed by decoder adaptation, using filtered image data and prefix-depth sampling across residual training.

C TRAINING DETAILS

  • Training first fine-tunes the backbone, optimizes the two residual branches separately, and adapts the VAE decoder; the VAE encoder remains frozen throughout.Residual training and decoder adaptation sample prefix depths k ∈ {5, 6, 7, 8, 9, 10} on a balanced deterministic schedule.Source: C TRAINING DETAILS · p. 20

C.1 AUTOREGRESSIVE PRIOR FINE-TUNING

  • The Infinity-2B autoregressive prior is fine-tuned for 2,000 iterations on 1024 × 1024 images, then frozen for subsequent training stages.The fine-tuning uses AdamW, a global batch size of 64, and a learning rate of 6 × 10−5.Source: §C.1 · p. 20

C.2 MIXED-LATENT VAE DECODER FINE-TUNING

  • The VAE decoder is fine-tuned for 35,000 steps using an even mixture of continuous and quantized latents to maintain compatibility with both distributions.Adversarial training begins after 10,000 steps.Source: §C.2 · p. 20

C.3 QUANTIZATION RESIDUAL GENERATOR TRAINING

  • The quantization-residual generator uses a conditional DiT trained first with flow matching and then with image-space perceptual supervision.The DiT predicts 32 residual channels from autoregressive decoding features; the two stages run for 3,000 and 2,000 steps, respectively.Source: §C.3 · p. 21

C.4 GENERATION RESIDUAL CODEC TRAINING

  • The Generation Residual Codec is trained with latent-space rate-distortion supervision, synthesis-only perceptual refinement, and joint rate, reconstruction, and perceptual fine-tuning.The stages run for 5,000, 300, and 2,000 steps, respectively.Source: §C.4 · pp. 21–22

C.5 RESIDUAL-AWARE VAE DECODER ADAPTATION

  • After the residual branches are frozen, the VAE decoder is adapted for 1,000 iterations using the compensated latent representation.The final stage adapts the decoder to the latent distribution after both residual compensations.Source: §C.5 · p. 22

C.6 OPTIMIZATION SETTINGS

  • All stages use AdamW with linear warm-up, cosine learning-rate decay, and gradient-norm clipping at 1.Optimizer settings vary by component, including weight decay for the generation residual codec and zero weight decay for the quantization-residual generator and decoder stages.Source: §C.6 · p. 22

D BITSTREAM AND RATE ACCOUNTING

The generation-residual stream contributes a small share of the total bitrate, with its share declining as more token-prefix scales are transmitted.

D BITSTREAM AND RATE ACCOUNTING

  • Total bitrate accounting measures the text condition, transmitted token prefix, and generation-residual stream; the residual latent is entropy-coded using byte-rANS.Text rate is computed from its UTF-8 byte length.Source: D BITSTREAM AND RATE ACCOUNTING · p. 22
  • 1.98 × 10−4–3.14 × 10−4 bpp is the generation-residual bitrate across evaluated prefix depths, and its total-rate share falls from 6.64% at k = 5 to 0.36% at k = 10.The token prefix increasingly dominates the rate as prefix depth grows; Figure 11 visualizes component bitrates and the residual stream’s total-rate fraction on DIV2K.Source: D BITSTREAM AND RATE ACCOUNTING · p. 22, Figure 11 · p. 23

E EVALUATION PROTOCOL

The evaluation uses fixed-resolution DIV2K and CLIC2020 reconstructions and assesses distortion, perceptual similarity, and distributional fidelity. Additional rate-quality curves report competitive CLIP similarity with comparable PSNR and MS-SSIM at low bitrates.

E EVALUATION PROTOCOL

  • Evaluation uses 1024 × 1024 saved 8-bit RGB reconstructions from 100 DIV2K validation images and 428 CLIC2020 test images.The protocol evaluates distortion-oriented fidelity, perceptual similarity, and distributional fidelity; transmitted prefix depths are k ∈ {5, 6, 7, 8, 9, 10} within a 13-scale hierarchy.Source: E EVALUATION PROTOCOL · p. 23
  • PSNR and MS-SSIM measure distortion, while LPIPS, DISTS, and CLIP image-to-image similarity measure perceptual similarity.PSNR is averaged across images in dB; the perceptual metrics are computed at original resolution, with metric-specific input scaling and preprocessing.Source: E EVALUATION PROTOCOL · p. 23
  • Distributional fidelity is evaluated with FID, KID, CMMD, and FD-DINOv2 using image or patch feature samples.The protocol yields 2,500 DIV2K patches and 10,700 CLIC2020 patches for distribution estimation; Table 8 summarizes DIV2K rate decomposition in average bpp.Source: E EVALUATION PROTOCOL · p. 23, Table 8 · p. 24
  • On DIV2K, KID decreases as prefix depth increases and becomes negative at k = 9 and k = 10.The paper retains raw signed KID values for BD-rate computation and clips values below 2^-20 only for the logarithmic plot; Table 9 reports ResARC KID across prefix depths.Source: E EVALUATION PROTOCOL · p. 24, Table 9 · p. 24

F.1 ADDITIONAL RATE-QUALITY CURVES

  • At low bitrates, ResARC achieves competitive CLIP similarity while maintaining comparable PSNR and MS-SSIM.These additional curves complement the main perceptual evaluation with pixel-level and structural fidelity measures.Source: §F.1 · p. 24

F.2 BD-RATE COMPARISON

ResARC achieves lower BD-rates than ARPC across DISTS and distributional-fidelity metrics on both datasets, while remaining competitive on LPIPS. It also outperforms diffusion-based codecs on DISTS and overall distributional fidelity at comparable bitrates.

F.2 BD-RATE COMPARISON

  • 42.94% and 50.98% BD-rate reductions on DIV2K, and 46.51% and 51.22% on CLIC2020, under DISTS and FID respectively versus ARPC.Table 10 reports signed BD-rate results relative to ARPC across six perceptual quality metrics.Source: §F.2 · p. 24
  • ResARC has the lowest reported BD-rates under DISTS and all four distributional-fidelity metrics on both datasets, while remaining competitive under LPIPS.The reported distributional metrics are FID, KID, CMMD, and FD-DINOv2.Source: §F.2 · p. 24
  • At comparable bitrates, ResARC achieves lower DISTS and better overall distributional fidelity than diffusion-based codecs such as StableCodec.The comparison is reported against diffusion-based codecs including StableCodec.Source: §F.2 · p. 24

G MORE VISUAL RESULTS

Increasing transmitted prefix depth raises bitrate while progressively improving fine-detail and structure recovery. Additional qualitative comparisons report finer detail preservation and lower full-image DISTS for ResARC in every displayed example.

G.1 VISUAL EFFECT OF PREFIX DEPTH

  • Figure 12 adds rate-quality comparisons with leading generative codecs on DIV2K validation and CLIC2020 test images.The additional visual comparisons also include per-image bitrate and DISTS labels for CLIC2020 and DIV2K examples.Source: Figure 12 · p. 25, Figure 14 · p. 28, Figure 15 · p. 28
  • As prefix depth rises from k = 5 to k = 10, bitrate increases and fewer suffix scales need autoregressive generation, with progressively better fine-detail and structure recovery.The examples include butterfly markings, mushroom textures, railing patterns, and facial details; Figure 13 compares the same five images across prefix depths.Source: §G.1 · pp. 24–25, Figure 13 · p. 26

G.2 ADDITIONAL VISUAL COMPARISONS

  • ResARC preserves finer details and image structures and achieves the lowest full-image DISTS in every displayed CLIC2020 and DIV2K example.The examples include facial details, headlight contours, glass boundaries, plate rims, and petal textures; Figures 14 and 15 provide additional visual comparisons.Source: §G.2 · p. 25, Figure 14 · p. 28, Figure 15 · p. 28

Related papers

Loading 2609.39451v1…