PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-scale Image Representation Compression

Tianhao Peng, Ho Man Kwan, Fan Zhang, Shan Liu, David Bull

arXiv:2609.04274v1eess.IVcs.CVcs.MM

TL;DR

Overfitted image codecs have promising performance, but their weights and quantization parameters are not jointly optimized under the rate-distortion objective. MIRC combines joint optimization with a shared multi-scale representation; on CLIC2020 professional validation, it achieves a 10.5% BD-rate saving against VVC VTM 22.0.

  • Problem

    Overfitted image codecs train latents for rate-distortion but leave network weights full precision and select their quantization parameters separately after training.

    Source: §I · p. 1

  • Method

    MIRC jointly optimizes the image representation, entropy model, and quantization parameters, and uses a multi-scale representation with parameter sharing.

    Source: §I · pp. 1–2

  • Results

    10.5% BD-rate saving against VTM 22.0 is reported for MIRC on CLIC2020, and its configurations span 1.2 to 2.9 kMAC per pixel.

    Source: §I · p. 2, B. Rate-distortion performance · p. 4

  • Takeaways & Limitations

    MIRC offers shared multi-scale decoder configurations at different decoding budgets and performs on par with the strongest overfitted image codec.

    Source: §IV · p. 4

    Encoding remains costly, making acceleration of the overfitting process the main direction for future work.

    Source: §IV · p. 4

Abstract

from arXiv · show

Overfitted codecs have demonstrated promising performance for image and video compression. In particular, for image compression, the Cool-chic family of models has shown competitive performance against scene-agnostic models, with orders of magnitude lower decoding complexity at the cost of a longer overfitting process. However, these overfitted image codecs are not fully optimized toward the rate-distortion objective: their network weights remain in full precision during training, and the associated quantization parameters are selected in a separate post-training stage. Furthermore, their synthesis operates at a single scale, which overlooks cross-scale redundancy. In this paper, we propose MIRC, an overfitted image codec in which every coded component, including the latents, the synthesis network, and the entropy models, is quantized and entropy coded under a single rate-distortion objective, adopting the end-to-end compression pipeline of the neural video representation codec NVRC. We further introduce a multi-scale representation with cross-stage parameter sharing, which improves coding efficiency at a small transmitted overhead. On the CLIC2020 professional validation set, MIRC achieves a 10.5% BD-rate saving against VVC (VTM 22.0). Moreover, MIRC offers a family of configurations spanning 1.2 to 2.9 kMAC per pixel, so the decoding budget can be selected to match the deployment target.

I. INTRODUCTION

Neural image codecs have advanced in compression performance but remain computationally challenging to deploy, while overfitted codecs offer a lighter alternative with substantial encoding time. MIRC jointly optimizes coded components and adds a shared multi-scale representation; on CLIC2020 professional validation, it saves 10.5% BD-rate against VVC VTM 22.0.

I. INTRODUCTION

  • Neural image codecs have advanced compression performance, but their high computational complexity challenges deployment on typical consumer devices.Overfitted codecs instead fit a compact network to one image, though their overfitting process makes encoding time substantial.Source: §I · p. 1
  • MIRC optimizes the image representation, entropy model, and quantization parameters together under a single rate-distortion objective, avoiding post-training parameter selection.Existing overfitted codecs train the representation first and select quantization parameters afterward, which the paper describes as limiting representation capacity and compression performance.Source: §I · p. 1, Figure 1 · p. 1
  • MIRC adds a multi-scale representation with parameter sharing to retain HiNeRV's representational capacity while reducing overhead for single-image coding.The original HiNeRV representation's large parameter cost is amortized effectively over video frames, making direct image coding inefficient.Source: §I · p. 2
  • 10.5% BD-rate saving against VVC VTM 22.0 (All Intra) on CLIC2020 professional validation is reported for MIRC.MIRC also achieves at least 5% coding gains over other state-of-the-art overfitted codecs, including Cool-chic 4.0.Source: §I · p. 2

II. METHOD

MIRC codes an image-specific representation consisting of multi-scale latent grids and a lightweight synthesis network. Entropy models code the latents and synthesis weights, and a meta entropy model codes those models for inclusion in the bitstream.

II. METHOD

  • MIRC represents each image with optimized parameters that are compressed into a bitstream for transmission.The representation contains multi-scale latent grids and a lightweight synthesis network that decodes them into the reconstructed image.Source: §II · p. 2
  • An autoregressive entropy model codes the latent grids, while a non-parametric weight entropy model codes the synthesis weights.Both entropy models are themselves coded by a meta entropy model and included in the bitstream.Source: §II · p. 2
  • The compression framework covers quantization, entropy coding, end-to-end optimization, and the proposed multi-scale representation.Source: §II · p. 2

A. Compression framework

MIRC quantizes and entropy-codes its coded components under a single rate-distortion objective. It uses separate entropy models for latent grids and synthesis weights, with hierarchical coding for the model parameters.

A. Compression framework

  • Latents use one shared quantization step across grids, while each weight tensor has its own transmitted step, adaptively set for fixed 8-bit quantization.The weight step is computed from that tensor’s minimum and maximum; transmitted steps add negligible per-tensor overhead.Source: A. Compression framework · p. 2
  • The autoregressive model codes spatially correlated latents by predicting Gaussian mean and scale from previously decoded neighbors.The residual MLP maps each causal context to the distribution parameters.Source: A. Compression framework · p. 2
  • A shared non-parametric model codes synthesis weights, and a meta entropy model codes the entropy-model parameters hierarchically.Sharing the weight model across synthesis layers keeps its transmitted overhead from growing with the network.Source: A. Compression framework · pp. 2–3
  • The objective sums rates for quantized weights, entropy-model parameters, and latents with λ-weighted distortion; soft-rounding with additive Gaussian noise approximates rounding during training.The rate terms are negative log-likelihoods under the entropy models, and distortion can be mean squared error.Source: A. Compression framework · p. 3
  • Gradients reach every coded component, including entropy-model weights, unlike Cool-chic’s full-precision network weights and post-training quantization-parameter search.This places the coded components within the same rate-distortion optimization.Source: Figure 2 · p. 2, A. Compression framework · p. 3

B. Multi-scale representation

MIRC’s multi-scale decoder pairs latent grids with decoder stages to modulate features, while optional cross-stage parameter sharing reduces the overhead of coding additional stages.

B. Multi-scale representation

  • At each stage, upsampled features are modulated by affine parameters projected from the quantized latent grid, then processed by decoder blocks.The first stage has no upsampled features and is driven by the predicted shift.Source: B. Multi-scale representation · p. 3
  • Latent grids match decoder scales: the first stage uses an L-level pyramid, while each later upsampling stage adds a single-level grid.The finest first-stage level and later grids match their decoder-feature resolution; the remaining pyramid levels are lower resolution.Source: B. Multi-scale representation · p. 3
  • Each decoder block uses depthwise-separable convolution, GELU, and a skip connection; a final linear layer and sigmoid produce image pixels.Source: B. Multi-scale representation · p. 3
  • Sharing the n-th decoder block across stages, with per-stage AdaLN scaling and shifting, helps limit the added parameter overhead of a multi-scale decoder.A shared block can be reused across scales or trained toward the scale contributing most to the loss.Source: B. Multi-scale representation · p. 3

A. Experimental setup

The experiments evaluate MIRC on Kodak and CLIC2020 professional validation using PSNR, compare it with conventional, overfitted, and autoencoder codecs, and report rate-distortion, complexity, and timing.

A. Experimental setup

  • Evaluation uses Kodak’s 24 images at 768 × 512 and 41 CLIC2020 professional images at resolutions from 384×512 to 1370×2048, measuring RGB-space PSNR.Source: A. Experimental setup · p. 3
  • Comparisons include VTM 22.0 All Intra, Cool-chic 4.0, C3, MoRIC, and MLIC++; C3 and MoRIC rates are cross-entropy estimates, unlike the real bitstream sizes reported for the others.Source: A. Experimental setup · p. 3
  • The default model uses four stages, two blocks per stage, and three first-stage latent-grid levels; each image is optimized for 100K iterations at eight rate points.The rate points set λ from 32 to 4096; training uses mixed precision, Adam, and a cosine learning-rate schedule from 10−2 to 10−6.Source: A. Experimental setup · pp. 3–4
  • Fig. 3 plots PSNR versus bpp on Kodak and CLIC2020, and BD-rate against VTM 22.0 versus decoding MAC per pixel.Source: Figure 3 · p. 4
  • Table I reports encoding and decoding time and complexity averaged on CLIC2020 professional using an NVIDIA RTX 4090 GPU and Intel Core i7-14700 CPU.Orange and blue indicate GPU and CPU computation, respectively.Source: Table I · p. 4
  • MIRC’s Low, Mid, and High configurations vary synthesis and autoregressive-model channels; only High uses the shared decoder block.The respective channel pairs are (8, 8), (12, 12), and (16, 16).Source: A. Experimental setup · p. 4

B. Rate-distortion performance

On CLIC2020 professional, MIRC reports a VTM 22.0 BD-rate saving at its largest configuration and retains gains as decoding complexity decreases.

B. Rate-distortion performance

  • −10.5% BD-rate against VTM 22.0 at its largest configuration, with at least 5% coding gains over other state-of-the-art overfitted codecs.Source: B. Rate-distortion performance · p. 4
  • A reduction from 2.9 to 2.0 kMAC per pixel costs 0.8% in BD-rate, while the 1.2 kMAC per pixel configuration reaches −6.5%.Source: B. Rate-distortion performance · p. 4
  • At around half the decoding complexity, the smallest MIRC variant outperforms every configuration of the other overfitted codecs.Source: B. Rate-distortion performance · p. 4
  • Table II reports ablation BD-rate relative to default MIRC High on ten CLIC2020 professional images.Source: Table II · p. 4

C. Ablation study

The ablations report gains from end-to-end optimization, multi-scale decoding, and block sharing, with the multi-scale benefit dependent on sharing.

C. Ablation study

  • The tested components are a single-scale decoder, an unshared multi-scale decoder, and a variant without learned network-parameter entropy models or the quantization proxy.The study reports that all three components improve compression performance.Source: C. Ablation study · p. 4
  • Removing end-to-end optimization hurts most (+13.01%), underscoring the importance of jointly optimizing parameter distributions with the quantization proxy.Source: C. Ablation study · p. 4
  • Multi-scale decoding without block sharing is worse than single-scale decoding (+4.95% versus +2.85%), so adding scales pays off only when blocks are shared.Source: C. Ablation study · p. 4

IV. CONCLUSION

MIRC combines a shared multi-scale decoder with joint optimization of coded components. It performs on par with the strongest overfitted image codec overall and outperforms other overfitted codecs at matched decoding complexity, while encoding remains costly.

IV. CONCLUSION

  • At matched decoding complexity, MIRC outperforms all other overfitted codecs by a clear margin and performs on par with the strongest overfitted image codec overall.Its shared multi-scale decoder spans a range of decoding budgets without post-training tuning of the coding pipeline.Source: §IV · p. 4
  • MIRC represents images with multi-scale latent grids and a lightweight multi-stage decoder, jointly optimizing every coded component under one training objective.The shared decoder spans a range of decoding budgets.Source: §IV · p. 4
  • MIRC’s encoding remains costly, making acceleration of the overfitting process the main direction for future work.Source: §IV · p. 4

D. Bull, “Enhanced neural video representation compression

The supplied section contains a bibliographic fragment for a 2026 arXiv preprint.

D. Bull, “Enhanced neural video representation compression

Related papers

Loading 2609.04274v1…