PaperlayerView exact revision on arXiv

Source-linked AI summary

PIC: Revisiting INR for Image Coding with Fast Encoding and Sub-Millisecond Decoding

Xiang Liu, Jinxiang Wang, Bin Chen, Zimo Liu, Mingyao Hong, Jiawei Li, Yaowei Wang, Shu-tao Xia

arXiv:2609.09020v1cs.CV

TL;DR

Representation-based image coding offers low decoding complexity but often requires minutes or hours to encode each image, limiting practical deployment. PIC generates a neural representation in one forward pass; its encoder is faster than prior representation-based approaches, and its decoder outperforms JPEG in speed at comparable rate-distortion performance.

  • Problem

    Representation-based methods offer low decoding complexity and can rival traditional codecs in rate-distortion performance, but encoding often takes minutes or hours per image.

    Source: §1 · p. 2

  • Method

    PIC uses an end-to-end trained network to generate a low-bit-rate neural representation in a single forward pass.

    Source: §1 · p. 2

  • Results

    The encoder is substantially faster than prior representation-based approaches, while the decoder outperforms JPEG in speed at comparable rate-distortion performance.

    Source: §5 · p. 15

  • Takeaways & Limitations

    PIC provides a foundation for extending INR-based compression to other modalities.

    Source: §5 · p. 15

    The decoder's complexity is higher than JPEG's, although hardware optimization lets it surpass commercial decoders in speed.

    Source: §3.3 · p. 9

Abstract

from arXiv · show

Implicit neural representation (INR) has achieved remarkable progress in novel view synthesis and image/video coding in recent years.Compared to conventional end-to-end image codecs, INR-based compressors demonstrate significant advantages in decoding complexity. However, their practical application has been hindered by the inferior encoding speed and underutilized decoding efficiency.In this work, we propose a feedforward INR image coding architecture, Practical INR Image Codec (PIC), that computes all the necessary information for INR network in a single forward pass, achieving an encoding speed of 20 FPS. Additionally, we implement a highly optimized decoder that reaches 2000 FPS decoding speed, significantly surpassing JPEG's performance at comparable rate-distortion (RD) performance. To the best of our knowledge, this work presents the first learning-based image codec that simultaneously outperforms or is comparable with JPEG in both RD performance and decoding speed while maintaining practical encoding speed. Code is available at https://github.com/actcwlf/PIC.

1 Introduction

Representation-based codecs offer low decoding complexity and can rival traditional codecs in rate-distortion, but their slow encoding limits deployment. PIC uses a feedforward architecture to balance encoding, decoding, and rate-distortion performance.

1 Introduction

  • Representation-based codecs can have very low decoding complexity and rival traditional codecs in rate-distortion, but often require minutes or hours to encode each image.Source: §1 · p. 2
  • PIC directly produces a low-bit-rate neural representation in one forward pass, combining fast encoding, ultra-fast decoding, and competitive rate-distortion performance.Source: §1 · p. 2
  • The optimized decoder is designed to translate representation models’ low-complexity decoding into practical high decoding speeds.Source: §1 · p. 2
  • PIC improves rate-distortion performance and encoding/decoding speed over prior representation-based approaches, and surpasses nvJPEG in decoding speed.Source: §1 · p. 3

2 Related Work

Related work spans learned transform codecs and representation-based compression, whose low decoding complexity can come with slow encoding. The paper identifies hypernetworks as a basis for feedforward INR encoding that considers both rate-distortion and coding efficiency.

2.1 End-to-end Image Compression

  • End-to-end codecs use neural networks as analysis and synthesis transforms, with quantization and estimated rate incorporated into rate-distortion optimization.Single-forward-pass encoding in end-to-end methods motivates exploring similar encoding schemes for INRs.Source: §2.1 · pp. 3–4

2.2 Representation-based Image Compression

  • Representation-based methods include coordinate-to-target mappings, learnable parameter or grid inputs, and structured models such as 3D Gaussian Splatting.Source: §2.2 · p. 4
  • These methods offer low decoding complexity and high reconstruction quality, but training-based encoding hinders deployment and the strengths are difficult to achieve together.Source: §2.2 · p. 4

2.3 Hypernetwork for INR

  • Hypernetwork-based INR methods have been explored in other tasks, but remain insufficiently investigated for compression; this work explores a feedforward INR encoder targeting rate-distortion and coding efficiency.Source: §2.3 · p. 5

3 Method

PIC generates a low-bit-rate INR representation in one forward pass, repurposing representation models for compression and designing entropy estimation, symbol coding, and hardware-aware decoding. Its pipeline combines multiresolution latents, DCT-based rate estimation, and a compact reconstruction network.

3 Method

  • PIC replaces per-image representation-model training with a feedforward encoder that generates the representation weights in one step.The method targets the slow encoding that results when encoding itself requires training.Source: §3 · p. 5, §3.1 · p. 6

3.1 Repurposing Representation Models for Compression

  • gr represents images with pyramid-like multiresolution latents that are jointly optimized with the synthesis-network weights.The latents are upsampled to the output resolution before reconstruction.Source: §3.1 · pp. 5–6
  • Channel-wise normalization and modulation split gs into shared MLPs and instance-dependent MLPi, with gm-produced γ and β fused into MLPi weights.This lets the modulation network generate corresponding reconstruction-network weights for each input.Source: §3.1 · p. 7

3.2 Entropy Estimation

  • PIC estimates DCT-parameter distributions for rate estimation; its 64-channel model treats DCT components independently and uses uniform-noise relaxation for end-to-end training.Latents are divided into 8 × 8 patches and transformed with DCT before entropy estimation.Source: §3.2 · pp. 7–8

3.3 Full Pipeline and Hardware-affinity Implementation

  • ZpR zigzag-reorders quantized DCT symbols, trims trailing zeros, and partially run-length-encodes zeros; RLE also reduces image-specific Huffman-table storage overhead.The symbol coding table is image-specific, and its storage overhead matters particularly for small images.Source: Figure 2 · p. 8, §3.3 · p. 8
  • The final objective balances entropy and distortion, while width-16 reconstruction layers align with Tensor Core TF32 operations and Slang keeps training consistent with inference.The distortion metric and λ set the rate-distortion trade-off; the implementation relies on NVIDIA Tensor Cores, with similar functionality noted for AMD rocWMMA.Source: §3.3 · pp. 8–9
  • The decoder has higher complexity than JPEG's, yet the authors report surpassing commercial decoders by fully leveraging hardware capabilities.The paper presents hardware use as the means by which the decoder achieves this performance.Source: §3.3 · p. 9

4 Experiments

PIC is evaluated on Kodak and CLIC against representation-based, neural, and traditional codecs. It offers competitive rate-distortion performance and fast decoding, while experiments examine visual quality, architecture choices, bitrate estimation, and decoder speed.

4.1 Experiment Setup

  • Training uses 84,991 LSDIR images; evaluation covers 24 Kodak images and 41 high-resolution CLIC images, with PSNR, MS-SSIM, LPIPS, and BPP reported.The setup also reports encoding and decoding time and complexity.Source: §4.1 · pp. 9–10

4.2 Quantity and Quality Results

  • At high bit rates, PIC achieves better PSNR than JPEG and comparable MS-SSIM and LPIPS, while outperforming representation-based methods with similar decoding speed across quality metrics.The comparisons cover Kodak and CLIC; the reported results are averaged under each method’s corresponding hyperparameter settings.Source: Figure 3 · p. 10, Figure 4 · p. 11, §4.2 · pp. 11–13

4.1 Experiment Setup

  • Benchmarks include representation-based codecs, Cool-Chic, learned Factorized and Hyperprior models, nvJPEG, and HEVC; experiments run on one RTX 4090D.PIC uses MSE as its distortion loss, while other methods retain their original loss settings.Source: §4.1 · pp. 10–11

4.2 Quantity and Quality Results

  • On two visualized images, PIC is marginally superior to JPEG and comparable in texture details and artifacts, while outperforming COIN and GaussianImage.The comparisons use the best-quality original-paper settings for COIN and GaussianImage on the CLIC image.Source: Figure 5 · p. 12, §4.2 · p. 13
  • PIC decodes faster than Factorized and Hyperprior models and accelerates decoding by more than three orders of magnitude over other representation-based approaches, ranking second only to GaussianImage.Its encoding speed is comparable to Factorized and Hyperprior models.Source: §4.2 · p. 12
  • PIC still has a rate-distortion gap to state-of-the-art approaches, although it surpasses JPEG and is described as practically deployable.The paper presents this as an overall qualification of PIC’s practical performance.Source: §4.2 · p. 13

4.3 Ablation Study

  • Sharing one synthesis network across images significantly degrades performance, as do removing normalization or modulation; the full synthesis design is needed for the desired performance.The shared-network variant degenerates into an end-to-end model, with limited decoder capacity identified as the reason for its inferior performance.Source: §4.3 · pp. 13–14
  • Entropy-network approximation causes unavoidable bitrate discrepancies that post-processing may amplify, though estimated and actual rates remain broadly consistent; width 16 is the preferred decoder setting.Width 8 and width 16 have no significant decoding-speed difference because width 8 is zero-padded to a 16 × 16 matrix for Tensor Cores; width 16 balances speed, rate-distortion, and hardware compatibility.Source: §4.3 · p. 14

5 Conclusion

PIC generates INR network parameters in a single forward pass, with an encoder faster than prior representation-based approaches and a decoder faster than JPEG at comparable rate-distortion performance. The appendices describe fusing modulation into the synthesis network and CUDA optimizations for symbol expansion and synthesis.

5 Conclusion

  • PIC generates all network parameters in a single forward pass, making its encoder substantially faster than prior representation-based approaches.The conclusion describes PIC as an end-to-end INR image-coding framework.Source: §5 · p. 14
  • PIC’s optimized decoder outperforms JPEG in speed while delivering comparable rate-distortion performance, though there remains headroom to improve rate-distortion performance.The conclusion presents the framework as a practical learning-based codec and a foundation for extending INR compression to other modalities.Source: §5 · pp. 14–15

A Fusing Modulations to Synthesis Net

  • The encoding process fuses normalization and modulation parameters into MLPi; decoding then requires only its fused weights and biases.The experiments use the first three layers as MLPs and the last layer as MLPi.Source: A Fusing Modulations to Synthesis Net · pp. 18–19

B.1 Fast Chunking Algorithms

  • The fast chunking algorithm precomputes end-of-block offsets to parallelize pRLE zero-symbol expansion, while synthesis layers are fused into one CUDA kernel using WMMA matrix multiplications.The algorithm uses thrust::copy_if to obtain the offsets.Source: §B.1 · p. 19

B.2 Details of Baseline Methods

The baseline comparisons reproduce COIN and GaussianImage using their original code and retain each baseline’s original loss settings.

B.2 Details of Baseline Methods

  • Baseline evaluations reproduce COIN and GaussianImage with original code and preserve original loss settings, while using MSE-trained versions where methods offer metric-specific models.PIC itself uses MSE as its training distortion metric; the Factorized model uses its MSE-optimized pretrained version.Source: §B.2 · p. 19

B.3 Evaluation Protocol

The evaluation protocol controls hardware access and JPEG implementation choices for speed comparisons, and measures serial decoding without bitstream disk I/O. The section also describes parallel expansion of encoded symbols using precomputed end-of-block offsets.

B.3 Evaluation Protocol

  • Speed measurements use exclusive hardware access because occupancy and PyTorch’s asynchronous design can interfere, and compare against a GPU-optimized JPEG binding implemented because nvNVJPE is closed source.The implementation also optimizes nvJPEG input and output components.Source: §B.3 · pp. 19–20
  • Symbol expansion uses end-of-block offsets to divide the stream among CUDA threads and expands pRLE symbols into zero runs in zigzag order.The algorithm writes each expanded block into a 64-symbol output region.Source: §B.3 · p. 20
  • All methods are evaluated by serially decoding dataset images in the same process, with JPEG bitstreams kept in memory rather than written to disk.The protocol states that same-process encoding and decoding warms up nvJPEG and avoids I/O time.Source: §B.3 · p. 20

C.1 Extended Results

Extended results compare rate-distortion performance, encoding and decoding speed, and decoding complexity, while latent visualizations show how detail and bit allocation shift across settings.

C.1 Extended Results

  • PIC shows balanced rate-distortion, speed, and complexity capabilities without significant weaknesses; autoencoders encode quickly but have high complexity, while Cool-chic trades low complexity for slower encoding and decoding.Fig. 8 uses BD-rate as the vertical axis for comparing these dimensions.Source: §C.1 · p. 20, Figure 8 · p. 21
  • The nvJPEG timing snippet synchronizes CUDA around the timed loop and reports total decoding time divided by the number of images.The snippet iterates over the encoded-image batch.Source: §C.1 · p. 21

C.2 Latents Visualizations

  • At smaller λ, higher-resolution latents retain more detail, whereas at bigger λ, more detail appears in relatively lower-resolution latents, reflecting bit-allocation tendencies.The visualizations use λ = 0.01 and 0.001; normalized views do not directly indicate symbol magnitudes.Source: §C.2 · p. 21

C.3 Ablation study of model architecture

The ablations examine run-length encoding choices and synthesis-network architecture, identifying settings that favor coding performance and practical decoding.

C.3 Ablation study of model architecture

  • Figure 9 visualizes the unnormalized value for each latent.Source: Figure 9 · p. 22
  • RLE on 0 achieves the best performance among four settings, and can outperform encoding additional symbols in some situations.The authors also note that RLE on 0 is easier to implement.Source: §C.3 · pp. 21–22, Table 2 · p. 22
  • Placing MLPi at the end of the synthesis network gives the best performance among the tested positions.The table compares beginning, middle, and end placements, with the default end placement as the BD-Rate reference.Source: §C.3 · p. 22
  • Across the widths examined, a synthesis-network width of 16 is the optimal choice when considering decoding speed, rate-distortion performance, and hardware compatibility.Source: §C.3 · p. 22

C.4 Ablation of modulation

The modulation mechanism is tested on an architecture similar to N-O Cool-Chic; the modulated variant surpasses PIC in rate-distortion performance.

C.4 Ablation of modulation

  • Figure 10 visualizes normalized values for each latent.Source: Figure 10 · p. 23
  • The modulation mechanism is effective on an architecture similar to N-O Cool-Chic, with modulation applied between its first two linear layers.Source: §C.4 · pp. 22–23
  • N-O Cool-Chic* with modulation surpasses PIC in rate-distortion performance.The passage identifies this as potential for further advances in the end-to-end INR paradigm's architecture and training recipes.Source: §C.4 · p. 23

C.5 Ablation of bitstream

The bitstream analysis finds that encoded latents dominate the Kodak bitstream, while codebook size has little impact on total encoded size.

C.5 Ablation of bitstream

  • Encoded latent variables account for the vast majority of the bitstream on Kodak, while codebook size increases with the bitstream but has little impact on overall encoded size.Source: §C.5 · p. 23, Figure 12 · p. 24
  • Table 4 reports a synthesis-network-width ablation, with BD-Rate calculated relative to width 16.Source: Table 4 · p. 24
  • Figure 12 shows the proportions of different components in the encoded bitstream on the Kodak dataset.Source: §C.5 · p. 23, Figure 12 · p. 24
Loading 2609.09020v1…