PaperlayerView exact revision on arXiv

Source-linked AI summary

Channel-wise Autoregressive Entropy Models for Learned Image Compression

David Minnen, Saurabh Singh

arXiv:2007.08739v1eess.IVcs.CVcs.ITcs.LG

TL;DR

Learned image compression models use causal context effectively but decode serially, limiting parallel hardware utilization. This paper combines channel-conditioning and latent residual prediction, with round-based training, to reduce serial processing while improving rate-distortion performance. The resulting architecture reports substantial rate savings over context-adaptive models and BPG, especially at low bit rates.

  • Problem

    Causal backward adaptation improves entropy prediction but requires serial decoding, limiting efficient GPU and TPU utilization.

  • Method

    The paper combines channel-conditioning and latent residual prediction with training that uses rounded latent values in synthesis transforms.

  • Results

    6.7% and 11.4% average rate savings are reported over context-adaptive models on Kodak and Tecnick, respectively.

  • Takeaways & Limitations

    The combined architecture outperforms corresponding context-adaptive models while minimizing serial processing, with gains strongest at low bit rates.

  • Takeaways & Limitations

    The authors suggest that limited transform capacity or Adam optimization may prevent the models from reaching conditionally independent latent channels.

Abstract

from arXiv · show

In learning-based approaches to image compression, codecs are developed by optimizing a computational model to minimize a rate-distortion objective. Currently, the most effective learned image codecs take the form of an entropy-constrained autoencoder with an entropy model that uses both forward and backward adaptation. Forward adaptation makes use of side information and can be efficiently integrated into a deep neural network. In contrast, backward adaptation typically makes predictions based on the causal context of each symbol, which requires serial processing that prevents efficient GPU / TPU utilization. We introduce two enhancements, channel-conditioning and latent residual prediction, that lead to network architectures with better rate-distortion performance than existing context-adaptive models while minimizing serial processing. Empirically, we see an average rate savings of 6.7% on the Kodak image set and 11.4% on the Tecnick image set compared to a context-adaptive baseline model. At low bit rates, where the improvements are most effective, our model saves up to 18% over the baseline and outperforms hand-engineered codecs like BPG by up to 25%.

1. INTRODUCTION

Learned image codecs commonly combine forward adaptation through side information with backward adaptation from causal context. The paper targets comparable rate-distortion performance while reducing the serial decoding that limits hardware parallelism.

  • Autoencoder codecs transform pixels into quantized latent representations and reconstruct pixels through jointly optimized analysis and synthesis networks.
  • Forward adaptation uses side information such as local histograms or a learned hyperprior, which integrates efficiently into end-to-end networks.
  • Backward adaptation predicts symbols from causal spatial and previously decoded-channel context, but decoding is inherently serial on GPUs and TPUs.
  • The proposed architecture combines channel-conditioning and latent residual prediction to match context-adaptive rate-distortion performance while minimizing serial processing.Training synthesis transforms with rounded latent values further boosts rate-distortion performance.
  • 6.7% and 11.4% rate savings are achieved over context-adaptive models on Kodak and Tecnick, respectively.At low bit rates, savings exceed 16% versus the context-adaptive baseline and reach 25% relative to BPG.

2. CHANNEL-CONDITIONAL ENTROPY MODELS

Channel-conditioning replaces spatially autoregressive context with a small sequence of channel slices. Each slice can process its spatial positions in parallel while conditioning later slices on previously decoded slices and the hyperprior.

  • The hyperprior architecture transmits an image-dependent hyper-latent tensor and derives entropy parameters for compressing image latents.
  • In the two-slice design, the first slice uses only hyperprior parameters, while the second also uses the decoded first slice.
  • Channel-conditional models split the latent tensor into roughly equal channel slices and condition each slice's entropy parameters on previously decoded slices.
  • Channel-conditioning is autoregressive across channels rather than spatial dimensions, so each slice waits for earlier slices but processes its W × H × C/N values in parallel.
  • Spatial autoregression can require W × H sequential steps, whereas channel-conditioning exposes substantially more parallel computation.Wavefront processing adds some spatial parallelization but remains less parallel than channel-conditioning.

3. LATENT RESIDUAL PREDICTION

Latent residual prediction reduces the quantization error introduced when real-valued latents are quantized. It can improve both reconstruction distortion and entropy coding when later slices use residual-corrected earlier slices.

  • Quantizing real-valued latents creates residual error r = y − Q[y] that increases distortion after synthesis transforms reconstruct the image.
  • Latent residual prediction estimates this error from the hyperprior and previously decoded slices, then adds the prediction to quantized latents slice by slice.
  • Because later entropy parameters condition on earlier slices containing LRP corrections, latent residual prediction can reduce entropy as well as distortion.
  • Earlier synthesis-transform augmentations concatenated extra features with latents, increasing computation, and lacked channel-conditioning.Without channel-conditioning, those approaches could affect distortion but not entropy modeling.

4. TRAINING WITH ROUNDED LATENT VALUES

Quantization complicates gradient-based training because rounding yields unusable gradients. The paper therefore evaluates a mixed strategy that retains uniform noise for entropy learning but uses rounded tensors in synthesis transforms.

  • Quantization hinders gradient-based optimization because its gradients are zero or infinite.
  • Common workarounds replace quantization with uniform noise or apply straight-through gradients while retaining rounding.
  • The mixed approach uses uniform noise for entropy-model learning but passes rounded tensors to synthesis transforms.
  • Using the rounding-based method everywhere performed worse than the uniform-noise baseline.

5. EXPERIMENTAL RESULTS

The full model combines channel-conditioning, latent residual prediction, and round-based training to improve rate-distortion performance, especially at low bit rates. Results show gains over context-adaptive models and BPG, while isolating benefits from the individual components.

  • Overall performance: The full model outperforms standard codecs and learning-based spatial-context codecs on Kodak rate-distortion curves.The full model uses 10 CC slices, LRP, and round-based training.
  • Overall performance: 13.9% average BD rate savings over BPG and 6.7% over the context-adaptive baseline are achieved by the largest 10-slice model.The model’s savings over BPG range from 10% at higher quality levels to 25% at the lowest.
  • Channel-conditioning: More channel-conditional splits improve rate savings by creating additional opportunities to model dependencies between latent channels.The benefit diminishes as the number of splits increases and requires extra computation.
  • Latent residual prediction: More than 6% compression improvement from LRP occurs at low bit rates for the model with 10 channel-conditioning splits.LRP slightly reduces rate-distortion performance at high bit rates and has little benefit without channel-conditioning.
  • Rounding-based optimization: Round-based training provides significant benefits at low and moderate bit rates compared with uniform-noise training.For the 5-split CC plus LRP model, savings exceed 15% at the lowest bit rates.

6. DISCUSSION

The discussion interprets the gains from channel-conditioning and latent residual prediction as evidence that existing learned codecs are not yet ideal. It attributes this gap potentially to transform capacity or optimization limitations while emphasizing the resulting architecture’s parallelizability.

  • Interpretation: Significant gains from CC and LRP suggest that existing models do not yet produce ideally conditionally independent latent channels.The ideal would be a latent space whose channels are conditionally independent given the hyperprior.
  • Open questions: The observed improvements may reflect insufficient capacity in the four-layer analysis and synthesis transforms or difficulty finding a suitable Adam-optimized minimum.The authors identify these as possible explanations and state that further research is needed.
  • Implications: The architecture outperforms a corresponding context-adaptive model while minimizing serial processing.The paper proposes combining channel-conditioning, latent residual prediction, and round-based training.
  • Training effects: Round-based training shows a significant benefit at low and moderate bit rates, complementing the discussion’s evidence that training choices affect rate-distortion performance.The comparison is against using uniform noise everywhere.

A. ARCHITECTURE DETAILS

The architecture uses slice-specific channel-conditioning and latent residual prediction transforms, with configurations tailored to varying slice inputs. Simplified GDN and rounded-latent training improve training behavior without reducing rate-distortion performance.

  • Slice-specific transforms: Each channel-conditioning and latent residual prediction transform is trained separately for each slice, with output depth latent_depth / num_slices.Later slices receive concatenated previous slices, increasing input depth from 320 for slice 1 to 608 for slice 10 in the example.
  • Latent depth: The standard low- and moderate-rate models use 320 latent channels, while high-rate Kodak models use 512 channels.The larger bottleneck improves rate-distortion performance above roughly 2.0 bpp on Kodak.
  • Normalization: The model uses a simplified GDN in which αij and εi are fixed at 1.0 while βi and γij remain trainable parameters.The symbols xi and zi denote input and output vectors, and i and j index channels.
  • Slice sizing: When latent_depth is not evenly divisible by num_slices, all slices use the floor quotient except the final slice, which receives the remaining channels.Architectures typically choose num_slices that evenly divides latent_depth.
  • Training interaction: Rounded-latent synthesis-transform training is slightly faster and more stable without reducing rate-distortion performance.

B. TRAINING DETAILS

Training uses long Adam optimization on randomly processed web-image patches, with staged learning-rate and rate-distortion-weight schedules. Partial conditioning reduces transform size and enables more parallel computation, while retaining most rate-distortion performance.

  • Optimization: All experimental models were trained for 5,000,000 Adam steps with a staged learning-rate schedule.Later experiments found that a better schedule could improve rate-distortion performance by 1–2% even with 4M total steps.
  • Optimization: The rate-distortion weight is doubled during the first half of training before switching to the target λ.Higher λ encourages lower distortion and higher bit rate and appears to help low-bit-rate models avoid a sub-optimal entropy model.
  • Data preparation: Training uses nearly 2M filtered web images, batch size eight, and randomly cropped 256×256 patches after random downscaling.Downscaling reduces pre-existing compression artifacts and discourages overfitting to a particular scale.
  • Partial conditioning: Reducing slice support decreases later-transform size and creates more opportunities for parallel calculations, improving runtime.
  • Partial conditioning: Conditioning on the previous N slices performs slightly worse and offers fewer parallelization opportunities than conditioning on the first N slices.
  • Partial conditioning: Reducing slice support causes a relatively small rate-distortion reduction, while zero supporting slices performs much worse in the 10-slice setting.The authors hypothesize that early slices preserve high-level information needed by later slices.

C.2. Smaller Hyperprior and µ′, σ′ Tensors

The hyperprior-derived mean and scale tensors are relatively wide, but experiments show that shrinking related transforms and tensor depths has limited rate-distortion impact. Partial slice support also remains effective down to two supporting slices.

  • Hyperprior tensors: Each slice predicts its own mean and scale values from previous slices and hyperprior tensors, whose size affects transform size and speed.The main-paper models use 320 channels for the hyperprior-derived tensors.
  • Slice support: Reducing supporting slices in a 10-slice model incurs relatively little penalty down to two slices, whereas zero slices substantially worsens rate-distortion performance.The figure measures extra space as negative rate savings.
  • Hyper-analysis transform: Shrinking the hyper-analysis transform from [320, 256, 192] to [128, 86, 64] has only a small impact on rate-distortion performance.

D. RATE-DISTORTION COMPARISONS

Rate-distortion comparisons show that the proposed model performs strongly against learning-based and standard codecs across image-quality settings. Its rate savings are especially large at lower quality levels and remain substantial across all quality levels relative to JPEG.

  • Architecture sensitivity: Reducing hyper-synthesis output channels from 320 to 240 increases file size by less than 2%, while 40-channel tensors are at most about 5% larger.
  • JPEG comparison: More than 70% rate savings versus JPEG occur at lower quality levels, while savings exceed 55% at all quality levels.The comparison uses equivalent PSNR and JPEG 4:2:0 as the reference.
  • Reconstructed images: At very high compression rates, MS-SSIM optimization typically preserves texture better but performs worse on high-contrast and high-frequency content such as text.The reconstructed-image examples compare BPG with models optimized for MSE, L1, and MS-SSIM.
  • Loss functions: The benefits of channel-conditioning and latent residual prediction are independent of the distortion loss function used.The paper identifies combining the entropy model with more sophisticated perceptual metrics as future research.
  • JPEG comparison: The rate-savings graph compares the proposed method with a wide range of learning-based and standard codecs on Kodak using PSNR.

F.1. Samples with a Random Hyperprior

The channel-conditional compression model can be treated as a generative model by sampling a hyperprior and then latent slices, producing random images that reveal architectural and representational properties. These samples show limited semantic coherence, while channel-wise conditioning provides broader receptive fields than earlier models but less long-range coherence than spatial autoregression.

  • Sampling procedure: The model generates random images by sampling a hyperprior, deriving slice-wise entropy models, sampling each latent slice, and applying the synthesis transform.The procedure first samples the factorized hyperprior, then obtains µ′ and σ′, samples latent slices sequentially, concatenates them, and reconstructs an RGB image.
  • Sample structure: Fully convolutional models exhibit block artifacts because strided convolutions downscale data and the highest-level factorized entropy model makes samples spatially independent.The artifacts are especially visible in earlier, less sophisticated architectures.
  • Sample structure: The channel-wise autoregressive model has less long-range coherence than the spatially autoregressive model but a larger receptive field than earlier models.Its receptive field grows with each successive latent slice.
  • Known-hyperprior interpretation: The hyperprior stores low-frequency color data together with some texture and orientation information, including recognizable fence and building patterns.Using a real image's hyperprior requires 0.0143 bpp, nearly 1680x compression compared with typical image-compression rates.
  • Progressive decoding: Progressive decoding produces a rough image from the hyperprior and improves it after each latent slice, but its rate-distortion curve is much worse than separately optimized models.The full synthesis transform must be rerun for every progressively decoded image, making the approach costly and mainly useful for temporary previews.
  • Semantic content: Random samples from a model trained on nearly two million semantically unconstrained web images show local coherence but no obvious semantic information.The authors suggest that this may reflect the MSE loss or insufficient model capacity, unlike GANs trained on semantically constrained domains.
Loading 2007.08739v1…