PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Image Understanding from Deep Compression without Decoding

Robert Torfason, Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, Luc Van Gool

arXiv:1803.06131v1cs.CV

TL;DR

The paper asks whether image understanding can operate directly on learned compressed representations instead of decoded RGB images. It integrates compression encoders with classification and segmentation networks, achieving comparable or better performance with up to 2× lower computational complexity, especially at aggressive compression rates.

  • Problem

    Image understanding commonly operates after compressed images are decoded to RGB, leaving the usefulness of learned compressed representations for direct inference to be established.

  • Method

    The paper adapts learned compression representations for image classification and semantic segmentation, and jointly trains compression and classification networks when optimizing both tasks.

  • Results

    Classification is essentially as accurate with compressed representations while requiring 1.5×–2× fewer operations, and segmentation is as accurate at moderate rates and more accurate at aggressive rates.

  • Takeaways & Limitations

    Inference directly from learned compressed representations can reduce computation while preserving or improving classification and segmentation performance, with joint training also improving image quality.

  • Takeaways & Limitations

    The deep encoder and learning process have higher time and memory complexities, and extending the method to other computer-vision tasks remains future work.

Abstract

from arXiv · show

Motivated by recent work on deep neural network (DNN)-based image compression methods showing potential improvements in image quality, savings in storage, and bandwidth reduction, we propose to perform image understanding tasks such as classification and segmentation directly on the compressed representations produced by these compression methods. Since the encoders and decoders in DNN-based compression methods are neural networks with feature-maps as internal representations of the images, we directly integrate these with architectures for image understanding. This bypasses decoding of the compressed representation into RGB space and reduces computational cost. Our study shows that accuracies comparable to networks that operate on compressed RGB images can be achieved while reducing the computational complexity up to $2\times$. Furthermore, we show that synergies are obtained by jointly training compression networks with classification networks on the compressed representations, improving image quality, classification accuracy, and segmentation performance. We find that inference from compressed representations is particularly advantageous compared to inference from compressed RGB images for aggressive compression rates.

1 INTRODUCTION

The paper performs classification and semantic segmentation directly on learned compressed representations, bypassing RGB reconstruction. This achieves comparable or improved accuracy with lower computation, and joint training improves both compression and understanding performance.

  • Motivation: Learned compression produces feature-map representations that can support inference without reconstructing RGB images.The approach integrates compression encoders with inference networks for classification and segmentation.
  • Motivation: The approach targets transmitted-image settings by saving RGB reconstruction and part of feature extraction before cloud processing.A cloud photo-storage application is given as a representative use case for immediate indexing and search.
  • Contributions: The method adapts an image compression autoencoder, ResNet, and DeepLab for classification and semantic segmentation from compressed representations.Only minor changes to the original networks and corresponding training procedures are required.
  • Results: Classification from compressed representations is essentially as accurate as from decompressed images while requiring 1.5×–2× fewer operations.The comparison uses the original classifier applied after image reconstruction.
  • Results: Semantic segmentation is as accurate at moderate compression rates and more accurate at aggressive rates when using compressed representations.Segmentation from compressed representations also requires significantly fewer operations than segmentation from decompressed images.
  • Joint training: Joint compression-classification training increases SSIM and MS-SSIM while improving classification and segmentation accuracy.The networks are trained together so compressed representations support both reconstruction and inference objectives.

2 RELATED WORK

Prior work studied inference from features produced by engineered image and video codecs, as well as feature learning with autoencoders. Learned compressed representations extend this direction to neural compression features.

  • Compressed images: Engineered image codecs have been used to extract features for hyperspectral-image classification and neural-network classification after DCT compression.The DCT-based approach reportedly provides 2 to 10× faster training with minor classification-accuracy loss.
  • Compressed video: Compressed video has supported direct inference for video classification and action recognition because its temporal structure lends itself to feature extraction.The cited examples use representations produced by engineered video codecs.
  • Feature learning: Autoencoders and their denoising or sparsity-penalized variants have been used to learn image features for classification and regression.The paper relates its approach to unsupervised feature learning through autoencoders.
  • Caveat: Compression artifacts from learned and engineered codecs can compromise inference performance.Prior work specifically studied the effect of JPEG artifacts on neural-network image classification.

3 LEARNED DEEPLY COMPRESSED REPRESENTATION

The learned compression system uses a convolutional autoencoder whose encoder produces a spatially reduced feature-map representation. Training balances reconstruction distortion against bitrate, yielding multiple operating points for evaluation.

  • Architecture: The convolutional autoencoder encodes 224 × 224 RGB images into representations with encoder complexity 3.56 · 10^9 FLOPs and decoder complexity 2.85 · 10^9 FLOPs.The representation has dimensions w/8 × h/8 × C, with C controlling the rate.
  • Rate-distortion training: Quantization trades increased reconstruction distortion for lower entropy and shorter bitstreams.The compression objective is the classical rate-distortion trade-off D + βR.
  • Rate-distortion training: The loss controls the MSE–entropy trade-off through β and a target entropy Ht, producing operating points measured in bits per pixel.MSE measures distortion, while H(q) estimates the entropy-based rate.
  • Operating points: Three operating points are evaluated at 0.0983 bpp, 0.330 bpp, and 0.635 bpp.On ILSVRC2012, these points outperform JPEG and JPEG2000 on SSIM and MS-SSIM.
  • Representation: The learned compressed representation is visualized as a subset of feature-map channels alongside the corresponding RGB image.The visualization illustrates the intermediate representation used by the method.

4 IMAGE CLASSIFICATION FROM COMPRESSED REPRESENTATIONS

The paper adapts ResNet to operate directly on 28 × 28 compressed representations, avoiding the decoder while preserving classification accuracy close to decoded-RGB baselines. At 0.635 bpp, compressed-representation inference achieves 87.85% top-5 accuracy versus 89.96% from original RGB images, while reducing ImageNet storage 5.8×.

  • Architecture: The encoder outputs 28 × 28 × C compressed representations, and cResNet variants are formed by removing RGB ResNet layers operating at larger spatial dimensions.cResNet-39 removes the first 11 layers, while cResNet-51 matches ResNet-50's computational cost by adding 14 × 14 residual blocks.
  • Architecture: The classification pipeline keeps the compression network fixed and feeds its encoder output directly to cResNets, whereas decoded-RGB baselines use the encoder-decoder output.The comparison uses the same compression operating points and evaluates validation accuracy for both input types.
  • Classification results: 0.5% is the top-5 accuracy gap between ResNet-50 and cResNet-51 at 0.635 bpp, shrinking to 0.3% at 0.0983 bpp.These architectures have the same computational complexity, and the compressed-representation model becomes relatively more competitive at the lower rate.
  • Classification results: 87.85% top-5 accuracy is achieved from the compressed representation at 0.635 bpp, compared with 89.96% from original uncompressed RGB images.At 0.635 bpp, ImageNet storage falls from 144 GB to 24.8 GB, a 5.8× reduction.
  • Computational cost: 2.2·10^9 FLOPs is the decoding-inclusive computational advantage of compressed-representation inference at fixed classification performance.At fixed computational cost, reconstructed compressed RGB images perform about 0.25% better, while at fixed classification cost compressed inference costs about 0.6 · 10^9 FLOPs more.

5 SEMANTIC SEGMENTATION FROM COMPRESSED REPRESENTATIONS

Semantic segmentation is adapted from the ResNet-based DeepLab architecture to use compressed representations directly. Performance is comparable to decoded-RGB inference at moderate rates and becomes better for compressed representations at the most aggressive rate, with lower computational cost.

  • Method: The segmentation models adapt ResNet-based DeepLab with atrous convolutions to enlarge receptive fields without aggressively subsampling feature maps.The cResNet-d and ResNet-d networks are fine-tuned after ImageNet pretraining, with the compression encoder and decoder fixed.
  • Evaluation: The evaluation uses PASCAL VOC-2012, containing 20 foreground classes and one background class, with 10,582 augmented training images and 1,449 validation images.Each pixel is annotated with one of the 21 classes.
  • Segmentation results: At 0.635 bpp, ResNet-50-d and cResNet-51-d perform equally well, while compressed representations lead by 0.37% at 0.330 bpp and 1.65% at 0.0983 bpp.The comparison measures segmentation performance for compressed representations against corresponding reconstructed compressed RGB images.
  • Visual comparison: At 0.0983 bpp, the reconstructed-RGB segmentation misses the back part of a train that the compressed-representation model captures.The visual comparison includes the original image, ground-truth mask, reconstructed RGB images, and predictions from both architectures across operating points.
  • Computational cost: 0.7% better mIoU is obtained from compressed representations at fixed computational cost for the 0.0983 bpp operating point.At fixed mIoU, compressed representations reduce computational cost by about 3.3 · 10^9 FLOPs, or 6.1 · 10^9 FLOPs when decoding is included.

6 JOINT TRAINING FOR COMPRESSION AND IMAGE CLASSIFICATION

Joint training combines the compression network and classification network, optimizing reconstruction and classification together. Finetuning improves downstream classification and segmentation while largely preserving or slightly improving compression quality.

  • Formulation: Joint training combines compression and cResNet-51, feeding the compressed representation to both decoder and classifier.The networks are initialized from trained states and then finetuned jointly.
  • Formulation: The combined objective weights compression loss against classification cross-entropy through γ.Compression loss includes reconstruction error and an entropy-related rate term.
  • Formulation: Finetuning controls for operating-point and training-duration effects by separately finetuning compression, retraining cResNet-51, and comparing matched training schedules.The controlled comparison uses a fixed new operating point and nine cResNet-51 training epochs.
  • Joint training results: Joint training slightly increases SSIM and MS-SSIM while slightly decreasing PSNR, indicating little overall change in compression performance.All three metrics are reported with higher values treated as better.
  • Joint training results: Joint finetuning produces larger classification gains and significant segmentation gains than finetuning the compression network alone.At 0.635 bpp, classification is similar between the two strategies, but segmentation differs considerably.
  • Joint training results: At 0.0983 bpp, joint training improves classification by 2% and mIoU by 1.7% over compression-only training.Matching these gains by enlarging cResNet-51 would require 75% and 40% more computational complexity, respectively.

7 DISCUSSION

The paper finds that inference directly from learned compressed representations can match decoded-image performance while reducing runtime and memory demands. Joint training improves both compression quality and image-understanding accuracy, although the approach has higher encoder and training complexity than standard compression.

  • Performance: Direct inference from learned compressed representations supports classification and semantic segmentation with comparable performance to decoded RGB-image pipelines.The approach was validated on both tasks with only minor changes to the original networks and training procedures.
  • Memory: Removing reconstruction offers potential benefits for real-time, memory-constrained applications, including systems using specialized hardware.The approach combines avoided image reconstruction with shallower models and aggressive compression rates.
  • Synergy: Jointly training compression and inference networks improves compression quality together with classification and segmentation accuracy.The reported gains include increases in SSIM and MS-SSIM.
  • Limitations: The deep encoder and learning process have higher time and memory complexity than current standard compression methods such as JPEG and JPEG2000.The discussion notes that dedicated deep-learning hardware may reduce this disadvantage over time.
  • Performance: The approach is particularly suited to aggressive compression rates and settings where memory or storage constraints are critical.These rates are also where deep compression considerably outperforms standard methods.
  • Future work: The study covers classification and semantic segmentation, while extension to other computer-vision tasks remains future work.The paper specifically identifies broader task coverage as an open direction.

A.1 COMPRESSION ARCHITECTURE AND TRAINING PROCEDURE

The compression model is a convolutional autoencoder that maps images to quantized compressed representations and uses a mirrored decoder for reconstruction. Training varies the rate-related configuration across three operating points and uses a differentiable quantization surrogate.

  • Encoder: The encoder produces a w/8 × h/8 × C feature-map representation that is quantized into discrete symbols forming the compressed representation z.The channel count C is a rate-related hyperparameter.
  • Decoder: The decoder reconstructs the image from the compressed representation using upsampling and deconvolutions that mirror the encoder.The reconstructed output is an approximation to the input image.
  • Quantization: The quantization operator behaves like hard quantization in the forward pass and a differentiable approximation in the backward pass.This allows the quantization step to be trained with gradient-based optimization.
  • Training: Three compression operating points are obtained at 0.0983 bpp, 0.330 bpp, and 0.635 bpp by changing β, Ht, and C.The corresponding channel counts are C = 8, 16, and 32, respectively.

A.2 IMAGE COMPRESSION METRICS, PERFORMANCE AND VISUALIZATION

The paper evaluates compression quality with PSNR, SSIM, and MS-SSIM, comparing learned compression operating points with JPEG and JPEG 2000 while visualizing decoded images and compressed-representation channels.

  • PSNR, SSIM, and MS-SSIM are used to measure image-compression performance, with higher values indicating better quality.PSNR depends monotonically on mean squared error, while SSIM and MS-SSIM better reflect perceived image similarity.
  • The compressed representation is visualized through its four highest-entropy channels alongside reconstructed images and the original RGB image.These channels are shown for each operating point to illustrate the information retained before reconstruction.
  • Figure 8 compares MS-SSIM, SSIM, and PSNR against rate in bpp for JPEG 2000, JPEG, and learned compression operating points.The plot reports all three metrics as functions of bitrate, with higher values preferred.
  • The compressed-representation channels resemble quantized, downscaled versions of the original image, motivating inference directly from them instead of decoded RGB images.The visualization links the structure of the learned representation to its potential use for image understanding.

A.3 ARCHITECTURE TABLE

The architecture table details the ResNet and cResNet layer structures, including convolution dimensions, spatial input sizes, building blocks, and downsampling locations.

  • Table 3 reports convolution dimensions and layer-by-layer structures for the ResNet and cResNet architectures.The table provides a more detailed view of the networks used in the experiments.
  • ResNet uses 224 × 224 RGB inputs, whereas cResNet uses 28×28 compressed representations derived from the same image size.The reported architecture dimensions distinguish the RGB and compressed-representation processing paths.
  • Downsampling in the listed architectures is performed by conv3_1, conv4_1, and conv5_1 with stride 2.Building blocks are shown in brackets together with the numbers of stacked blocks.

A.4 TRAINING CLASSIFICATION

The experiments train and compare classification and segmentation systems using RGB images, reconstructed images, or compressed representations, including joint compression-task finetuning and runtime evaluation.

  • Training setup: Classification experiments use ResNet implementations trained from scratch with batch size 64, learning rate 0.025, SGD momentum 0.9, and ImageNet preprocessing.RGB inputs use 224×224 crops, while compressed representations use 28 × 28 crops.
  • Training setup: Segmentation training uses batch size 10, 20k SGD iterations, momentum 0.9, and a learning-rate schedule adapted from Chen et al. (2016).The segmentation architecture uses modified preprocessing and a separate final classification-layer learning rate.
  • Segmentation comparison: At 0.0983 bpp, segmentation from compressed representations captures the back of the train that segmentation from reconstructed RGB images misses.Across the other shown operating points, the segmentation performance is visually similar between the two inputs.
  • Finetuning: Finetuning improves compression metrics to higher values at lower bpp, with joint finetuning improving SSIM and MS-SSIM more than compression-only finetuning but improving PSNR less.This pattern is reported at both the 0.0983 and 0.635 bpp operating points.
  • Runtime evaluation: Runtime experiments report per-image inference costs for compressed representations and reconstructed RGB images, including decoding time for the RGB path.Benchmarks were run on a GeForce Titan X GPU using TensorFlow v1.3.
Loading 1803.06131v1…