PaperlayerView exact revision on arXiv

Source-linked AI summary

Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform

Myungseo Song, Jinyoung Choi, Bohyung Han

arXiv:2108.09551v1eess.IVcs.CVcs.LG

TL;DR

Learned compression often uses uniform spatial rates, and many models are optimized for only one target rate. This paper conditions a variable-rate compressor on pixel-wise quality maps and reports strong performance against multiple independently trained fixed-rate models.

  • Problem

    Most learned compression methods use a uniform compression ratio across image pixels, and models are typically optimized for a single target rate.

    Source: §1 · pp. 1–2

  • Method

    The model uses a real-valued pixel-wise quality map to guide spatially adaptive compression and estimates task-aware maps at test time without retraining.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The algorithm achieves outstanding performance compared with multiple models trained independently at fixed rates.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The framework supports task-aware image compression across computer-vision tasks without training separate models for individual tasks.

    Source: §5 · p. 8

Abstract

from arXiv · show

We propose a versatile deep image compression network based on Spatial Feature Transform (SFT arXiv:1804.02815), which takes a source image and a corresponding quality map as inputs and produce a compressed image with variable rates. Our model covers a wide range of compression rates using a single model, which is controlled by arbitrary pixel-wise quality maps. In addition, the proposed framework allows us to perform task-aware image compressions for various tasks, e.g., classification, by efficiently estimating optimized quality maps specific to target tasks for our encoding network. This is even possible with a pretrained network without learning separate models for individual tasks. Our algorithm achieves outstanding rate-distortion trade-off compared to the approaches based on multiple models that are optimized separately for several different target rates. At the same level of compression, the proposed approach successfully improves performance on image classification and text region quality preservation via task-aware quality map estimation without additional model training. The code is available at the project website: https://github.com/micmic123/QmapCompression

1. Introduction

The paper addresses the limits of uniform spatial compression and models optimized for a single rate. It proposes conditioning compression on pixel-wise quality maps, including maps estimated for target tasks without retraining.

1. Introduction

  • The approach targets two limitations: most learned methods use uniform compression across pixels, and existing models are typically optimized for one rate.The introduction notes that extending single-rate models to multiple rates is not straightforward.Source: §1 · pp. 1–2
  • A single compression model uses a real-valued, pixel-wise quality map to support arbitrary rates and spatially varying reconstructed-image quality.The map defines pixel-wise weights for the distortion objective and guides bit allocation across the image.Source: §1 · pp. 2–3
  • Task-specific quality maps can be estimated at test time without retraining, enabling compression optimized for a target task.The paper describes backpropagation-based estimation and also allows maps from pretrained task models.Source: Figure 1 · p. 1, §1 · pp. 2–3
  • The proposed architecture uses spatially-adaptive feature transforms and is reported to outperform fixed-rate models over a practical range of bitrates.The paper also reports efficient acquisition and effectiveness of the estimated task-specific maps for target tasks.Source: §1 · p. 3

2. Related work

Related work covers deep compression, variable-rate and ROI-guided methods, and spatially-adaptive feature transformations. The paper positions its approach at the intersection of these lines of work.

2. Related work

  • The section reviews deep-network compression, variable-rate methods, ROI-guided compression, and spatially-adaptive affine transforms.These are presented as background areas for the proposed algorithm.Source: §2 · p. 3

2.1. Deep image compression

  • Most deep compression models learn for a fixed Lagrange multiplier and thus provide one rate-distortion operating point per model.Prior work also improves rate-distortion trade-offs with context-based entropy prediction, latent importance maps, or hyper-priors.Source: §2.1 · p. 3

2.2. Variable-rate compression

  • Prior single-model variable-rate methods use recurrent iteration counts or multiple Lagrange multipliers, but discrete levels and quantization choices can limit practicality or output quality.The section also describes continuous rate control through interpolation of parameters in a pretrained discrete-rate model.Source: §2.2 · p. 3

2.3. ROI-based compression

  • Existing ROI-based methods use binary masks to prioritize regions, with the reviewed approaches remaining limited to binary importance masks.The section surveys methods using GANs, ROI-weighted distortion, or supervised ROI prediction.Source: §2.3 · p. 3

2.4. Spatially-adaptive affine transform

  • Spatial feature transform modulates intermediate features without normalization, while this approach also uses the network’s original input when generating spatially varying parameters.The accompanying architecture identifies condition networks as components for compression conditioned on the quality map.Source: §2.4 · p. 3, Figure 3 · p. 4

3. Image Compression Guided by Quality Map

The framework uses a pixel-wise quality map to control compression quality across an image, combining variable-rate and region-of-interest-guided compression. Spatial Feature Transform modules condition the compression network on that map, and task-specific maps can be optimized at encoding time.

3.1. Overview

  • A pixel-wise quality map conditions encoding and defines the desired quality levels after compression, integrating variable-rate and ROI-guided compression.The encoder transforms the source into a latent representation, which is quantized, entropy-coded, and decoded into a reconstruction.Source: §3.1 · p. 4

3.2. Rate-distortion loss

  • Each pixel’s quality-map value sets its distortion weight, enabling explicit spatial control over bit allocation.The model minimizes rate plus weighted distortion; a spatially uniform map reduces to a single scalar rate-distortion weight.Source: §3.2 · p. 4

3.3. Spatially-adaptive feature transform

  • Spatial Feature Transform modules use a condition to generate element-wise scale and shift parameters for intermediate features.The model gives encoder, decoder, and hyper-encoder their own condition networks, with progressively resized condition features transformed using convolutions.Source: §3.3 · pp. 4–5

3.4. Network architecture and pipeline

  • Built on a non-autoregressive Mean & Scale Hyperprior, the architecture adds separate condition networks and SFT modules to the encoder, decoder, and hyper-encoder.The condition networks derive prior features from the image and quality map.Source: §3.4 · p. 5, Figure 5 · p. 5
  • The hyper-encoder produces side information from the latent representation, while the decoder uses a surrogate quality map derived from quantized side information.The surrogate is used because the original quality map may not be available at the decoder.Source: §3.4 · p. 6

3.5. Task-aware image compression

  • At encoding time, a task-specific quality map can be optimized using a pretrained task model, with λ controlling the rate-task trade-off.The map is constrained to values in [0, 1] and optimized by standard backpropagation.Source: §3.5 · p. 6
  • Classification-specific maps can prioritize semantic regions, and optimized maps yield higher rate-task performance than competing alternatives without task-specific training.Uniform quality maps remain an option for decent general reconstruction quality.Source: §3.5 · p. 6

4. Experiments

On Kodak, the variable-rate model outperforms its corresponding single-rate M&S model and is competitive with models using autoregressive context; task-aware maps improve classification and preserve text regions. Experiments also examine qualitative reconstructions and the effect of source conditioning.

4.1. Training and evaluation

  • Training uses COCO crops, while Kodak, a sampled ImageNet set, and ICDAR2015 support rate-distortion, classification, and text-preservation evaluations, respectively.The model is trained for 2M iterations; the quality-map transform is chosen to give approximately 0.1–1.0 bpp on Kodak.Source: §4.1 · pp. 6–7

4.2. Rate-distortion performance

  • The model outperforms the corresponding single-rate M&S model and is as good as or better than context-model approaches, including a variable-rate method, on Kodak rate-distortion performance.The single-rate approaches require multiple independently trained models to cover a wide rate range; the proposed model does not use a context model.Source: §4.2 · p. 7

4.3. Task-aware compression

  • Task-aware quality-map optimization improves classification accuracy at test time in under one second, while Grad-CAM also outperforms uniform quality maps.The evaluation uses pretrained VGG16 for optimization and pretrained ResNet18 for classification accuracy.Source: §4.3 · p. 7
  • Manually supplied text-region ROI masks preserve text regions even at harsh compression rates.Figure 7 contrasts ROI-mask quality maps in the left column with uniform maps in the right; annotations give bpp, PSNR, and MS-SSIM.Source: §4.3 · p. 7, Figure 7 · p. 7

4.4. Qualitative analysis

  • On a Kodak image, the uniform-map model reconstructs complex textures better than M&S Hyperprior and BPG, with the best PSNR and MS-SSIM at lower or equal bitrate.Figure 8 compares cropped reconstructions; Figure 9 illustrates reconstructions and bit allocation for uniform and non-uniform quality maps.Source: §4.4 · p. 7, Figure 8 · p. 8, Figure 9 · p. 8

4.5. Ablation Study

  • Ablations show that SFT layers and module refinement improve variable-rate compression quality, while source conditioning enhances adaptivity and performance.The comparison with the model without source conditioning attributes the difference to image-dependent condition features under the same quality map.Source: §4.5 · p. 8

5. Conclusion

The paper concludes with a single-model compression framework controlled by a real-valued pixel-wise quality map, and automatic task-specific map estimation at test time without additional training. Experiments report strong rate-distortion performance against independently trained fixed-rate models and applicability to computer-vision tasks.

5. Conclusion

  • A real-valued pixel-wise quality map controls spatially adaptive compression rates using a single model.The framework uses a network based on spatially-adaptive feature transform.Source: §5 · p. 8
  • Task-specific quality maps can be estimated automatically at test time without additional training, supporting task-aware image compression.Source: §5 · p. 8
  • Experiments report strong rate-distortion performance against multiple independently trained fixed-rate models and applicability to various computer-vision tasks.Source: §5 · p. 8

A. Additional qualitative results

Additional examples show text-preserving and non-uniform-map compression, including non-uniform examples from COCO validation images excluded from training. Additional Kodak comparisons report performance degradation for a naive spatially adaptive M&S Hyperprior variant relative to the trained M&S model.

A. Additional qualitative results

  • Non-uniform quality-map examples use COCO validation images excluded from training, while uniform-map Kodak comparisons report better visual quality and PSNR/MS-SSIM than M&S Hyperprior, BPG, and JPEG at similar bitrates.Additional text-preserving examples use ROI masks.Source: A. Additional qualitative results · p. 11

B. Additional RD performance comparisons

C. Model complexity

The model’s complexity is compared with baseline models in parameters and encoding/decoding time. The authors report that source conditioning adds marginal coding complexity while improving performance, and that one M&S model’s parameter count understates the six models needed for broad rate coverage.

C. Model complexity

  • The model outperforms M&S WIDE despite a similar parameter count, while source conditioning adds only marginal coding complexity compared with W/O SC.The comparison used a Titan Xp GPU and the same entropy coder for all models.Source: C. Model complexity · p. 11
  • M&S requires six independent models to cover a wide rate range, so its actual parameter count is six times the per-model count reported in Table 1.Source: C. Model complexity · p. 11
  • M&S + Context and Choi et al. [12] require more coding time than the proposed model because their autoregressive context models use serial coding.Masked convolution reduced M&S + Context encoding time from 5004 ms to 82 ms, but decoding remained slow and non-parallel.Source: C. Model complexity · p. 11

D. Experiment details

The experiments specify how rate-distortion curves and classification-aware quality maps were evaluated. They also describe the classification test set, optimization procedure, and the human-evaluation table’s scope.

D.1. Rate-distortion comparison

  • Rate-distortion curves for the proposed model average metrics over test images at 21 uniform-map quality values, while other curves use published or author-provided results.The proposed-map values span q = 0 to 1 in steps of 0.05; smaller spacing produced almost identical curves.Source: §D.1 · p. 12

D.2. Classification-aware compression

  • Classification-aware optimization starts from random quality maps on 510 sampled ImageNet images, then minimizes the test-set loss using L-BFGS.The set contains 102 categories with five images sampled per category.Source: §D.2 · p. 12
  • Experiments used λ values from 0.0001 to 1000 and results at 3 and 5 iterations for each plot; accuracies converged to the known upper bound, the accuracies on original images.Source: §D.2 · p. 12
  • Table 2 reports average response rates from 33 participants across 16 cases using semantic ROI masks; parentheses give ROI/non-ROI quality values.Source: Table 2 · p. 12
  • Grad-CAM quality maps are formed by scaling the CAM output, and Figure 11 visualizes the maps and reconstructions through optimization iterations.The scale α ranges from 0 to 1 in steps of 0.1.Source: §D.2 · p. 12

E. Human evaluation

The human evaluation compares reconstructions at nearly equal bitrates and finds that allocating higher quality to semantic regions is generally preferred. It also indicates that very poor non-semantic quality can still harm perception.

E. Human evaluation

  • The evaluation used 16 cases from MSRA10K, with 33 participants choosing the best and worst of three reconstructions per case.The reconstructions used different quality maps and had almost the same bitrates.Source: E. Human evaluation · p. 12
  • Higher quality values in semantic regions receive more best-image preferences at the same bitrate, while ROI-based maps receive fewer worst-image selections than uniform maps.Source: E. Human evaluation · p. 12
  • ROI2 receives more worst-image votes than ROI1 despite its higher semantic-region quality, indicating that very poor non-semantic quality is not negligible for human perception.Source: E. Human evaluation · p. 13

F. Example quality maps for training

Training uses randomly generated spatial quality maps, with one of four map types selected for each mini-batch instance.

F. Example quality maps for training

  • The four training-map types are uniform, semantic with random class values, a gradient between two random values, and a Gaussian-mixture kernel-density map with random parameters.Source: F. Example quality maps for training · p. 13

G. Practical aspects of task-aware compression

Task-aware compression reflects task-specific preferences for spatial image quality. External task models can provide quality maps even when ground-truth labels or task losses are unavailable.

G. Practical aspects of task-aware compression

  • Task-aware compression is intended to reflect preferences for spatial image quality that depend on the target task.The passage gives street-view construction and video conferencing as examples of settings with different regional quality preferences.Source: G. Practical aspects of task-aware… · p. 13
  • External task models, including ROI detection and Grad-CAM, can provide quality maps when ground-truth labels or task losses are unavailable.For Grad-CAM, the authors used each test image’s highest-scoring predicted class rather than its ground-truth label.Source: G. Practical aspects of task-aware… · p. 13
  • For video conferencing, task-aware compression can preserve human-region quality while reducing background quality without conveying a person’s location or personal information.The authors also describe using it to preserve an object region regardless of the target label.Source: G. Practical aspects of task-aware… · p. 13
  • Figures 13–14 show reconstructions guided by different and non-uniform quality maps; text ROI masks are used for the maps in Figure 13’s third and fourth rows.Figure 13 annotations report bpp, PSNR, and MS-SSIM for the corresponding reconstructions.Source: Figure 13 · p. 14, Figure 14 · p. 15
  • On four Kodak images, the model outperforms M&S Hyperprior, BPG, and JPEG in visual quality, PSNR, and MS-SSIM at similar bit rates.The model’s rate is matched to M&S Hyperprior by adjusting the uniform quality map.Source: Figure 15 · p. 16, Figure 16 · p. 17, Figure 17 · p. 18
Loading 2108.09551v1…