Source-linked AI summary
Scalable Image Coding for Humans and Machines
Hyomin Choi, Ivan V. Bajic
TL;DR
Captured visual content increasingly serves machine analytics while requiring occasional human viewing, motivating a codec that supports both uses efficiently. The paper learns layered latent representations that assign simpler tasks to base information and more complex tasks to enhancement layers. The resulting 2- and 3-layer systems achieve 37%-80% bitrate savings on machine vision tasks while remaining comparable to state-of-the-art codecs for input reconstruction.
Problem
Visual content must support automated machine vision and occasional human viewing, but existing multi-task approaches may require reconstructing the entire latent representation for each task.
Method
The paper develops an end-to-end learned codec whose latent representation is partitioned into separately decoded base and enhancement layers for nested machine and reconstruction tasks.
Results
37%-80% bitrate savings are reported on machine vision tasks versus relevant benchmarks, with input reconstruction comparable to state-of-the-art image codecs.
Takeaways & Limitations
Latent-space scalability supports object detection, segmentation, and human-viewable input reconstruction through task-specific subsets of one coded representation.
Abstract
from arXiv · showhide
At present, and increasingly so in the future, much of the captured visual content will not be seen by humans. Instead, it will be used for automated machine vision analytics and may require occasional human viewing. Examples of such applications include traffic monitoring, visual surveillance, autonomous navigation, and industrial machine vision. To address such requirements, we develop an end-to-end learned image codec whose latent space is designed to support scalability from simpler to more complicated tasks. The simplest task is assigned to a subset of the latent space (the base layer), while more complicated tasks make use of additional subsets of the latent space, i.e., both the base and enhancement layer(s). For the experiments, we establish a 2-layer and a 3-layer model, each of which offers input reconstruction for human vision, plus machine vision task(s), and compare them with relevant benchmarks. The experiments show that our scalable codecs offer 37%-80% bitrate savings on machine vision tasks compared to best alternatives, while being comparable to state-of-the-art image codecs in terms of input reconstruction.
I. INTRODUCTION
The paper develops scalable learned image coding for applications combining human and machine vision, addressing the inefficiency of reconstructing an entire latent space for every task. Its layered designs separate task-relevant information across bitstreams and support object detection, segmentation, and input reconstruction.
- The approach targets visual content used for automated analytics with occasional human viewing, alongside prior compress-then-analyze and analyze-then-compress paradigms.
- The paper extends a 2-task model to separate base and enhancement bitstreams so the enhancement layer need not be decoded for the base task.
- A 3-layer Feature Pyramid Network model supports object detection, object segmentation, and input reconstruction.
- The proposed codec splits one latent space into base and enhancement layers, decoding only the required portion for inference while using all layers for input reconstruction.
III. PROPOSED METHODS
The method uses a compressible latent representation and task-specific latent transforms to bypass image reconstruction when machine inference is sufficient. Data-processing analysis assigns simpler tasks to nested latent subsets, yielding two- and three-layer systems for detection, segmentation, and reconstruction.
- A. The backbone: The codec maps an input image through analysis, quantization, and entropy coding, while synthesis reconstructs the image from the decoded latent representation.The analysis and synthesis transforms are parameterized by φ and θ, and entropy coding operations are omitted from the displayed formulation.
- A. The backbone: The data-processing analysis motivates using only part of the latent representation for inference while retaining the full representation for reconstruction.
- A. The backbone: The 2-layer system supports object detection from the base layer and input reconstruction from the full latent space.
- A. The backbone: The 3-layer design assigns object detection to the base layer, segmentation to the base plus first enhancement layer, and reconstruction to the full latent space.This ordering follows the stated information relationship between detection and segmentation.
- A. The backbone: Machine vision outputs can be derived directly from the corresponding latent portion through a latent-space transform, without reconstructing the input image.
APPENDIX A DEEP FEATURE COMPRESSIBILITY
For non-generative feedforward DNNs, intermediate latent representations are theoretically no less compressible than the input, for both lossless and lossy compression. Equality requires exact input reconstruction from the latent outputs, while lossy compression may still be more efficient even when the representation is invertible.
- The joint entropy of any selected collection of DNN layer outputs is no larger than the input entropy when the layers are jointly compressed.This result extends single-layer information-bottleneck arguments to multiple layers, which is useful for networks with skip or parallel connections.
- Exact equality in the lossless bound requires reconstructing the input deterministically from all selected layer outputs.Equivalently, the conditional entropy H(X|Y) must be zero.
- Intermediate representations from non-generative feedforward DNNs are at least as compressible as the input for lossless and lossy compression.The lossless result bounds joint latent entropy by input entropy, while the lossy theorem bounds the latent rate-distortion function by the input rate-distortion function.
- For lossy compression, distortion is measured at the model output rather than directly between the original and quantized input or latent representation.The formulation uses d(f(x), f(x̂)) for input quantization and the corresponding mapping through h for intermediate-layer quantization.
- Exact input recoverability is necessary but not sufficient for equality between latent and input rate-distortion functions.Even invertible latent mappings may permit more efficient quantization, so latent compression can outperform input compression at the same output distortion.