Source-linked AI summary
Image coding for machines: an end-to-end learned approach
Nam Le, Honglei Zhang, Francesco Cricri, Ramin Ghaznavi-Youvalari, Esa Rahtu
TL;DR
Human-oriented codecs optimize visual quality, while machine consumers may need a different rate-distortion trade-off. The paper proposes an end-to-end learned codec with adaptive loss weighting; against VVC, it reports bitrate savings at equal task performance for object detection and instance segmentation.
Problem
Human-oriented codecs measure distortion by visual-quality metrics, although computer-vision tasks may be the only consumers and may not need all information in compressed images.
Source: §1 · p. 1Method
The paper proposes an end-to-end learned codec that uses pretrained task-network loss as distortion and dynamically weights competing losses during training.
Source: §1 · p. 1, §2.3 · p. 3Results
Against VVC’s Pareto front, the codec saves 37.87% bitrate for object detection and 32.90% for instance segmentation at equal task performance.
Source: §3.2 · p. 4Takeaways & Limitations
The flexible loss-weighting strategy trains toward desired objectives while achieving a trade-off between them during training.
Source: §4 · p. 4The system is trained on specific task-network architectures; preserving critical input information for other tasks or architectures remains future work.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Over recent years, deep learning-based computer vision systems have been applied to images at an ever-increasing pace, oftentimes representing the only type of consumption for those images. Given the dramatic explosion in the number of images generated per day, a question arises: how much better would an image codec targeting machine-consumption perform against state-of-the-art codecs targeting human-consumption? In this paper, we propose an image codec for machines which is neural network (NN) based and end-to-end learned. In particular, we propose a set of training strategies that address the delicate problem of balancing competing loss functions, such as computer vision task losses, image distortion losses, and rate loss. Our experimental results show that our NN-based codec outperforms the state-of-the-art Versa-tile Video Coding (VVC) standard on the object detection and instance segmentation tasks, achieving -37.87% and -32.90% of BD-rate gain, respectively, while being fast thanks to its compact size. To the best of our knowledge, this is the first end-to-end learned machine-targeted image codec.
1. INTRODUCTION AND BACKGROUND
Human-oriented codecs optimize visual quality, although computer-vision tasks may be the only consumers and may not need all image information. The paper proposes an end-to-end learned codec that targets task performance and reports that it outperforms VVC on object detection and instance segmentation.
1. INTRODUCTION AND BACKGROUND
- Designing a codec specifically for machine consumers may improve coding efficiency because task networks may not need all information in human-targeted compressed images.The paper uses “machines” to refer to computer-vision tasks such as object detection or image classification.Source: §1 · p. 1
- Earlier machine-oriented methods preserve standard bitstreams and fine-tune parts of traditional codecs, adding capability rather than replacing the human-oriented pipeline and.The paper positions its approach as an extension of learned image coding with machines as the main end-users.Source: §1 · p. 1
- Prior loss-balancing methods discuss dynamic weighting, but the paper says they do not provide the full control over objective priorities that it seeks.The paper’s contribution addresses balancing competing losses in multi-task learning.Source: §1 · p. 1
- The proposed end-to-end learned system optimizes rate-distortion trade-offs using the pretrained task network’s training loss as distortion and reports outperforming VVC on two vision tasks.The experiments cover object detection and instance segmentation.Source: §1 · p. 1
2. PROPOSED METHOD
The proposed codec uses a task-focused neural auto-encoder and probability model, trained end to end to balance rate, image distortion, and task losses. Its dynamic, staged loss weighting addresses competition among those objectives.
2. PROPOSED METHOD
- The codec encodes images into quantized latents, entropy-codes them using a learned probability model, then decodes images for a pretrained task network.The task network returns the corresponding task results from the decoded image.Source: §2 · p. 2
2.1. Auto-encoder
- A residual CNN auto-encoder decodes representations for task performance rather than image reconstruction, using few filters in intermediate and final layers to limit time and resource use.The decoder output retains image shape because the pretrained task networks expect three-channel images.Source: §2.1 · p. 2, Figure 2 · p. 2
2.2. Probability model
- The probability model learns a latent distribution for rate estimation; the rate loss totals the coded latent and hyper-latent bitstreams.The mean-and-scale hyperprior models latents with Gaussian mixtures, and the hyper-latent stream carries embeddings needed for decompression.Source: §2.2 · p. 2
2.3. Training strategy
- Training combines rate, MSE, and task losses with dynamic weights, first training an MSE-only base model and then raising rate and task weights while keeping the MSE weight at 1.The five phases introduce task loss, introduce rate loss, enhance task performance, and search trade-offs once stable; checkpoints can target different bitrates.Source: §2.3 · p. 3, Figure 3 · p. 3
3. EXPERIMENTS AND DISCUSSION
The experiments evaluate task-specific codecs on Cityscapes against VVC, then show bitrate savings and task-focused reconstruction behavior. The discussion also notes that broader suitability across tasks and architectures remains future work.
3.1. Experimental setup
- The evaluation uses 500 Cityscapes validation images, common COCO/Cityscapes classes, BPP and mAP@[0.5:0.05:0.95], with VVC as the baseline.VVC versions span 28 combinations of seven QPs and four resolutions; downsampled images are upscaled before task evaluation.Source: §3.1 · pp. 3–4
- The experiments define loss weights as epoch-dependent, using 0 before p2, 2fw(e−p2,1.01) for p2≤e<p3, c for p3≤e<p4, and c+2fw(e−p4,1.02) for e≥p4.The milestones are p1=50, p2=75, p3=120, and p4=165; fw(x,a)=10−3(ax−1) and c=2fw(p3−p2−1,1.01).Source: §3.1 · p. 4
3.2. Experimental results
- The codec has 1.5M parameters and encodes a 2048 × 1024 image in around 0.15 seconds on one RTX 2080Ti GPU.The paper gives around 125 seconds for VTM-8.2 encoding on an Intel Xeon Gold 6154 CPU cluster, a different hardware setup.Source: §3.2 · p. 4
- Low-bitrate reconstructions compress regions less important to the task network more aggressively while preserving object shapes and edges.The authors note that suitability for different tasks or NN architectures remains future work.Source: §3.2 · p. 4
- 37.87% bitrate savings for object detection and 32.90% for instance segmentation at the same task performance, versus VVC’s Pareto front.The proposed systems also achieve higher mAP than the best-quality VVC setting while using around 60% of its bitrate.Source: Table 1 · p. 4, §3.2 · p. 4
4. CONCLUSIONS
The paper concludes that its end-to-end learned codec outperforms VVC for machine-oriented compression and uses flexible loss weighting to train toward competing objectives.
4. CONCLUSIONS
- The proposed NN-based codec outperforms VVC by a large margin when compression targets machines.The conclusion also highlights a flexible loss-weighting strategy for training toward the desired objectives and their trade-off.Source: §4 · p. 4
- At low bitrates, decoded instance-segmentation outputs show strong background suppression.The figure caption describes decoded outputs at different bitrates.Source: Figure 5 · p. 4