Source-linked AI summary
You've Seen Enough: Quality-Constrained Image Coding for Machines
Khoa Pham-Dinh, Sanaz Nami, Hamed Rezazadegan Tavakoli, Moncef Gabbouj, Farhad Pakdaman
TL;DR
Human-oriented image compression can preserve fidelity that does not match machine-task needs, while joint optimization leaves the quality level implicit. This paper trains a codec with an explicit quality target and a penalty, reporting BD-rate savings at equal task performance on Cityscapes.
Problem
Human-fidelity metrics correlate weakly with vision-model accuracy, and joint optimization leaves realized quality implicit rather than guiding it to a desired level.
Source: §1 · p. 1Method
The method constrains joint codec and segmentation training to a user-defined quality target and penalizes distortion beyond that target.
Source: §1 · p. 1, §3.3 · p. 2Results
At equal task performance, the bilinear penalty achieves a BD-rate of −22.82% over joint optimization and −29.81% over the Baseline.
Source: §4.2 · p. 3Takeaways & Limitations
On Cityscapes, the approach yields consistent mIoU-axis BD-rate savings over both baselines while keeping reconstruction close to the requested quality.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Visual data is increasingly consumed by machine-vision systems rather than by human observers. Image Coding for Machines (ICM) compresses images assuming the main observer is a computer vision application and that the human observer needs to inspect or validate the decisions. Inspired by just-noticeable distortion, which sets the quality to the just-acceptable level for human observers, we aim to cap the human-observed quality at a desired level, with the goal of using the remaining coding capacity to improve the machine performance. We recast joint compression-segmentation training as a constrained optimization problem in which the codec must meet a predefined acceptable target visual quality while a task term consumes the remaining coding capacity. We solve this by designing a penalty function to guide the quality to the desired target. We propose two penalty functions, an absolute function and a bilinear function, the latter applying a steeper slope once the target visual quality is exceeded. Experimental results show that, under the quality constraint, the proposed method achieves a BD-rate of $-22.82\%$ over an unconstrained joint rate--distortion--task optimization and $-29.81\%$ over a simple rate--distortion baseline, showcasing bitrate reduction with the same task performance. This is achieved while the codec also meets the target visual quality with a reasonable error and without adding any complexity overhead.
1. INTRODUCTION
Machine-vision systems increasingly consume images directly, but codecs optimized for human fidelity may spend bits on details that do not support vision tasks. The paper proposes explicitly targeting acceptable visual quality and evaluating the resulting compression and segmentation trade-off.
1. INTRODUCTION
- Human-oriented fidelity metrics such as PSNR and MS-SSIM correlate weakly with vision-model accuracy, while task-critical structure can be lost during reconstruction.This motivates codecs that preserve semantic information rather than pixel fidelity.Source: §1 · p. 1
- Image coding for machines must support both machine tasks and human inspection, unlike feature coding, which targets machine-only consumption.The paper situates its setting in pixel-domain ICM, where a decoded image remains interpretable to a human.Source: §1 · p. 1
- Joint rate–distortion–task training leaves realized quality implicit, prompting the question of how to control output quality while reallocating capacity to the task.The motivation draws on just-noticeable-difference studies, which suggest that quality beyond a human observer's good-enough level is unnecessary.Source: §1 · p. 1
- The proposed constrained formulation targets user-defined quality and uses a penalty to discourage further quality improvement beyond the target.It minimizes rate and task losses subject to reaching the specified quality.Source: §1 · p. 1
- The paper evaluates the formulation on Cityscapes semantic segmentation, including coding-gain and task-performance analyses.The stated contribution is improved compression ratio for segmentation while meeting target quality.Source: §1 · pp. 1–2
2. RELATED WORK
Related work covers learned compression approaches to controlling rate or distortion, machine-oriented coding paradigms, and spatial bit allocation. The paper's approach differs by constraining distortion to an explicit target in ICM.
2. RELATED WORK
- Rate–distortion optimization is prevalent in learned image compression, but it is ill-suited to targeting specific operating points precisely.A given image can yield a slightly different bitrate or quality, making rate matching laborious and imprecise.Source: §2 · p. 2
- A distortion-constrained approach enables targeting specific distortion values and pointwise bitrate comparisons, with compression performance on par with classical rate–distortion.The passage contrasts this with bitrate regulation through quantization and the prevailing λ-based formulation.Source: §2 · p. 2
- Other learned compression work penalizes deviations from a target bitrate, while another approach predicts bitrate across predefined λ values to select a target rate.The latter fits a modified exponential R-λ model and encodes in a single pass.Source: §2 · p. 2
- MVRNet allocates more bits to task-critical regions and fewer to redundant areas using a spatially varying quantization map.The passage reports that this preserves machine-task accuracy at lower bitrates.Source: §2 · p. 2
3. METHODOLOGY
The method trains an image codec with a frozen segmentation network, then replaces implicit distortion weighting with an explicit quality target and penalty. Two penalties differ in how they respond once that target is reached.
3.1. Framework
- The codec's analysis and synthesis transforms encode an image to a quantized latent and reconstruct it for a frozen segmentation network, whose task-loss gradients update the codec.Training uses rate, MSE distortion, and pixel-wise cross-entropy task losses.Source: §3.1 · p. 2, Figure 1 · p. 2
3.2. Baselines
- Joint Optimization is the unconstrained joint model that adds task loss during training; the framework also compares against a pure rate–distortion baseline.The pure rate–distortion codec is trained for human vision and feeds its reconstruction to the task network only at inference.Source: §3.2 · p. 2
3.3. Quality-Constrained Formulation
- The user specifies a target PSNR, which is converted to a distortion target because PSNR is a monotonic transform of MSE.The formulation minimizes rate and task losses subject to meeting the target quality.Source: §3.3 · p. 2
- The penalty replaces plain distortion: before the target is reached it behaves like rate–distortion, then discourages spending bits on further visual quality.The two-phase behavior applies when D ≥ Dt and when D < Dt, respectively.Source: §3.3 · p. 2
3.4. Penalty Functions
- The absolute penalty is symmetric about the target, rewarding sub-target distortion and penalizing quality beyond the target at the same rate.The bilinear penalty instead applies a steeper slope after the target, addressing overshoot at low bitrates where the absolute penalty can underperform.Source: §3.4 · pp. 2–3, Figure 2 · p. 2
- The bilinear penalty's tunable post-target slope makes it more flexible, and it performs better than the absolute penalty in the reported experiments.The experiments use λ′1 = 2λ1 as a suitable value.Source: §3.4 · p. 3
4. RESULTS AND ANALYSIS
On Cityscapes, the proposed codec improves rate–mIoU efficiency over the baselines while holding reconstruction near a target quality. Analysis shows bits shifting toward task-relevant regions under a tight bitrate budget.
4.1. Experimental Setup
- Evaluation uses Cityscapes with 2,975 training and 500 validation images, a frozen Mask2Former task network, and bpp, PSNR, mIoU, and mIoU-axis BD-rate metrics.The codec is Cheng2020; training uses 256 × 256 patches and validation uses center crops.Source: §4.1 · p. 3
4.2. Quantitative Results
- −22.82% BD-rate over Joint Optimization and −29.81% over the Baseline at equal task performance; fixed-target curves keep PSNR near target while improving mIoU.The bilinear penalty performs best throughout the rate–performance curves.Source: Table 1 · p. 3, §4.2 · p. 3, Figure 4 · p. 3
4.3. Analysis
- At ≈0.030 bpp, the proposed codec trades 5 dB PSNR (31.1 vs. 36.5 dB) for a 9.2-point mIoU gain (42.4% vs. 33.2%), while preserving sharp boundaries.The added error is spread more uniformly across smooth, texture-poor areas, where it matters less for segmentation.Source: §4.3 · p. 3, Figure 5 · p. 4
- Bits move toward vehicles, poles, and object boundaries and away from road and sky; relative to JO, Proposed recovers foreground regions JO mislabels.The per-pixel bit maps and segmentation error maps show this redistribution and its task-performance payoff.Source: §4.3 · p. 4, Figure 5 · p. 4
- The proposed latent is less spatially smooth, concentrates budget in a few high-energy channels, and activates task-informative channels more strongly than Baseline and JO.The analysis associates spatially regular latents with greater predictability and fewer encoding bits.Source: §4.3 · p. 4
5. CONCLUSION
The quality-constrained formulation yields mIoU-axis BD-rate savings over both reference approaches while keeping reconstruction close to the requested quality. The paper concludes that target-distortion coding is feasible and proposes spatially adaptive targets for future work.
5. CONCLUSION
- On Cityscapes, the formulation yields consistent mIoU-axis BD-rate savings over pure rate–distortion and unconstrained joint optimization while holding reconstruction close to requested quality.The codec shifts bits toward task-relevant regions.Source: §5 · p. 4
- The conclusion identifies target-distortion coding as feasible and reports that moderate penalty enforcement improves task accuracy.Both are presented as findings of the study.Source: §5 · p. 4
- Future work may adapt target quality spatially, using JND as the enforcement threshold so bits follow local regional sensitivity.Source: §5 · p. 4