Source-linked AI summary
Learning to Resize Images for Computer Vision Tasks
Hossein Talebi, Peyman Milanfar
TL;DR
The paper addresses whether conventional image resizers limit task accuracy in vision systems. It jointly trains CNN-based resizers with task models, finding consistent gains over baseline classifiers while prioritizing machine performance over perceptual quality. The approach is also extended to image quality assessment, with performance subject to memory-related batch-size constraints.
Problem
Conventional bilinear and bicubic resizing is widely used for efficiency, but its effect on task performance is under-optimized and may limit recognition accuracy.
Method
The paper jointly trains a CNN-based image resizer with baseline vision models, allowing task-specific resizing at arbitrary scaling factors without pixel or perceptual losses.
Results
The learned resizer adapts to various classification models, consistently improves over baseline image classifiers, and successfully extends to image quality assessment.
Takeaways & Limitations
Task-optimized learned resizers can replace traditional preprocessing when machine vision performance, rather than visual quality, is the primary objective.
Takeaways & Limitations
Observed gains for larger resizers may be constrained by the smaller batch sizes required under limited memory.
Abstract
from arXiv · showhide
For all the ways convolutional neural nets have revolutionized computer vision in recent years, one important aspect has received surprisingly little attention: the effect of image size on the accuracy of tasks being trained for. Typically, to be efficient, the input images are resized to a relatively small spatial resolution (e.g. 224x224), and both training and inference are carried out at this resolution. The actual mechanism for this re-scaling has been an afterthought: Namely, off-the-shelf image resizers such as bilinear and bicubic are commonly used in most machine learning software frameworks. But do these resizers limit the on task performance of the trained networks? The answer is yes. Indeed, we show that the typical linear resizer can be replaced with learned resizers that can substantially improve performance. Importantly, while the classical resizers typically result in better perceptual quality of the downscaled images, our proposed learned resizers do not necessarily give better visual quality, but instead improve task performance. Our learned image resizer is jointly trained with a baseline vision model. This learned CNN-based resizer creates machine friendly visual manipulations that lead to a consistent improvement of the end task metric over the baseline model. Specifically, here we focus on the classification task with the ImageNet dataset, and experiment with four different models to learn resizers adapted to each model. Moreover, we show that the proposed resizer can also be useful for fine-tuning the classification baselines for other vision tasks. To this end, we experiment with three different baselines to develop image quality assessment (IQA) models on the AVA dataset.
1. Introduction
The paper argues that conventional image resizing is an under-optimized component of vision systems and proposes jointly learned, task-specific resizers. These resizers adapt to classification models, support arbitrary scaling factors, and extend to image quality assessment.
- Motivation: Image down-scaling is widely used because batch processing, memory limits, and computational cost require smaller, uniform image resolutions.The resolution–batch-size trade-off can significantly affect recognition accuracy.
- Motivation: Nearest-neighbor, bilinear, and bicubic resizers are fast and flexible but were not designed for modern machine perception tasks.The paper positions them as off-the-shelf preprocessing methods that may be inadequate for task performance.
- Proposed approach: The proposed resizer is jointly trained with classification models to improve classification performance rather than merely reproduce conventional visual processing.It is specifically developed as a preprocessing model for vision tasks and aims to replace off-the-shelf resizers.
- Results: The resizer adapts to various classification models and consistently improves over the baseline image classifier.The paper reports this contribution for image classification experiments on ImageNet.
- Results: The learned resizer is not constrained by pixel or perceptual losses, producing machine-adaptive visual effects that differ from conventional image processing.These effects prioritize task performance over perceptual similarity.
- Scope: The model supports arbitrary down-scaling factors and is also shown to adapt successfully to image quality assessment.These properties allow searching for task-specific resolutions and extending the approach beyond classification.
2. Related Work
Related work has optimized image processing for recognition, enhancement, compression, and perceptual quality. The paper differs by jointly learning a resolution-changing preprocessing model with the recognition network and allowing task-specific effects unconstrained by perceptual quality.
- Recognition-aware processing: Recognition-oriented preprocessing has been applied to super-resolution, denoising, JPEG deblocking, compression, and low-light imaging.These studies evaluate or optimize image processing using recognition-driven objectives alongside perceptual or rate-distortion considerations.
- Paper distinction: The proposed approach differs by changing spatial resolution, jointly training the resizer and recognition model, and optimizing recognition performance regardless of perceptual quality.This joint training is intended to adapt the preprocessing model to the recognition architecture.
- Paper distinction: Unlike methods targeting specific enhancements such as denoising or deblurring, the proposed resizer can learn machine-friendly visual effects freely.The paper connects this flexibility to applications where visually pleasing images are not the end goal.
3. Proposed Framework
The framework introduces a lightweight, jointly trained CNN resizer for task-specific image scaling, designed to improve performance without imposing image-quality constraints. It supports classification and image quality assessment models through task-specific losses.
- Flexible resizing: The model supports arbitrary scaling factors, including up- and down-scaling, enabling exploration of the resolution–batch size trade-off.This flexibility is intended to help identify the optimal resolution for a given task.
- Resizer architecture: The architecture combines bilinear feature resizing with a skip connection that merges the resized image and CNN features.Bilinear feature resizing incorporates features computed at the original resolution, while the skip connection eases learning by directly passing the bilinearly resized image.
- Resizer architecture: The resizer uses r = 1 or 2 residual blocks, n = 16 kernels in intermediate layers, and 7 × 7 kernels in the first and last layers.The first layer's larger kernel provides a 7 × 7 receptive field at the original image resolution.
- Model complexity: The proposed model is lightweight, adds few trainable parameters, and remains effective even with the n = 16, r = 1 configuration.The paper contrasts its size with ResNet-50, which has about 23 million parameters.
- Joint training: The resizer is jointly trained with the baseline model loss and imposes no loss or regularization constraint on the resized image.Its objective is to learn an optimal resizer for the baseline vision task.
- Task losses: For classification, the framework uses cross-entropy with label smoothing, whereas AVA quality assessment uses Earth Mover’s Distance over human-rating distributions.The classification setup predicts 1000 ImageNet classes; the AVA setup predicts 10 rating logits and uses d = 2.
4. Experiments
Experiments evaluate learned resizers across ImageNet classification and AVA image-quality assessment, varying resolutions, architectures, and computational settings. The resizers generally improve task performance, transfer across baselines, and produce task-adapted visual changes that may not meet human perceptual standards.
- Classification: Four ImageNet classification baselines are trained with learned resizers across input resolutions, with batch sizes adjusted to respect memory limits.The experiments compare learned and conventional resizing configurations, reporting FLOPS for computational fairness.
- Classification: Learned resizers generally improve ImageNet classification over the default bilinear baseline, with gains varying across architectures and FLOPS.DenseNet-121 and MobileNet-v2 show the largest and smallest gains, respectively; several models outperform bilinear resizing at comparable FLOPS.
- Classification: Increasing input resolution benefits DenseNet-121, ResNet-50, and MobileNet-v2, whereas Inception-v2 benefits most from larger batch sizes.Equal input and output resolutions can also improve performance, but the best results usually use a larger resizer input resolution.
- Classification: The learned resizers emphasize high-frequency details and can create overly sharpened outputs that improve machine vision despite failing to meet human perceptual standards.Visual effects differ by model: Inception and DenseNet emphasize fine-grained detail, while EfficientNet produces stronger color shifts and modest detail manipulation.
- Quality Assessment: On AVA image-quality assessment, learned resizers consistently improve baseline models, though EfficientNet remains more challenging at higher FLOPS.For Inception-v2 and DenseNet-121, the learned resizer outperforms bicubic resizing with comparable FLOPS.
- Ablation: Larger resizers yield modest gains for DenseNet and MobileNet but not Inception or ResNet, possibly because memory limits force smaller batch sizes.The reduced batch size may inadvertently limit the observed performance gain of larger resizer models.
5. Conclusions
The paper presents learned image resizing as a way to improve image-recognition performance by optimizing preprocessing effects for machine vision rather than perceptual quality.
- The framework learns preprocessing effects that boost image-recognition model performance.
- The approach focuses on image resizing and omits intermediate pixel or perceptual losses, optimizing the resized outputs exclusively for machine vision tasks.
- Experiments show that task-optimized deep vision models can benefit from replacing traditional image resizers with learned resizers.
- The paper identifies customized preprocessing for machine vision as an area with significant room for further research and plans to extend the model to other vision tasks.