Source-linked AI summary
Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
Adrià Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, Antonio Torralba
TL;DR
Uniformly downsampling high-resolution images can discard task-relevant details needed by input-constrained neural networks. The paper introduces a differentiable saliency sampler that learns task-specific resampling, and reports improvements for gaze estimation and fine-grained classification, including 5% and 3% gains in top-1 and top-5 accuracy.
Problem
Input-constrained vision networks commonly downsample large images uniformly, which can lose information from regions and scales needed for a task.
Method
The paper adds a differentiable saliency sampler before an existing task network to allocate pixels toward task-relevant regions and train the full pipeline end-to-end.
Results
5% and 3% gains in top-1 and top-5 accuracies, respectively, are reported over the ResNet-101 baseline for fine-grained classification.
Takeaways & Limitations
The sampler is most useful when important features are small, sparse, or distributed across multiple scales, while preserving efficient fixed-memory processing.
Takeaways & Limitations
The method has a scope boundary because deformation near one magnified region can discourage preserving another, potentially harming tasks such as text recognition.
Abstract
from arXiv · showhide
We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer can be added as a preprocessing block to existing task networks and trained altogether in an end-to-end fashion. The effect of the layer is to efficiently estimate how to sample from the original data in order to boost task performance. For example, for an image classification task in which the original data might range in size up to several megapixels, but where the desired input images to the task network are much smaller, our layer learns how best to sample from the underlying high resolution data in a manner which preserves task-relevant information better than uniform downsampling. This has the effect of creating distorted, caricature-like intermediate images, in which idiosyncratic elements of the image that improve task performance are zoomed and exaggerated. Unlike alternative approaches such as spatial transformer networks, our proposed layer is inspired by image saliency, computed efficiently from uniformly downsampled data, and degrades gracefully to a uniform sampling strategy under uncertainty. We apply our layer to improve existing networks for the tasks of human gaze estimation and fine-grained object classification. Code for our method is available in: http://github.com/recasens/Saliency-Sampler
1 Introduction
The paper addresses information loss from uniformly downsampling high-resolution images by introducing a task-specific saliency sampler that reallocates pixels toward informative regions. The differentiable layer integrates with existing networks and improves performance on tasks involving small or sparse features.
- 1 Introduction: Uniform downsampling is standard for fitting variable-size images into constrained network inputs, but can lose information needed across spatial resolutions and locations.ImageNet classifiers commonly use 224 × 224 inputs even when source images reach several megapixels.
- 1 Introduction: The saliency-based sampling layer learns task-specific downsampling that emphasizes informative image regions while suppressing irrelevant content.It creates a distorted, caricature-like image representation by allocating more target pixels to task-relevant areas.
- 1 Introduction: The sampler is a differentiable plug-in that estimates saliency and can be trained end-to-end before an existing task network.Unlike sequential attention, it operates in a single pass at constant computational cost.
- 1 Introduction: Across gaze estimation and fine-grained recognition, adding the sampler consistently improves performance over baseline networks, especially when important features are small, sparse, or span multiple scales.The method is motivated by task-dependent sampling analogous to humans combining high-acuity and peripheral vision.
2 Related Work
Related work spans attention, saliency-based methods, and adaptive image sampling. The proposed sampler differs by using task saliency to resample available high-resolution data in one differentiable module rather than transforming or repeatedly cropping low-resolution inputs.
- 2 Related Work: Spatial Transformer Networks learn parametric transformations, whereas the saliency sampler allocates sampling density to task-salient regions.The paper notes that TPS-based transformers can suffer extreme transformations and fold-overs, while the proposed formulation implicitly prevents them.
- 2 Related Work: Unlike deformable convolutions, the sampler can use any available image resolution and can be applied to existing trained networks without modifying their convolutional layers.The paper also contrasts saliency-map estimation with the difficulty of directly learning local spatial offsets.
- 2 Related Work: Sequential attention methods recursively locate or crop multiple regions, whereas the proposed approach performs task-dependent sampling without requiring a predefined number of crops.This supports simultaneous emphasis of multiple non-collocated regions.
- 2 Related Work: Saliency-based approaches use task-relevant regions identified by CNNs, including class activation maps and iterative cropping for fine-grained classification.These methods motivate using learned task saliency to guide image processing.
- 2 Related Work: Adaptive image sampling also appears in image retargeting, but retargeting preserves content for human observers while this method creates an intermediate representation for another task.The paper distinguishes the objectives even though both can use saliency-driven deformation.
3 Saliency Sampler
The saliency sampler predicts a task-specific saliency map from a low-resolution image, then uses it to resample the original high-resolution image into a fixed-size representation. Its convolutional, differentiable formulation supports efficient end-to-end training while concentrating samples in salient regions.
- 3 Saliency Sampler: The saliency map is generated from the low-resolution image with a flexible CNN and normalized using a final softmax operation.This makes saliency task-specific while allowing the estimator architecture to vary.
- 3 Saliency Sampler: The sampler computes a mapping from output coordinates to the original image, producing J(x, y) = I(u(x, y), v(x, y)).The mapping aims to distribute samples proportionally to normalized saliency weights.
- 3 Saliency Sampler: Higher-saliency regions are sampled more densely because saliency mass attracts neighboring sampling locations.A Gaussian kernel regularizes the attraction field, with σ set to one third of the saliency-map width in the experiments.
- 3 Saliency Sampler: Convolutional computation makes the sampling coordinates efficient and preserves differentiability for backpropagation through standard CNNs.The method uses a deterministic grid sampler to obtain the resampled image.
- 3 Saliency Sampler: The basic formulation is biased toward the image center, so the method pads the saliency map with border values to avoid that effect.This is an explicit correction to the coordinate-construction procedure.
- 3 Saliency Sampler: The pipeline obtains a low-resolution image, predicts saliency, resamples the high-resolution image accordingly, and feeds the fixed-size result to the task network.Both the saliency network and task network are trained jointly.
4 Experiments
Experiments evaluate the saliency sampler for gaze tracking and fine-grained classification, comparing it with standard networks and alternative sampling methods across multiple datasets and architectures. The layer improves performance in these tasks, including gaze estimation, iNaturalist classification, and CUB-200 recognition.
- Gaze Tracking: The gaze model performs similarly to iTracker while compressing four 224×224 inputs into one 227 × 227 image, and it improves over competing sampling baselines.Improvements over Deformable Convolutions, both STN variants, and the Grid Estimator range from 0.62 to 1.92 cm for iPad and 0.17cm to 0.59 cm for iPhone.
- Fine-Grained Classification: 5% top-1 and 3% top-5 accuracy gains over the ResNet-101 baseline are reported on iNaturalist fine-grained classification.The method also outperforms the CAM-based method by several points and can zoom into arbitrary numbers of non-collocated regions.
- Fine-Grained Classification: 66% top-1 and 87% top-5 accuracy are achieved with Inception V3, compared with 64% and 86% for the original higher-resolution model.
- Fine-Grained Classification: Increasing the saliency network depth improves overall performance, but with diminishing returns.The ablation varies ResNet-18-based saliency networks with 6, 10, or 14 layers.
- CUB-200: 2.9% accuracy improvement is obtained on CUB-200, while the method outperforms the comparable 224 × 224 RN-50 DT-RAM by 1.7%.The 448×448 DT-RAM version remains more accurate but uses approximately 2 passes through RN-50 on average and a larger input size.
5 Discussion
The saliency sampler is most useful when task-relevant features are small, sparse, or distributed across scales, though its deformations may challenge tasks requiring geometric preservation. In practice, it handled collocated eyes without reducing gaze-prediction performance and was easier to train than spatial-sampling alternatives.
- The saliency sampler is most beneficial when important features are small, sparse, or appear across multiple image scales.
- Deformations near magnified regions could discourage strong deformations elsewhere, potentially harming tasks such as text recognition.The text presents this as a potential risk rather than an observed failure.
- Table 4 reports performance improvements from adding the sampling layer on the CUB-200 fine-grained classification dataset.The table caption identifies input resolution as the model's image resolution.
- The method handled magnifying both collocated eyes without hindering gaze-prediction performance, despite the task requiring geometric information preservation.
- The method proved easier to train than other approaches that modify spatial sampling, including Spatial Transformer networks.
6 Conclusion
The saliency sampler adapts image sampling to improve task performance while preserving memory and computational efficiency. It focuses sampling on task-relevant features for gaze tracking and fine-grained recognition, while supporting flexible, robust image-wide redistribution.
- 6 Conclusion: The saliency sampler adapts image sampling to improve task performance while preserving memory allocation and computational efficiency.It is designed as a CNN layer for image-processing tasks.
- 6 Conclusion: The method effectively locates and focuses on features important for gaze tracking and fine-grained object recognition.These tasks include recovering small features such as eyes and subtle differences between related animal species.
- 6 Conclusion: Unlike transformations restricted to predefined regions, the sampler redistributes sampling density across the entire image domain.A single scalar attention map also makes the method robust against irrecoverable degradation from fold-overs or singularities.