Source-linked AI summary
Shallow and Deep Convolutional Networks for Saliency Prediction
Junting Pan, Kevin McGuinness, Elisa Sayrol, Noel O'Connor, Xavier Giro-i-Nieto
TL;DR
Saliency prediction traditionally relied on hand-crafted features, motivating a data-driven approach that learns pixel-level attention maps end to end. The paper trains shallow and deep CNN regressors, using scratch training or transferred classification layers, and reports strong benchmark performance with different memory and speed trade-offs.
Problem
Saliency prediction requires pixel-level human-fixation maps rather than global image labels, while traditional methods rely on hand-crafted neuroscience-inspired features.
Method
The paper trains two end-to-end CNNs with Euclidean regression: a shallow network from scratch and a deep network initialized with the first three layers of a classification CNN.
Results
Both architectures perform strongly on saliency benchmarks; the shallow and deep networks are similar on iSUN, while the deep network performs better on MIT300.
Takeaways & Limitations
End-to-end regression is effective for saliency prediction, with the shallow network offering lower training-time memory and faster map generation.
Takeaways & Limitations
The deep network’s practical input size is constrained by GPU or RAM required to store intermediate layer outputs.
Abstract
from arXiv · showhide
The prediction of salient areas in images has been traditionally addressed with hand-crafted features based on neuroscience principles. This paper, however, addresses the problem with a completely data-driven approach by training a convolutional neural network (convnet). The learning process is formulated as a minimization of a loss function that measures the Euclidean distance of the predicted saliency map with the provided ground truth. The recent publication of large datasets of saliency prediction has provided enough data to train end-to-end architectures that are both fast and accurate. Two designs are proposed: a shallow convnet trained from scratch, and a another deeper solution whose first three layers are adapted from another network trained for classification. To the authors knowledge, these are the first end-to-end CNNs trained and tested for the purpose of saliency prediction.
1. Introduction
The paper frames saliency prediction as estimating pixel-level visual-attention maps from image data rather than relying on hand-crafted neuroscience-inspired features. It proposes end-to-end CNN regression using newly available large saliency datasets.
- Saliency maps represent the probability of visual attention through human eye-gaze fixation points.
- Saliency prediction differs from image classification because it requires costly human-fixation annotations and a spatially coherent score for every image pixel.
- The paper proposes two end-to-end regression architectures: a shallow convnet trained from scratch and a deep convnet reusing the first three layers of a classification network.
- The work presents these as the first convnets formulated and evaluated for saliency prediction as an end-to-end regression problem.
2. Related work
Prior work used convolutional features or multi-network architectures for saliency and salient-object detection. This paper instead targets saliency maps directly with end-to-end training on saliency-specific databases.
- Earlier convnet saliency methods combined feature maps with linear models, while DeepGaze transferred classification features and learned their combination.
- Other approaches used parallel or nested networks at multiple scales to model local and global context for salient-object detection.
- Fully convolutional segmentation methods established end-to-end pixel prediction, with later work improving spatial consistency using conditional random fields.
- This work focuses on predicting saliency maps rather than detecting salient objects, using end-to-end convnets trained on databases annotated for saliency prediction.
3. Shallow Convnet
The shallow model is a lightweight five-layer convnet trained from scratch for saliency regression. Its design limits depth and resolution to address overfitting, while regularization and output resizing produce final saliency maps.
- The shallow network has three convolutional and two fully connected learned-weight layers, totaling 64.4 million parameters.
- The architecture uses three convolution–ReLU–max-pooling stages and reduces [96 × 96] inputs to [10 × 10] feature maps.
- Norm-constraint regularization for maxout layers was essential to mitigate overfitting, whereas 0.5 dropout after the first fully connected layer provided little improvement.
- The 2,304-dimensional output is reshaped into a [48 × 48] saliency map, then resized to the input dimensions and Gaussian-filtered with standard deviation 2.0.
- The shallow convnet required 200 ms per image for saliency prediction.
- Separate models were trained from scratch on SALICON and iSUN, with the iSUN model reducing the third convolutional layer from 128 to 64 units.
4. Deep Convnet
The deep model adapts low-level filters from a pretrained classification CNN and adds saliency-specific layers. It uses a fully convolutional design that produces input-sized maps and can accommodate different image sizes within memory limits.
- The deep convnet transfers the first three convolutional layers from a classification network and trains new layers specifically for saliency.
- The architecture contains 10 weight layers and 25.8 million parameters, with pooling followed by a deconvolution layer that restores the input spatial dimensions.
- Validation experiments found that adding layers and feature maps generally improved accuracy, while dropout increased training time without significantly improving accuracy.
- The network was trained on 9,000 SALICON images with 1,000 held out for validation, using stochastic gradient descent with Euclidean loss.
- The model can process images of different sizes and match their aspect ratios, but practical input size remains constrained by GPU or RAM needed for intermediate outputs.
5. Experiments
The experiments compare the proposed shallow and deep convnets through memory requirements, cross-dataset evaluation, and benchmark performance. Results show similar iSUN validation performance, strong challenge results for the shallow model, and competitive MIT300 performance for the deep model.
- 5.1. Memory requirements: The networks require memory for learned parameters and intermediate feature-map data, with training blob estimates doubling test-time forward-pass storage.Parameters are fit during training, while intermediate blobs are created after each processing stage.
- 5.1. Memory requirements: Both networks use far fewer parameters than very deep classification networks, such as the 144-million-parameter, 19-layer VGG net.
- 5.1. Memory requirements: The shallow network uses less layer-output memory but more parameters because of its fully connected layers, while the deep network avoids overfitting with less regularization.Lower shallow-network blob memory also permits larger test-time batches.
- 5.3. Results: The experiments evaluate the convnets with multiple saliency metrics on MIT and LSUN benchmarks, using SALICON for training and iSUN and MIT300 for testing except in the LSUN challenge.The datasets vary in size and collection procedure, including controlled expert fixations and crowdsourced saliency maps.
- 5.3. Results: The two networks show similar performance on the 926-image iSUN validation set, while qualitative examples show shallow central bias and higher deep-network spatial resolution.The deep architecture retains larger feature maps, supporting higher spatial resolution.
- 5.3. Results: The shallow convnet won the 2015 LSUN saliency prediction challenge, with every reported measure demonstrating superior performance over other participants.For the challenge, training used separate dataset partitions without mixing images across datasets.
- 5.3. Results: On MIT300, the deep convnet achieved results similar to Deep Gaze 1, while the shallow model remained in the upper part of a 47-model benchmark table.The authors also report strong results across multiple datasets and benchmarks despite training only on SALICON rather than MIT1003.
6. Conclusions
Both architectures perform strongly on saliency benchmarks, while the shallow and deep models trade off speed and memory against dataset-dependent performance.
- Both architectures achieve excellent results on state-of-the-art saliency benchmarks compared with hand-crafted solutions.
- The deep network performs better than the shallow network on MIT300, while their results are similar on iSUN.
- The shallow network requires less training memory and generates saliency maps faster because it has fewer layers.
- Both networks rank highly on MIT300 despite not being trained on that dataset, indicating generalization across dataset biases.