Source-linked AI summary
A Deep Multi-Level Network for Saliency Prediction
Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, Rita Cucchiara
TL;DR
Saliency prediction must model human eye fixations despite the difficulty of combining diverse visual cues and the limitations of earlier deep models. This paper proposes an end-to-end architecture that combines CNN features from multiple levels, learns a prior, and uses a new loss function. It outperforms competitors on SALICON across reported metrics and achieves competitive results on MIT300.
Problem
Hand-crafted saliency features are difficult to combine, while early deep models were constrained by limited training data and single-level feature use.
Method
An end-to-end architecture combines low-, medium-, and high-level CNN features through an encoding network, learns a prior, and trains with a loss addressing saliency-map imbalance.
Results
12% on CC, 4% on AUC shuffled, and 1% on AUC Judd are the reported margins over competitors on SALICON; the method also achieves competitive results on MIT300.
Takeaways & Limitations
The architecture's multi-level feature combination and learned prior provide an effective saliency-prediction approach across the SALICON and MIT300 benchmarks.
Abstract
from arXiv · showhide
This paper presents a novel deep architecture for saliency prediction. Current state of the art models for saliency prediction employ Fully Convolutional networks that perform a non-linear combination of features extracted from the last convolutional layer to predict saliency maps. We propose an architecture which, instead, combines features extracted at different levels of a Convolutional Neural Network (CNN). Our model is composed of three main blocks: a feature extraction CNN, a feature encoding network, that weights low and high level feature maps, and a prior learning network. We compare our solution with state of the art saliency models on two public benchmarks datasets. Results show that our model outperforms under all evaluation metrics on the SALICON dataset, which is currently the largest public dataset for saliency prediction, and achieves competitive results on the MIT300 benchmark.
I. INTRODUCTION
Saliency prediction aims to model human eye fixations, but hand-crafted features and early deep models were limited by the variety of relevant visual stimuli and scarce training data. The paper addresses these challenges with an end-to-end architecture that combines multi-level CNN features, learns its own prior, and uses a new loss function.
- Saliency models can predict either initial eye fixations or important object regions; this paper focuses on eye-fixation prediction.
- Hand-crafted approaches combine low-level cues such as color, texture, and contrast with higher-level concepts including faces, people, text, and horizons.The wide range of factors contributing to visual saliency makes appropriate manual combination difficult.
- Early deep saliency models were constrained by limited training data, requiring few layers or pretrained architectures developed for other tasks.
- SALICON increased available training data through crowdsourcing, enabling deeper convolutional architectures with reduced overfitting risk.
- The proposed architecture combines multi-level CNN features end-to-end, learns a training-data-based prior, and introduces a loss function addressing saliency-map imbalance.
II. RELATED WORK
Earlier saliency methods used biologically inspired or hand-tuned features, while initial deep models were limited by available training data. The proposed model instead combines CNN features across levels through an encoding network and applies a learned prior.
- Classical saliency systems extracted multi-scale color, intensity, and orientation features to model biologically plausible visual attention.
- Other approaches combined low-level features with semantic information about faces, cars, and text to improve eye-fixation prediction.
- These earlier approaches generally relied on hand-tuned features or separately trained higher-level classifiers.
- The proposed architecture feeds CNN features into an encoding network that learns feature weighting and applies a learned prior to the predicted saliency map.
- The eDN deep model used a three-layer CNN but could not scale to outperform state-of-the-art methods because saliency training data were limited.
III. PROPOSED APPROACH
The proposed architecture combines multi-level CNN features, learns a saliency-specific prior, and trains the complete system end-to-end with a loss addressing saliency-map imbalance.
- Architecture: The model extracts low-, medium-, and high-level CNN features, encodes them into saliency-specific features, then applies a learned prior to produce the final prediction.The architecture comprises feature extraction, feature encoding, and prior learning components.
- Feature extraction: The feature extraction network is a 13-layer Fully Convolutional network that produces feature maps for the encoding network.It is built on the VGG-16 architecture.
- Feature extraction: Removing the final pooling stage and reducing the preceding stride rescale feature maps by a factor of 8 instead of 32.These changes limit the spatial-resolution reduction of the VGG-16-based network.
- Encoding network: Feature maps from the third pooling layer, last pooling layer, and last convolutional layer are concatenated into a 1280-channel tensor for saliency encoding.A dropout layer and subsequent convolutions learn 64 saliency-specific maps and combine them into the predicted feature map.
- Prior learning: The prior-learning network estimates a coarse w′ × h′ mask, upsamples it, and multiplies it pixelwise with the predicted saliency map.The mask is initialized to one, with w′ and h′ set to floor(w/10) and floor(h/10).
- Training: Training minimizes a loss combining pixelwise similarity, prediction normalization, fixation-probability weighting, and L2 regularization of the prior mask.The weighting gives greater importance to pixels with high ground-truth fixation probability, while regularization discourages unnecessary prior changes.
A. Experimental setup
The evaluation uses SALICON and MIT300, with saliency predictions compared against human-derived fixation maps using several complementary metrics.
- Datasets: SALICON contains 10,000 training, 5,000 validation, and 5,000 testing images from Microsoft CoCo, with maps generated from mouse movements.The dataset is described as the largest public saliency-prediction dataset.
- Datasets: MIT300 contains 300 natural images with fixation maps from 39 observers, and evaluation requires submission to its saliency benchmark.MIT1003 is suggested for fine-tuning and contains 1,003 eye-tracked images from 15 participants.
- Evaluation metrics: Similarity measures pixelwise overlap between predicted and human fixation probability distributions, with a score of one indicating identical maps.Both maps are treated as distributions that sum to one.
- Evaluation metrics: CC measures the Pearson linear relationship between predicted and human fixation maps, ranging from −1 to 1.Values near −1 or 1 indicate a perfect linear relationship.
- Evaluation metrics: EMD measures the minimum cost of transforming the predicted saliency distribution into the human-fixation distribution, so larger values indicate greater difference.This metric evaluates distributional discrepancy rather than direct pixelwise agreement.
- Evaluation metrics: NSS averages normalized saliency values at fixation locations, while AUC variants distinguish fixation points from sampled or shuffled non-fixation points.The experiments use AUC Judd, AUC Borji, and shuffled AUC.
B. Feature importance analysis
The analysis estimates feature importance through local sensitivity of predicted saliency outputs, then aggregates contributions across activation maps. Results show that conv3, conv4, and conv5 features all contribute, with high-level features most influential and medium-level features also substantial.
- In a linear formulation, the magnitude of each weight indicates the corresponding feature’s importance for a predicted saliency pixel.Zero-weighted features are ignored, while features with large absolute weights have greater effects.
- For the nonlinear model, local partial derivatives approximate which input features must change to affect the prediction.The approximation is equivalent to a first-order Taylor expansion.
- Importance is estimated across validation samples by averaging element-wise absolute gradients and L1-normalizing activation-map contributions.The method computes gradients of the output map’s mean and variance to reduce computation.
- All conv3, conv4, and conv5 features contribute to prediction; high-level features are most relevant, while medium-level features also play a considerable role.This supports combining activations from multiple CNN levels.
C. Comparison with state of the art
The model is evaluated on SALICON and MIT300, with quantitative, qualitative, and failure-case comparisons against other saliency approaches. It performs strongly on both benchmarks but fails in images lacking a clear object.
- On SALICON, the model outperforms all competitors by 12% on CC, 4% on AUC shuffled, and 1% on AUC Judd.
- On MIT300, the method outperforms most leaderboard approaches and achieves competitive results against the top-ranked methods.
- Qualitative examples from SALICON and MIT1003 show predictions more similar to ground truth than those generated by other methods.
- When images lack a clear, explicit object, center-biased eye fixations can cause the model to fail.
V. CONCLUSIONS
The paper concludes that its end-to-end architecture combines medium- and high-level CNN features with a learned prior for saliency prediction. Qualitative and quantitative comparisons support its effectiveness on major public datasets and benchmarks.
- The proposed architecture learns a nonlinear combination of medium- and high-level CNN features together with a prior for saliency maps.The complete model remains trainable end-to-end.
- Qualitative and quantitative comparisons demonstrate the proposal’s effectiveness on the biggest dataset and most popular public benchmark for saliency prediction.