Source-linked AI summary
End-to-End Saliency Mapping via Probability Distribution Prediction
Saumya Jetley, Naila Murray, Eleonora Vig
TL;DR
Saliency estimation methods often use losses designed for classification or regression even though they predict topographical maps from stochastic eye-fixation data. This paper models saliency maps as generalized Bernoulli distributions and trains a deep architecture with probability-distance losses, reporting superior performance over standard losses and state-of-the-art methods across four datasets.
Problem
Existing deep saliency methods use classification- or regression-oriented losses even though saliency estimation predicts topographical maps from stochastic, aggregated fixations.
Method
The paper represents saliency maps as generalized Bernoulli distributions and trains a deep network with softmax paired with probability-distance loss functions.
Results
The proposed method achieves superior performance over standard losses and state-of-the-art saliency methods across four datasets.
Takeaways & Limitations
Probability-based modeling and task-adapted learning criteria are effective for saliency-map prediction.
Takeaways & Limitations
The model may emphasize bodies and faces over text because SALICON contains many images with people and faces, while relatively little training data involves text.
Abstract
from arXiv · showhide
Most saliency estimation methods aim to explicitly model low-level conspicuity cues such as edges or blobs and may additionally incorporate top-down cues using face or text detection. Data-driven methods for training saliency models using eye-fixation data are increasingly popular, particularly with the introduction of large-scale datasets and deep architectures. However, current methods in this latter paradigm use loss functions designed for classification or regression tasks whereas saliency estimation is evaluated on topographical maps. In this work, we introduce a new saliency map model which formulates a map as a generalized Bernoulli distribution. We then train a deep architecture to predict such maps using novel loss functions which pair the softmax activation function with measures designed to compute distances between probability distributions. We show in extensive experiments the effectiveness of such loss functions over standard ones on four public benchmark datasets, and demonstrate improved performance over state-of-the-art saliency methods.
1. Introduction
The paper frames saliency prediction as learning topographical maps from eye-fixation data and proposes probability-distribution modeling with specialized losses for deep networks.
- Motivation: The work targets visual saliency maps from input images, motivated by applications including cropping, resizing, thumbnailing, and object recognition.Saliency prediction is presented as increasingly important as vision models and datasets grow.
- Proposed approach: The authors formulate each saliency map as a generalized Bernoulli distribution and introduce loss functions based on distances between probability distributions.The proposed losses are paired with softmax activation functions.
- Results: Training a deep architecture with the proposed losses outperforms standard Euclidean and Huber regression losses.The paper also compares the proposed probability-distance losses with one another.
- Architecture: The fully convolutional architecture generates a saliency map for a large image in 200ms on modern GPUs.This provides the model with a stated large-image inference speed.
- Evaluation: Experiments across four datasets compare the approach with alternative loss functions and state-of-the-art saliency methods, with Figure 1 illustrating a prediction.The paper reports the approach as effective in these comparisons.
2. Related work
Prior saliency methods span shallow or deep architectures and supervised or unsupervised learning, while recent end-to-end models increasingly learn from large attention datasets.
- Overview: Existing saliency approaches form four categories defined by shallow versus deep architectures and supervised versus unsupervised learning.The related-work review organizes methods using these two dimensions.
- Unsupervised shallow methods: Unsupervised shallow methods use bottom-up cues such as color, contrast, motion, self-information, and spectral properties with little learning.These approaches include biologically motivated and information-theoretic models.
- Supervised shallow methods: Supervised shallow methods learn saliency using models such as Markov chains, support vector machines, and AdaBoost, sometimes adding semantic detection maps.Reported top-down cues include faces, persons, cars, and the horizon.
- Hierarchical methods: Deep methods range from unsupervised feature reuse, such as DeepGaze, to supervised end-to-end models trained on large attention datasets.SALICON and TurkerGaze/iSUN enabled task-specific training of deep saliency architectures.
- Position of this work: The paper distinguishes its approach from prior regression and KL-divergence methods by studying probability-distance losses, with Bhattacharyya loss outperforming KL divergence across four standard metrics.The comparison is positioned as a probability-based alternative for saliency-map learning.
3. Saliency maps as probability distributions
The paper models saliency maps as probability distributions over pixels and trains a fully convolutional network with probability-distance losses. It uses Gaussian-smoothed fixation maps as targets, softmax outputs, and a VGGNet-based architecture with dataset-specific fine-tuning.
- 3. Saliency maps as probability distributions: Saliency maps are modeled as generalized Bernoulli distributions over pixels, with each pixel value representing its fixation probability.Ground-truth maps aggregate multiple observers’ fixations and omit temporal fixation information.
- 3.1. Learning to predict the probability of fixation: The network predicts distributions with softmax, while fixation maps are Gaussian-smoothed and normalized before serving as ground-truth targets.The predicted distribution is generated directly from the network’s final response map.
- 3.1. Learning to predict the probability of fixation: Training pairs softmax with χ2, total-variation, cosine, and Bhattacharyya distances, while also evaluating KL divergence and standard Euclidean or Huber losses.The proposed pairings are presented as probability-distribution objectives for stochastic-gradient training.
- 3.2. Training the prediction model: The fully convolutional model uses pretrained VGGNet convolutional layers followed by additional convolutional layers to produce a response map.The architecture contains 16 convolutional layers and 15,530,481 weights.
- 3.2. Training the prediction model: Dataset-specific fine-tuning is used because saliency datasets have collection-specific biases and SALICON mouse clicks only approximate eye fixations.Reported SALICON-to-eye-fixation agreement ranges from sAUC 0.655 to 0.965.
4. Experimental evaluation
The evaluation compares probability-distribution loss functions and the Bhattacharyya-based PDP model across saliency benchmarks, finding strong benchmark performance and generalization beyond free-viewing fixation data.
- Loss functions: Bhattacharyya distance is the best-performing loss, while KL-divergence also compares favorably with Euclidean and Huber regression losses.The distribution-based losses are reported to perform better than standard regression losses, with robustness to outliers attributed to suppressing large probability differences.
- Comparison to the state of the art: The Bhattacharyya-based PDP model outperforms all published results on the SALICON challenge across all three reported metrics.Table 3 presents the comparison among methods.
- Comparison to the state of the art: PDP performs comparably to state-of-the-art methods on MIT-300 after fine-tuning on MIT-1003 images.The MIT-300 comparison uses MIT-1003 as the fine-tuning set and is reported in Table 4.
- Comparison to the state of the art: PDP achieves the highest sAUC on OSIE using 10-fold cross-validation with 500 training and 200 validation images.The small OSIE training set was sufficient to adapt the model.
- Comparison to the state of the art: On VOCA-2012, PDP outperforms the state of the art both before and after fine-tuning, suggesting consistency between task-dependent and free-viewing fixations.VOCA-2012 is used to test generalization to task-dependent fixation prediction.
- Discussion: The probabilistic formulation treats normalized fixation maps as spatial distributions and accounts for variation across observers and data-collection paradigms.Qualitative predictions emphasize center-surround contrast, bodies, faces, and, less strongly, text.
5. Conclusion
The paper introduces a saliency formulation that predicts maps as probability distributions and trains a deep network with distance-based objectives. Across four datasets, the method outperforms other loss functions and state-of-the-art saliency estimators.
- The proposed deep-network model predicts saliency maps as probability distributions using an objective that penalizes distances between target and predicted maps.
- Across four datasets, the method achieves superior performance against alternative loss functions and state-of-the-art saliency estimation methods.