Source-linked AI summary
Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet
Matthias Kümmerer, Lucas Theis, Matthias Bethge
TL;DR
State-of-the-art saliency models capture limited fixation information, motivating reuse of ImageNet-trained recognition features for fixation prediction. Deep Gaze I uses the Krizhevsky network with point-process maximum-likelihood learning and substantially outperforms existing saliency models, while exposing high-level feature influences on fixations.
Problem
Existing saliency models explain limited fixation information, motivating models that capture high-level image features such as faces, text, and objects.
Method
Deep Gaze I reuses convolutions from the top-most layer of the ImageNet-trained Krizhevsky network and optimizes saliency by maximizing point-process log-likelihood.
Results
56% of explainable information gain was achieved versus 34% for eDN, and Deep Gaze I substantially outperformed state-of-the-art saliency models.
Takeaways & Limitations
ImageNet-trained features can boost saliency prediction and support fixation analysis involving high-level features such as faces, text, and popout.
Takeaways & Limitations
The MIT Saliency Benchmark reports performances only for saliency maps with the nonparametric prior, making shuffled AUC performance lower there.
Abstract
from arXiv · showhide
Recent results suggest that state-of-the-art saliency models perform far from optimal in predicting fixations. This lack in performance has been attributed to an inability to model the influence of high-level image features such as objects. Recent seminal advances in applying deep neural networks to tasks like object recognition suggests that they are able to capture this kind of structure. However, the enormous amount of training data necessary to train these networks makes them difficult to apply directly to saliency prediction. We present a novel way of reusing existing neural networks that have been pretrained on the task of object recognition in models of fixation prediction. Using the well-known network of Krizhevsky et al. (2012), we come up with a new saliency model that significantly outperforms all state-of-the-art models on the MIT Saliency Benchmark. We show that the structure of this network allows new insights in the psychophysics of fixation selection and potentially their neural implementation. To train our network, we build on recent work on the modeling of saliency as point processes.
1 METHODS
Deep Gaze I reuses spatial responses from a pretrained Krizhevsky network, combines and smooths selected features, adds a center-bias prior, and converts the result into fixation probabilities. The model is trained on fixation data with sparse weights and evaluated on held-out images using shuffled AUC.
- Feature extraction: The model removes the Krizhevsky network’s fully connected layers and uses spatial responses from selected convolutional, rectifier, pooling, and normalization layers.This preserves spatially located features while allowing arbitrary image input sizes after downsampling.
- Feature extraction: Selected response maps are rescaled to the largest layer’s size and individually normalized, producing up to 3712 responses per image location.The normalized features are then supplied to the saliency model.
- Saliency mapping: At each image location, weighted feature responses are linearly combined and convolved with a Gaussian kernel to produce the saliency map.The kernel width is controlled by σ.
- Probability model: The saliency prediction is linearly combined with a fixed center-bias prediction, then passed through a softmax to yield a probability distribution over the image.The center-bias term accounts for fixation locations being strongly biased toward the image center.
- Training and evaluation: The model uses ℓ1 regularization to encourage sparse weights and is trained on fixation locations by minimizing a cost function.Training used roughly half of MIT1003, while generalization was checked on 540 held-out images using shuffled AUC.
2 RESULTS
Deep Gaze I evaluates fixation prediction with information-theoretic and MIT benchmark measures, substantially outperforming existing models. Layer comparisons and feature analyses indicate that top-layer convolutional features generalize best and capture high-level image structure.
- 2.1 PERFORMANCE RESULTS: Log-likelihood compares probabilistic fixation models, while information gain measures efficiency relative to an image-independent baseline.One bit/fix means saving one bit per fixation versus the baseline.
- 2.1 PERFORMANCE RESULTS: The explainable-information ratio compares a model’s information gain with the gold standard’s possible gain from image-based fixation structure.The baseline models the image-independent prior, while the cross-validated gold standard captures spatial structure attributable to the image.
- 2.1 PERFORMANCE RESULTS: 56% of explainable information gain was explained by Deep Gaze I, compared with 34% for eDN.Figure 3 compares these percentages with influential saliency models after postprocessing the other models for nonlinearity, center bias, and blurring.
- 2.2 RESULTS ON MIT SALIENCY BENCHMARK: 71.69% shuffled AUC exceeded AWS at 67.90%, while 84.40% AUC exceeded BMS at 82.57% on the MIT benchmark.Shuffled AUC uses a uniform prior for Deep Gaze I, whereas AUC uses a nonparametric prior; the metrics therefore use different saliency-map definitions.
- 2.3 LAYER SELECTION: The final model used only layer-5 convolution outputs, and this architecture generalized best to test images in shuffled AUC comparisons.The top layer was selected because it was expected to contain high-level influences, while other units mainly provide nonlinear transformations for subsequent convolutions.
- 2.3 LAYER SELECTION: Using all layers reached 83% information gain explained on test-subject fixations within training images, suggesting overfitting to limited object variety rather than subjects.The authors therefore expect larger training image sets to improve performance.
- 2.4 ANALYSIS OF USED FEATURES: The ten most weighted layer-5 features responded to faces, text, and context-dependent pop-out patterns.The analysis used the nine highest- or lowest-response patches for each selected feature, depending on the feature’s weight sign.
- 2.4 ANALYSIS OF USED FEATURES: Deep Gaze I captured high-level objects such as faces and text, as well as more abstract concepts such as pop-out.The pop-out response depended strongly on image context and was not equivalent to simple luminance or color contrast.
3 DISCUSSION
Deep Gaze I substantially improves fixation prediction by using ImageNet-trained features that capture high-level influences such as faces, text, and popout. The authors frame these features as useful both for prediction and for studying human attention.
- 56% explained information gain versus 34% for state-of-the-art models.
- The model captures high-level influences including faces, text, and abstract popout features.
- Direct model performance indicates that deep layers provide a sufficiently rich feature space for fixation prediction through simple linear feature combinations.
- Convolution responses were the most informative network features for predicting fixations.
- The paper identifies ImageNet features as, to the authors’ knowledge, the first use of such features for predicting behaviour.
- Extending attention-based neural networks with Krizhevsky features is proposed as a direction for future research.
4 CONCLUSIONS
The paper shows that pretrained deep convolutional features and point-process maximum-likelihood learning can substantially improve saliency prediction. It presents this combination as a foundation for more predictive saliency models and further psychophysical and neuroscience investigations.
- Deep convolutional networks trained on computer-vision tasks boost saliency prediction.
- Figure 7 relates selected features to image patches, their full-image response maps, and the locations of response maxima.
- The approach is presented as enabling saliency models with high predictive power and implications for psychophysics and neuroscience.
- The saliency model is optimized by maximizing the log-likelihood of a point process.
- Combining ImageNet recognition features with principled maximum-likelihood learning is proposed as the basis for a Deep Gaze program targeting explainable fixation structure.
6 SUPPLEMENTARY MATERIAL
The regularization parameter has only a small effect on test performance when the final convolutional layer is used. Its value was selected by grid search.
- The regularization parameter λ was chosen using grid search.
- Changing λ had a visible but very small effect on test performance.