Source-linked AI summary
DeepGaze II: Reading fixations from deep features trained on object recognition
Matthias Kümmerer, Thomas S. A. Wallis, Matthias Bethge
TL;DR
Saliency prediction seeks to model where people fixate in images, but typical datasets are small and prior VGG-based gains left unclear whether retraining was necessary. DeepGaze II uses fixed object-recognition VGG-19 features with a trained nonlinear readout and probabilistic fixation modelling. It explains 87% of explainable information gain and ranks first on AUC and shuffled AUC on held-out MIT300, while examples indicate weaknesses on difficult text.
Problem
Saliency prediction uses fixation locations to study visually guided behaviour, while small datasets and prior VGG-based gains left the value of retraining unclear.
Method
DeepGaze II feeds fixed pretrained VGG-19 features into a nonlinear readout, trains it probabilistically, and models centre bias as an explicit prior.
Results
DeepGaze II explains 87% of explainable information gain and ranks first on AUC and shuffled AUC on the held-out MIT300 benchmark.
Takeaways & Limitations
The results support using object-recognition deep features as a versatile feature space for saliency prediction without retraining VGG features.
Takeaways & Limitations
The model can miss low-contrast, partially occluded, or perspective-distorted text, suggesting a need for better training on text.
Abstract
from arXiv · showhide
Here we present DeepGaze II, a model that predicts where people look in images. The model uses the features from the VGG-19 deep neural network trained to identify objects in images. Contrary to other saliency models that use deep features, here we use the VGG features for saliency prediction with no additional fine-tuning (rather, a few readout layers are trained on top of the VGG features to predict saliency). The model is therefore a strong test of transfer learning. After conservative cross-validation, DeepGaze II explains about 87% of the explainable information gain in the patterns of fixations and achieves top performance in area under the curve metrics on the MIT300 hold-out benchmark. These results corroborate the finding from DeepGaze I (which explained 56% of the explainable information gain), that deep features trained on object recognition provide a versatile feature space for performing related visual tasks. We explore the factors that contribute to this success and present several informative image examples. A web service is available to compute model predictions at http://deepgaze.bethgelab.org.
1 Introduction
Saliency prediction asks where observers fixate in an image, and deep transfer-learning models improved this task. DeepGaze II tests whether fixed VGG-19 features can support saliency prediction without retraining those features.
- Motivation: Saliency prediction concerns predicting fixation locations from the image being viewed.The paper frames this as an important component of understanding visually guided behaviour.
- Background: Deep neural networks markedly improved saliency prediction after 2014.
- Background: DeepGaze I used ImageNet-trained AlexNet features for saliency prediction, exemplifying transfer learning.Transfer learning reuses features learned on one task for a second task, with or without fine-tuning.
- Research question: VGG-based models outperformed DeepGaze I, leaving open whether retraining or VGG features themselves explained the improvement.DeepGaze II tests the latter possibility by using VGG features without retraining them.
- Contribution: DeepGaze II combines fixed pretrained VGG-19 features with a nonlinear readout, probabilistic log-likelihood training, and an explicit centre-bias prior.
2 Methods
DeepGaze II maps fixed VGG-19 feature activations to probabilistic fixation predictions through a nonlinear readout, Gaussian smoothing, and an explicit centre-bias prior. The readout is pretrained on SALICON, fine-tuned with image-level cross-validation on MIT1003, and evaluated on held-out images and MIT300.
- Probabilistic formulation: DeepGaze II uses probabilistic modelling and log-likelihood to train and evaluate fixation predictions.Information gain explained is used for in-house evaluation, alongside AUC and shuffled AUC from the MIT benchmark.
- Model architecture: A subset of fixed VGG feature maps feeds a four-layer 1×1-convolution readout network that produces a saliency map.The pointwise architecture can represent nonlinear combinations of VGG features without learning substantially different spatial features.
- Model architecture: The saliency map is Gaussian-blurred, combined with an explicit centre-bias prior, and converted to an image probability distribution with a softmax.The centre bias is modelled as a prior so alternative task-dependent centre biases can be substituted.
- Implementation: The implementation uses Caffe for VGG and Lasagne and Theano for the remaining model components.
- Cross-validation: Ten models are fine-tuned on nine-tenths of MIT1003, each using a different tenth for stopping and image-level cross-validation.For MIT1003 evaluation, each image uses a model that was not trained on that image; other datasets use a mixture of the ten models.
- Training: The readout network is pretrained on 10000 SALICON images, with MIT1003 used to determine when training should stop.SALICON supplies pseudofixations, while MIT1003 performance is monitored for overfitting.
3 Results
DeepGaze II performs strongly on saliency prediction, explaining substantially more information gain than earlier models and achieving top MIT benchmark performance. Image-level analyses show close agreement with the gold standard, while examples and ablations identify strengths, limitations, and likely sources of improvement.
- 3.1 Information gain explained: 87% of the explainable information gain was accounted for by DeepGaze II, compared with 56% for DeepGaze I.Information gain explained is measured relative to the gold standard and baseline model.
- 3.1 Information gain explained: DeepGaze II was the only evaluated model with no images showing negative information gain relative to the centre-bias baseline.Its image-level predictions clustered closer to the gold standard than those of AIM, eDN, and DeepGaze I.
- 3.2 MIT saliency benchmark: DeepGaze II beat the nearest competitors SALICON and DeepFix by 1% on the MIT300 benchmark, with a larger margin for shuffled AUC.The larger shuffled-AUC margin may partly reflect whether competing models excluded centre bias.
- 3.3 Model prediction examples: In low-performing examples, DeepGaze II captured fixations on bakery goods but missed low-contrast, partially occluded store text and perspective-distorted signage.The authors identify better training on text as one potential avenue for improvement.
- 3.4 Reasons for improvement over DeepGaze I: The largest improvement over DeepGaze I came from replacing AlexNet with pretrained VGG features, while SALICON pretraining produced a marginal improvement.Combining SALICON pretraining with VGG features yielded the largest intermediate-model improvement; the nonlinear readout slightly decreased performance independently.
4 Discussion
DeepGaze II achieves state-of-the-art saliency prediction using transfer learning from VGG-19 features, while its probabilistic evaluation shows strong information-gain performance and top MIT300 AUC rankings. The model’s design may improve transfer success by limiting retraining and overfitting, and its predictions are available through a web service.
- DeepGaze II uses transfer learning from VGG-19 to achieve state-of-the-art saliency-prediction performance.
- 87% of explainable information gain is accounted for by DeepGaze II, while some high-performing models were excluded because their code was unavailable.
- DeepGaze II ranks first on the held-out MIT300 benchmark for AUC and shuffled AUC, although its AUC advantage over other models is modest.
- Not retraining VGG features reduces learned parameters and may lower overfitting, making the transfer test more stringent.
- A web service computes DeepGaze II predictions for arbitrary images.