PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model

Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, Rita Cucchiara

arXiv:1611.09571v4cs.CV

TL;DR

Saliency prediction must model human fixation patterns beyond the limitations of hand-crafted features and feed-forward approaches. The paper introduces SAM, combining attentive ConvLSTM refinement, resolution-preserving CNN features, and learned Gaussian priors. Extensive evaluation reports state-of-the-art performance across major saliency benchmarks and validates the contributions of its components.

  • Problem

    Traditional features fail to capture the wide variety of causes shaping saliency maps, while deep saliency approaches had not incorporated machine attention models.

  • Method

    SAM uses an Attentive ConvLSTM to refine saliency features, modified CNNs to preserve spatial detail, and trainable priors to model fixation bias.

  • Results

    The proposed solution achieves state-of-the-art results on major saliency datasets, with a large-margin improvement on SALICON and competitive results on MIT300 and CAT2000.

  • Takeaways & Limitations

    Attentive refinement, learned priors, and resolution-preserving feature extraction each contribute to improved saliency predictions across evaluated datasets.

Abstract

from arXiv · show

Data-driven saliency has recently gained a lot of attention thanks to the use of Convolutional Neural Networks for predicting gaze fixations. In this paper we go beyond standard approaches to saliency prediction, in which gaze maps are computed with a feed-forward network, and present a novel model which can predict accurate saliency maps by incorporating neural attentive mechanisms. The core of our solution is a Convolutional LSTM that focuses on the most salient regions of the input image to iteratively refine the predicted saliency map. Additionally, to tackle the center bias typical of human eye fixations, our model can learn a set of prior maps generated with Gaussian functions. We show, through an extensive evaluation, that the proposed architecture outperforms the current state of the art on public saliency prediction datasets. We further study the contribution of each key component to demonstrate their robustness on different scenarios.

I. INTRODUCTION

Saliency prediction seeks to model where people look, but hand-crafted features and feed-forward deep models leave room for machine-attentive refinement. SAM addresses this gap with an attentive ConvLSTM, resolution-preserving CNN features, and learned fixation priors, achieving strong benchmark results.

  • Traditional saliency features capture low-level cues and semantic concepts but fail to represent the wide variety of causes shaping visual saliency maps.
  • Deep networks improved saliency prediction, yet prior approaches had not incorporated machine attention models into the task.
  • SAM uses an Attentive ConvLSTM that sequentially focuses on spatial locations to iteratively refine saliency features over an image.
  • The architecture modifies VGG-16 and ResNet-50 to reduce CNN downscaling and preserve spatial resolution for saliency prediction.
  • The network learns fixation bias automatically through trainable priors rather than manually integrating that information.
  • The model is evaluated on SALICON, MIT300, and CAT2000, with a large-margin state-of-the-art improvement on SALICON and competitive generalization on the other datasets.

II. RELATED WORK

Related work progresses from hand-crafted low-level and semantic cues to deep saliency networks, while SAM distinguishes itself through attentive refinement, learned Gaussian priors, and a multi-metric loss.

  • Early saliency methods combined low-level feature maps such as color, intensity, orientation, contrast, and texture into global maps.
  • Later approaches added semantic detectors for concepts including faces, persons, cars, and horizons.
  • Deep learning substantially improved saliency prediction through convolutional architectures and large eye-fixation datasets such as SALICON.
  • Prior-map methods used learned or predefined centered priors, whereas SAM learns multiple Gaussian prior parameters end-to-end from data.
  • SAM combines an attentive mechanism, learned priors, and a balanced loss over different saliency metrics, features absent from the compared methods.
  • Activation-explanation methods based on backpropagation differ from saliency prediction because they do not aim to replicate human fixations.

B. Salient Object Detection

Salient object detection identifies binary salient-object regions, whereas saliency prediction estimates a density map of human eye fixations. The paper’s model instead uses a convolutional recurrent architecture to iteratively process saliency features and refine predictions.

  • Salient object detection produces a binary map indicating salient-object presence, unlike saliency prediction, which predicts a density map of eye fixations.
  • The proposed recurrent network is convolutional and processes saliency features iteratively rather than generating binary predictions for spatially transformed image regions.
  • SAM’s attentive convolutional model recurrently attends to different tensor regions while processing saliency features.
  • Predictions are progressively refined as ConvLSTM outputs at different timesteps modify the initial CNN saliency map.

A. Attentive Convolutional LSTM

The Attentive ConvLSTM extends LSTM processing from temporal vector sequences to spatial image features. It computes spatial attention from the input and previous hidden state, then iteratively updates feature maps to refine saliency predictions.

  • The model substitutes LSTM dot products with convolutional operations so recurrent processing can operate on spatial features.The resulting gates, memory states, and hidden states are 3-D tensors with 512 channels.
  • The LSTM receives image-derived feature stacks and produces refined feature maps for the learned-prior module through sequential internal-state updates.Three sigmoid gates drive the updates.
  • At each iteration, attention is computed by convolving the previous hidden state with the input, applying tanh, and producing a one-channel map.
  • A softmax converts the attention output into a normalized spatial map, which is multiplied element-wise across every feature-map channel.
  • ConvLSTM predictions become progressively more accurate by modifying the initial CNN saliency map across timesteps.

B. Learned Priors

The model addresses human gaze’s center bias with learned Gaussian prior maps rather than fixed priors. Their parameters are learned and concatenated with ConvLSTM features before further convolutional processing.

  • Human observers’ gazes tend to be biased toward image centers, partly because photographers place objects of interest there.
  • Unlike methods using predefined priors, the network learns its own priors as 2-D Gaussian functions with freely learned means and covariance parameters.
  • The network learns N Gaussian functions, using N = 16 in the experiments, to generate relative prior maps.
  • Learned prior maps are concatenated with the 512-channel refined feature tensor, producing 528 channels before a 512-filter convolution.

C. Dilated Convolutional Network

The model uses dilated convolutions to preserve higher-resolution saliency features while retaining convolutional filter scale and parameter count. Modified VGG-16 and ResNet-50 backbones reduce feature-map downscaling and support final saliency-map reconstruction.

  • CNN feature extraction can severely rescale images and worsen saliency accuracy, motivating a higher-resolution strategy that preserves filter scale and parameter count.The approach also enables use of pre-trained weights and reduces the need for convolutional-layer fine-tuning.
  • The original architectures repeatedly downscale feature maps through max-pooling and strided blocks, producing an 8 × 10 output from a 240 × 320 input.
  • VGG-16 removes its last max-pooling layer and applies dilation after reducing the stride of the preceding pooling layer.
  • ResNet-50 removes strides and introduces dilated convolutions in its final two blocks, using hole sizes 1 and 3.
  • The modified networks rescale saliency maps by a factor of 8 instead of 32 in the original VGG-16 and ResNet-50 models.
  • Dilated convolutions in prior layers provide large receptive fields for modeling an object’s saliency relative to its neighborhood.The specified layers use kernel size 5, hole size 3, and a 17 × 17 receptive field.
  • The final one-filter convolution produces the saliency map, which is bilinearly upsampled to the original image dimensions.

D. Loss function

The paper defines a composite loss that combines NSS, CC, and KL-Div to capture multiple saliency quality factors and balance evaluation metrics.

  • Loss design: The overall loss is a linear combination of NSS, CC, and KL-Div, weighted by scalars that balance their contributions.The predicted saliency map, density ground truth, and binary fixation map provide the loss inputs.
  • Component metrics: NSS quantifies normalized saliency values at eye-fixation locations.The predicted map is normalized to zero mean and unit standard deviation.
  • Component metrics: CC measures the linear relationship between predicted saliency and ground-truth density maps as Pearson’s correlation coefficient.It treats both maps as random variables.
  • Component metrics: KL-Div measures information loss when the predicted saliency distribution approximates the ground-truth density distribution, with lower values indicating better approximation.A regularization constant is included in its formulation.
  • Validation: Experiments compare the composite loss with individual metrics and alternative probability distances to assess balance across evaluation metrics.The evaluation includes single-metric losses and traditional probability-distance losses.

IV. EXPERIMENTAL SETUP

The experimental setup uses public saliency datasets and compares the proposed loss with its individual components across multiple datasets, models, and metrics.

  • Setup: The evaluation section describes the datasets, metrics, and implementation details used to assess the proposed model.These elements define the experimental setup.
  • Datasets: SALICON is described as the largest available saliency dataset, containing 10,000 training, 5,000 validation, and 5,000 testing images.Its images are taken from Microsoft COCO, with eye fixations simulated by mouse movements.
  • Loss comparison: Figure 5 compares the proposed loss with its individual components for SAM-VGG and SAM-ResNet on SALICON, MIT1003, and CAT2000 validation sets.Rows correspond to CC, sAUC, AUC, and NSS, while colors represent the loss functions.

B. Evaluation Metrics

The paper evaluates saliency predictions with complementary location-based and distribution-based metrics, while describing dataset preprocessing and training choices used in the experiments.

  • Metric categories: Saliency metrics are categorized as location-based, which evaluate discrete fixation locations, or distribution-based, which compare continuous maps.This distinction reflects different ground-truth representations.
  • Location-based metrics: AUC, sAUC, and NSS are widely used location-based metrics, with sAUC designed to penalize center-bias exploitation.AUC emphasizes predictions at fixated locations, while NSS evaluates saliency at fixation points.
  • Distribution-based metrics: CC, SIM, and EMD are common distribution-based metrics that compare predicted and ground-truth saliency distributions.CC treats false positives and negatives symmetrically, SIM measures distribution intersection, and EMD penalizes spatially distant false positives.
  • Preprocessing: MIT1003 and MIT300 images are padded to a 4:3 aspect ratio before resizing, whereas CAT2000 images are resized to 180 × 320.The preprocessing accounts for differing image sizes across datasets.
  • Preprocessing: Predictions are blurred with a Gaussian filter whose standard deviation is set to 7 after validation.The same postprocessing choice is applied across datasets.
  • Training: Training uses RMSprop with batch size 10, loss weights α = −1, β = −2, and γ = 10, and learning-rate reductions by a factor of 10.The learning rate is reduced every two epochs for ResNet and every three epochs for VGG.

V. EXPERIMENTAL EVALUATION

The experimental evaluation combines loss-function comparisons, component ablations, and qualitative analyses across multiple saliency datasets. Results support contributions from the combined loss, dilated convolutions, attentive ConvLSTM, learned priors, and the complete architecture.

  • Loss-function analysis: The combined loss achieves competitive results across all metrics, unlike individual or traditional losses that perform unevenly across metrics.On SALICON, single KL-Div, CC, or NSS losses favor their corresponding metrics while degrading others; the combined loss balances them.
  • Model ablation analysis: The ablation study evaluates a plain CNN, dilated convolutions, ConvLSTM, learned priors, and the complete model on three validation datasets.Both SAM-VGG and SAM-ResNet variants are tested, and each proposed component contributes to final performance.
  • Model ablation analysis: The complete architecture further improves the ResNet-based configurations by 0.4% and 0.5% after adding all proposed components.The reported ResNet baseline is 0.771 CC, with gains from dilated convolutions, Attentive ConvLSTM, and learned priors.
  • Model ablation analysis: VGG-based and ResNet-based networks achieve almost the same performance, allowing speed and memory needs to guide model choice.The authors report that choosing either backbone does not considerably affect prediction performance.
  • Qualitative analysis: Qualitative predictions become consistently more similar to ground truth as the key components are added.The comparison uses SAM-ResNet and variants containing only selected components.

C. Contribution of the attentive model and learned priors

The attentive model refines saliency predictions over iterations, while learned Gaussian priors improve center-bias modeling. Benchmark comparisons show strong performance on SALICON and CAT2000, with limited AUC gains and near-best MIT300 results.

  • Attentive model: Refinement improves performance through four attentive ConvLSTM iterations, after which CC, sAUC, and AUC saturate and NSS slightly decreases.The analysis reports no further significant improvements for t > 4.
  • Learned priors: Multiple learned Gaussian priors outperform learning an entire unconstrained prior map according to all evaluated metrics.The comparison uses validation sets from SALICON, MIT1003, and CAT2000.
  • State-of-the-art comparison: On SALICON, SAM-ResNet surpasses the other ResNet-based model by 1.5% in NSS, 1.3% in CC, and 0.4% in sAUC.SAM-VGG exceeds all other VGG-based methods by 12.7% in NSS and 5.6% in CC.
  • State-of-the-art comparison: The model reaches first place in the LSUN Challenge 2017 saliency prediction task.This result is reported alongside the test-set comparisons.
  • State-of-the-art comparison: The method achieves state-of-the-art results on CAT2000 except sAUC and results very close to the best on MIT300.CAT2000 gains are especially reported for SIM, CC, NSS, and EMD.
  • Evaluation caveat: AUC metrics show limited gains because they emphasize true positives without substantially penalizing false positives.The authors note that hazy or blurred saliency maps can therefore achieve high AUC values despite differing visually from ground truth.
  • Qualitative comparison: Qualitative examples show high saliency on people, faces, objects, and predominant cues, including scenes without strong salient regions.The model can also qualitatively infer relative importance among different people in the same scene.

VI. CONCLUSION

The paper concludes with a Saliency Attentive Model centered on an Attentive ConvLSTM for sequentially enhancing fixation predictions. Learned priors and resolution-preserving CNN extensions support its state-of-the-art results and reproducibility.

  • Conclusion: The Attentive ConvLSTM sequentially enhances saliency predictions for human eye fixations on natural images.Its image-refinement formulation could potentially extend to other tasks where refinement is useful.
  • Conclusion: Multiple learned priors capture an important property of human gazes, while the architecture addresses CNN downscaling effects.The paper validates each component through extensive evaluation.
  • Conclusion: The model achieves state-of-the-art results on two important saliency-prediction datasets and releases source code and pretrained models.The conclusion frames the release as supporting further research efforts.
Loading 1611.09571v4…