PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations

Srinivas S. S. Kruthiventi, Kumar Ayush, R. Venkatesh Babu

arXiv:1510.02927v1cs.CV

TL;DR

Visual saliency prediction must account for both low-level stimuli and high-level semantics, while fully convolutional networks ordinarily cannot model location-dependent patterns. DeepFix addresses these issues with an end-to-end, fully convolutional architecture combining multiscale features, large receptive fields, and Location Biased Convolutional layers. It achieves state-of-the-art results on MIT300 and CAT2000 and outperforms prior methods by a considerable margin.

  • Problem

    Saliency prediction is difficult because hand-crafted features inadequately capture diverse low- and high-level visual cues, while ordinary fully convolutional networks cannot learn location-dependent patterns such as centre-bias.

  • Method

    DeepFix is a fully convolutional neural network trained end-to-end to predict pixel-wise saliency, using multiscale inception-style convolutions, large receptive fields, and Location Biased Convolutional layers.

  • Results

    DeepFix achieves state-of-the-art results on MIT300 and CAT2000 and outperforms other methods by a considerable margin.

  • Takeaways & Limitations

    Combining hierarchical semantic features, global context, and learned location dependence provides an effective approach to predicting human eye fixations.

Abstract

from arXiv · show

Understanding and predicting the human visual attentional mechanism is an active area of research in the fields of neuroscience and computer vision. In this work, we propose DeepFix, a first-of-its-kind fully convolutional neural network for accurate saliency prediction. Unlike classical works which characterize the saliency map using various hand-crafted features, our model automatically learns features in a hierarchical fashion and predicts saliency map in an end-to-end manner. DeepFix is designed to capture semantics at multiple scales while taking global context into account using network layers with very large receptive fields. Generally, fully convolutional nets are spatially invariant which prevents them from modeling location dependent patterns (e.g. centre-bias). Our network overcomes this limitation by incorporating a novel Location Biased Convolutional layer. We evaluate our model on two challenging eye fixation datasets -- MIT300, CAT2000 and show that it outperforms other recent approaches by a significant margin.

I. INTRODUCTION

Visual saliency prediction models human eye fixations, but hand-crafted features struggle with diverse low- and high-level cues. DeepFix addresses this with an end-to-end fully convolutional network that captures multiscale semantics, global context, and location-dependent patterns.

  • Motivation: Visual attention directs gaze toward salient and informative scene regions through bottom-up visual cues and top-down factors such as prior knowledge and task goals.Salient stimuli may include bright colors, distinctive textures, familiar faces, text, or motion.
  • Motivation: Classical saliency models use multiscale low-level features but inadequately capture high-level semantic aspects such as faces and text.The diversity of possible low- and high-level causes makes effective hand-crafted feature design difficult.
  • DeepFix: DeepFix predicts pixel-wise saliency maps from whole images end-to-end without hand-crafted features.The network is a fully convolutional model with 20 convolutional layers designed to predict human eye fixations.
  • DeepFix: Inception-style parallel kernels capture object semantics at multiple scales, while very large receptive fields capture global scene context.The large-receptive-field layers replace densely connected inner-product layers toward the network’s end.
  • DeepFix: Location Biased Convolutional layers let DeepFix learn location-dependent patterns such as the centre-bias that ordinary fully convolutional nets cannot model.Fully convolutional layers are otherwise location invariant under spatial shifts of the input.

II. RELATED WORK

Earlier saliency methods drew on biologically inspired, information-theoretic, or pretrained CNN features. DeepFix instead emphasizes a deep, task-specific network that processes whole images while modeling multiple scales and global context.

  • Classical approaches: Classical saliency research used biologically inspired multiscale feature representations derived from visual-attention theories.These approaches were motivated by neuroscience and psychology and aggregated feature maps to model selective attention.
  • Information-theoretic approaches: Information-theoretic approaches treated statistical rarity or local self-information as important factors in predicting attention allocation.These methods provide an alternative perspective to neuroscience- and psychology-driven saliency models.
  • Deep learning approaches: Some recent methods use features from pretrained CNNs, while others train convolutional networks specifically for saliency prediction.Deep learning progress has enabled hierarchical, semantically rich visual representations.
  • DeepFix: DeepFix argues that a deep, task-specific convolutional network trained end-to-end can outperform approaches using off-the-shelf CNN features.Its architecture combines substantial depth with saliency-specific training.
  • Comparison with prior CNNs: Multiresolution-CNN captures scale-specific information with an ensemble of patch-based CNNs but fails to capture global context because it presents isolated patches.DeepFix processes the image as a whole and captures multiple scales through inception-style convolution blocks.

III. PROPOSED APPROACH

DeepFix is a fully convolutional network that predicts pixel-wise saliency maps end-to-end. Its architecture combines multi-scale semantic processing, global context, and enlarged receptive fields while preserving spatial resolution.

  • DeepFix predicts pixel-wise saliency values from an image in an end-to-end manner.
  • The network uses five VGG-like convolution blocks with small kernels and later layers that retain reduced spatial dimensions.The first three blocks use stride-2 max-pooling, producing W/8 × H/8 spatial dimensions that are retained afterward.
  • Kernels with holes enlarge receptive fields without increasing memory footprint.The fifth block uses hole size 2, giving 3×3 filters a 5×5 receptive field; later layers use hole size 6 for very large receptive fields.
  • Inception modules capture object-level semantics at multiple scales using parallel convolutions with different kernel sizes.
  • The final saliency map is upsampled from W/8 × H/8 to the original image resolution using bicubic interpolation.

B. Centre-bias in Eye Fixations

Human eye fixations show a centre-bias, with many fixations occurring in central image regions. DeepFix addresses this spatial tendency by learning location-dependent patterns through Location Biased Convolutional layers rather than adding an explicit centre-bias term.

  • B. Centre-bias in Eye Fixations: A significant number of human eye fixations fall in the central region of an image.
  • B. Centre-bias in Eye Fixations: Centre-bias can reflect photographer positioning, viewing strategy, and image uninterestingness.
  • B. Centre-bias in Eye Fixations: Earlier saliency models used explicit centre-bias terms to account for spatial biases in eye fixations.
  • B. Centre-bias in Eye Fixations: DeepFix instead learns location-dependent patterns implicitly through a novel Location Biased Convolutional layer.

1) Location Biased Convolutional (LBC) layer:

The Location Biased Convolutional layer introduces spatial information into an otherwise location-invariant convolution. It does so by concatenating fixed location-specific features with the input, preserving shared weights while enabling location-dependent responses.

  • 1) Location Biased Convolutional (LBC) layer:: Fully convolutional layers cannot learn location-dependent patterns because their responses are location invariant.
  • 1) Location Biased Convolutional (LBC) layer:: The LBC layer concatenates a data-independent location feature L(x, y) with the input feature I(x, y).
  • 1) Location Biased Convolutional (LBC) layer:: This design adds parameters linearly with the dimension of L(x, y), independently of the input blob’s spatial size.
  • 1) Location Biased Convolutional (LBC) layer:: The location-specific features remain fixed while filter weights are learned to combine stimulus and location information for saliency prediction.
  • 1) Location Biased Convolutional (LBC) layer:: DeepFix uses 16 location-specific feature components derived from Gaussian maps with different horizontal and vertical variances.

A. Training DeepFix

DeepFix training proceeds in two stages: broad pretraining on SALICON followed by training on eye-fixation datasets. The paper uses SALICON, CAT2000, and MIT1003 for training or validation, while MIT300 is held out for evaluation.

  • A. Training DeepFix: VGG-16 initialization and separate learning rates are used for pretrained and randomly initialized layers.The initial learning rates are 2 × 10^-4 for VGG-initialized layers and 2 × 10^-3 for the remaining layers.
  • A. Training DeepFix: DeepFix trains in two stages, first on SALICON and then on smaller datasets with eye-fixation ground truth.
  • A. Training DeepFix: SALICON provides 15,000 training and validation images during the first training stage.
  • A. Training DeepFix: CAT2000 contains 4,000 images across 20 categories, with 1,800 images used for second-stage training.
  • A. Training DeepFix: MIT1003 contributes 900 images to second-stage training and 103 images to validation for evaluation on MIT300.
  • A. Training DeepFix: MIT300 contains 300 natural images whose held-out ground truth is used to evaluate DeepFix on the MIT saliency benchmark.

B. Evaluation

DeepFix is evaluated on held-out MIT300 and CAT2000 test sets using multiple saliency metrics and qualitative comparisons. The evaluation includes distance, fixation-based, correlation, similarity, and AUC measures.

  • B. Evaluation: The MIT Saliency Benchmark evaluates models on EMD, NSS, Similarity, CC, AUC-Judd, AUC-Borji, and shuffled-AUC.The benchmark uses multiple metrics because no single metric fully evaluates saliency models.
  • B. Evaluation: Ground-truth and predicted saliency maps are compared using Earth Mover’s Distance, which measures the cost of transforming one 2D distribution into the other.Lower EMD indicates a better estimated saliency map.
  • B. Evaluation: NSS measures the mean normalized saliency assigned at human eye-fixation locations.The normalized saliency map has zero mean and unit standard deviation.
  • B. Evaluation: CC measures the linear relationship between the ground-truth and estimated saliency maps.A score near +1 indicates an almost perfect linear relationship.
  • B. Evaluation: The qualitative comparison presents images alongside GBVS, eDN, BMS, Mr-CNN, DeepFix, and ground-truth maps.Figure 7 shows qualitative results on MIT1003 validation images.
  • B. Evaluation: The Similarity metric sums the minimum values at corresponding pixels between normalized probability distributions.Its formulation compares normalized estimated and ground-truth maps.

C. Results

DeepFix produces qualitatively plausible saliency maps and achieves state-of-the-art quantitative results on CAT2000 and MIT300. Its performance is attributed to deep, multiscale, context-aware representations and learned location patterns, although gains are smaller on AUC metrics.

  • C. Results: DeepFix captures saliency from low-level cues such as colour and shape and high-level cues such as text, animal faces, and humans.The qualitative results show that the model can weigh these factors according to their relative importance.
  • C. Results: For images without striking salient regions, DeepFix uses a learned centre-directed fixation pattern to predict saliency.The model also prioritizes faces over bright clothing in one example, indicating image-dependent weighting of cues.
  • C. Results: The qualitative test-set figures use MIT1003 validation images because ground-truth saliency maps for the test sets are not publicly available.This limits the qualitative visualization to validation data.
  • C. Results: The architecture’s performance is attributed to its depth, multiscale semantic filters, global context, and implicit learning of location-dependent fixation patterns.These design elements are linked to richer semantic representations and the reported qualitative performance.
  • C. Results: DeepFix achieves state-of-the-art results on both CAT2000 and MIT300, outperforming other methods by a large margin on most metrics.The strongest gains are reported for NSS, EMD, CC, and Similarity.
  • C. Results: AUC metrics show less comparable improvement because false positives receive relatively weak penalties and blurred saliency maps can score well.Shuffled AUC can also penalize DeepFix for learning the centre-bias: one example scores 0.68 versus 0.70 for eDN.

D. Ablation Analysis

The ablation analysis attributes improved saliency prediction to implicitly learning location-dependent patterns with LBC layers, while benchmark gains are strongest on non-AUC metrics.

  • LBC ablation: Implicitly learning location-dependent patterns with LBC layers results in better saliency prediction than removing LBC layers or adding an explicit centre bias.The comparison uses DF-No-LBC, DF-Explicit-CB, and DF-LBC on the MIT1003 validation set.
  • Evaluation caveat: SALICON results used for comparison were taken from the MIT Saliency Leaderboard, although the corresponding model and paper were not publicly available when written.This qualifies the comparability and reproducibility of that baseline.
  • Metric analysis: AUC metrics show smaller gains because they primarily reward true positives and are less sensitive to false positives.The paper illustrates this limitation with a blurred prediction receiving the same shuffled AUC as a close prediction.

V. CONCLUSION

DeepFix is a fully convolutional network for predicting human eye fixations using multi-scale semantic features, large receptive fields, and location-biased convolutions. The combined design outperforms other state-of-the-art methods and LBC outperforms explicit bias addition in ablation analysis.

  • V. CONCLUSION: DeepFix predicts human eye fixations with a fully convolutional neural network using inception modules and filters with holes.These components capture semantic features at multiple scales and global context through large receptive fields.
  • V. CONCLUSION: Location Biased Convolutional filters enable the network to learn location-dependent patterns.This addresses the limitation of location-invariant fully convolutional networks.
  • V. CONCLUSION: DeepFix outperforms traditional explicit bias addition and other state-of-the-art methods by a considerable margin.The explicit-bias comparison is supported by ablation analysis, while the broader comparison concerns the combined design.
Loading 1510.02927v1…