PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection

Nian Liu, Junwei Han

arXiv:1610.01708v1cs.CV

TL;DR

Traditional saliency detection relies on hand-designed features and local or global contrast, motivating a model that better integrates spatial and scene context. DSCLRCN learns local features with CNNs, uses DSLSTM for global spatial interactions, and embeds scene features through DSCLSTM. It achieves state-of-the-art performance on two benchmark datasets, while ablations report that DSCLSTM significantly improves saliency detection performance.

  • Problem

    Traditional saliency models rely on hand-designed image features and local or global contrast, while many deep models infer saliency mainly from local contexts.

  • Method

    DSCLRCN learns local features with pretrained CNNs, uses DSLSTM for global spatial context, and embeds scene features into DSCLSTM for scene modulation.

  • Results

    DSCLRCN achieves state-of-the-art performance on two benchmark datasets, and DSCLSTM significantly improves saliency detection performance.

  • Takeaways & Limitations

    The results support incorporating both global spatial interconnections and scene context modulation in computational saliency models.

Abstract

from arXiv · show

Traditional saliency models usually adopt hand-crafted image features and human-designed mechanisms to calculate local or global contrast. In this paper, we propose a novel computational saliency model, i.e., deep spatial contextual long-term recurrent convolutional network (DSCLRCN) to predict where people looks in natural scenes. DSCLRCN first automatically learns saliency related local features on each image location in parallel. Then, in contrast with most other deep network based saliency models which infer saliency in local contexts, DSCLRCN can mimic the cortical lateral inhibition mechanisms in human visual system to incorporate global contexts to assess the saliency of each image location by leveraging the deep spatial long short-term memory (DSLSTM) model. Moreover, we also integrate scene context modulation in DSLSTM for saliency inference, leading to a novel deep spatial contextual LSTM (DSCLSTM) model. The whole network can be trained end-to-end and works efficiently when testing. Experimental results on two benchmark datasets show that DSCLRCN can achieve state-of-the-art performance on saliency detection. Furthermore, the proposed DSCLSTM model can significantly boost the saliency detection performance by incorporating both global spatial interconnections and scene context modulation, which may uncover novel inspirations for studies on them in computational saliency models.

I. INTRODUCTION

The paper targets bottom-up eye-fixation prediction in natural scenes, where traditional saliency methods rely on hand-designed representations and local or global contrast. DSCLRCN learns local features and combines global spatial context with scene context in an end-to-end model, achieving strong benchmark performance.

  • Motivation: Bottom-up visual attention guides fixation toward distinctive regions while helping observers ignore visually plain areas.The paper focuses on predicting where people look during free viewing of natural scenes.
  • Motivation: Traditional saliency models use hand-designed low-, middle-, and high-level image features before applying local or global contrast inference.Local contrast compares each location with nearby surroundings, whereas global approaches consider the whole image.
  • Motivation: Biological evidence indicates that spatially separated visual locations interact through lateral inhibition, motivating holistic rather than purely local saliency assessment.Neighboring similar features can inhibit one another, and visual-cortex neurons compete when multiple stimuli are present.
  • Proposed approach: DSCLRCN learns saliency-related local features with CNNs, then uses DSLSTM to incorporate long-term global spatial interactions.The DSLSTM is intended to mimic lateral interconnections among image locations.
  • Proposed approach: DSCLSTM embeds scene features into DSLSTM to model scene modulation alongside global context during saliency inference.Scene context can make objects such as traffic signs or exceptional objects salient in particular environments.
  • Results: DSCLRCN achieves state-of-the-art performance on two benchmark datasets, outperforms 14 contemporary methods, and operates efficiently at test time.The complete model is trained end-to-end and directly outputs a saliency map for each input image.

II. RELATED WORK

Related work spans hand-crafted local- and global-context saliency methods and newer deep models for eye-fixation prediction. The paper positions DSLSTM and DSCLSTM as mechanisms for extending spatial context and adding scene modulation.

  • Traditional saliency models: Traditional saliency methods evaluate image locations using either local context, such as center-surround differences, or whole-image global context.Examples include Difference of Gaussians for local contrast and frequency-domain or Bayesian methods for global evaluation.
  • Deep saliency models: Deep models have been applied to saliency detection, including eye-fixation prediction, by learning image features with convolutional or neuromorphic architectures.Several approaches then classify local locations using linear SVMs or related predictors.
  • Deep saliency models: Many earlier deep eye-fixation models still assess saliency in local contexts because of local features, pixel-wise classifiers, or limited receptive fields.This limitation motivates a mechanism for propagating information across spatial locations.
  • Spatial recurrent modeling: DSLSTM is proposed to construct interconnections among image locations and incorporate long-term global context using recurrent modeling.The approach adapts LSTM’s long-term context capability from temporal sequence processing to spatial feature maps.
  • Contextual recurrent modeling: DSCLSTM extends DSLSTM by embedding scene features as contextual information, jointly modeling global context and scene modulation for saliency detection.This design follows contextual-LSTM work that incorporates external topics into recurrent computation.

A. Local Image Feature Extraction using CNNs

The local feature extractor uses pretrained CNNs to produce spatial feature maps while preserving resolution and, for ResNet-50, combining features from multiple depths. The resulting map has 512 channels and spatial stride 8.

  • CNN backbone: Pretrained VGG-16 or ResNet-50 convolutional layers extract local image features using fully convolutional architectures.The models are adapted to preserve spatial information for saliency prediction.
  • CNN backbone: Dilated convolutions expand receptive fields without reducing feature-map resolution or coverage.This preserves a relatively large spatial representation while incorporating broader local context.
  • Multiscale features: ResNet-50 multiscale extraction combines conv4 and conv5 features after channel reduction and normalization.The concatenated representation is reduced to a final feature map with 512 channels.
  • Feature-map output: Both CNN feature extractors use stride 8, producing a feature map with spatial dimensions reduced by a factor of 8.The extracted representation is subsequently used as the local feature map for the saliency network.

B. DSCLSTM for Context Incorporation

LSTM uses gated memory to retain or discard information over long-term sequences. Its input, forget, output, and modulation gates update memory and hidden states.

  • Reviewing LSTM: LSTM addresses vanishing gradients by using a memory cell to preserve states over long-term sequences.
  • Reviewing LSTM: Each LSTM unit updates its memory and hidden state through input, forget, output, and input-modulation gates.
  • Reviewing LSTM: The forget gate controls retained previous memory, while the input gate controls how much current information updates the memory cell.
  • Reviewing LSTM: Selective memorization and forgetting allow LSTM memory cells to propagate long-term and complex contextual dependencies.
  • Reviewing LSTM: Bidirectional LSTM uses parallel forward and reverse scans to capture both past and future information.

2) DSLSTM for Global Context Incorporation

DSLSTM extends spatial LSTM processing by scanning feature maps horizontally and vertically in four directions. Stacking two SLSTMs increases the model’s capacity for long-range global spatial dependencies.

  • DSLSTM for Global Context Incorporation: SLSTM applies four LSTMs to a local feature map, using bidirectional horizontal and vertical scans to blend context.
  • DSLSTM for Global Context Incorporation: Horizontal scans concatenate left-to-right and right-to-left hidden states, giving each location context from both sides.
  • DSLSTM for Global Context Incorporation: Vertical scans then process the fused feature map from top to bottom and bottom to top.
  • DSLSTM for Global Context Incorporation: Progressive four-direction scanning propagates information between all locations, incorporating long-term global contextual dependencies efficiently.
  • DSLSTM for Global Context Incorporation: Stacking two SLSTMs increases the capacity to learn longer-range and more complex spatial contextual dependencies.

3) Scene Context Modulation: DSCLSTM

DSCLSTM embeds scene features into DSLSTM to combine global spatial context with scene-context modulation. Scene information enters at the first time step and propagates across the feature map through memory cells.

  • Scene Context Modulation: DSCLSTM: DSCLSTM incorporates scene context into DSLSTM because scene modulation influences visual attention.
  • Scene Context Modulation: DSCLSTM: Scene features are added to the four LSTM gate computations at the first time step rather than at every time step.
  • Scene Context Modulation: DSCLSTM: The four directional LSTMs in DSLSTM are adjusted to receive scene context from their initial time steps.
  • Scene Context Modulation: DSCLSTM: Memory cells propagate scene contextual information to subsequent time steps across the whole feature map.
  • Scene Context Modulation: DSCLSTM: Each DSCLSTM output location integrates both global context and scene modulation before saliency assessment.

C. Saliency Assessment

DSCLRCN converts DSCLSTM features into a normalized saliency map and upsamples it to the original image size. Training uses negative NSS with human fixation ground truth, while testing directly outputs a map.

  • Saliency Assessment: A 1×1 convolution and Softmax generate the saliency map, with Softmax normalizing the whole map for lateral competition.
  • Saliency Assessment: Because the map is produced at stride 8, a bilinear-kernel deconvolution upsamples it by stride 8.
  • Saliency Assessment: Negative Normalized Scanpath Saliency against human eye fixations trains the complete DSCLRCN end-to-end through backpropagation.
  • Saliency Assessment: At test time, DSCLRCN accepts an image and directly produces its saliency map.

IV. EXPERIMENTS

The experiments evaluate DSCLRCN on two eye-fixation benchmark datasets using ablation studies and comparisons with state-of-the-art methods.

  • DSCLRCN is evaluated on two benchmark datasets for saliency detection.
  • The evaluation includes model ablations to measure the contribution of each component.
  • The study compares DSCLRCN quantitatively and qualitatively with other state-of-the-art methods.

A. Datasets

The study uses SALICON and MIT300 eye-fixation benchmarks and evaluates predictions with complementary saliency metrics, including AUC, NSS, and CC.

  • Datasets: SALICON contains 10,000 training, 5,000 validation, and 5,000 testing images selected from MS COCO.
  • Datasets: MIT300 contains 300 natural indoor or outdoor scenes, with MIT1003 serving as training and validation data because of similar eye-tracking setups.
  • Evaluation metrics: The evaluation considers EMD, NSS, CC, SIM, AUC, sAUC, KL, and IG, whose sensitivities to prediction errors differ.
  • Evaluation metrics: AUC evaluates classification of fixation and non-fixation locations by thresholding a normalized saliency map and tracing the ROC curve.

C. Implementation Details

The implementation trains DSCLRCN end-to-end with dataset-specific preprocessing, SGD optimization, recurrent initialization choices, and GPU-accelerated testing.

  • Data Processing: SALICON uses its training and validation sets directly, while MIT300 training fine-tunes on 903 MIT1003 images and validates on 100 images with horizontal flipping augmentation.
  • Data Processing: Images are resized to 480 × 640 for the local feature extractor and 227 × 227 for the scene feature extractor during training and testing.
  • Testing: Testing resizes output maps to the input dimensions and applies Gaussian filtering with σ = 0.035min(P, Q) and filter size 4σ.
  • Optimization: The whole network is trained with momentum SGD, batch size 20, scheduled learning-rate reductions, and dataset-specific iteration settings.
  • Software and speed: The implementation uses Caffe and Matlab, GPU acceleration, and reports a testing time of 0.27s per image.

1) Influence of the Receptive Field size

Ablation results indicate that larger receptive fields, global spatial recurrence, and scene modulation each improve saliency detection, while qualitative results show close agreement with ground truth.

  • Influence of the Receptive Field size: Larger receptive fields improve saliency detection, with especially large gains from FCN5 to FCN6 because FCN5 covers too little of the image.
  • Effectiveness of Global Context Incorporation: Adding one SLSTM layer to FCN7 improves performance, while two layers in DSLSTM provide further improvement.
  • Effectiveness of Global Context Incorporation: Increasing SLSTM depth beyond two layers does not produce further meaningful improvement.
  • Effectiveness of Scene Modulation: Adding scene modulation to FCN7_DSLSTM improves saliency detection performance, indicating informative scene-context contributions.
  • Qualitative comparison: On MIT1003 validation images, DSCLRCN produces the closest qualitative match to ground truth, with more accurate detections and fewer false positives.

4) Boosting the Performance using Deeper and Multiscale Features

DSCLRCN’s performance improves with stronger and multiscale feature extractors, while multilayer connections can hurt the FCN7-based variant. Across benchmark comparisons, DSCLRCN achieves strong quantitative and qualitative results against prior saliency models.

  • Deeper features: ResNet50 features further improve saliency detection performance over the corresponding weaker feature extractor.The paper attributes this to the greater power of ResNet50 features.
  • Multiscale features: Multiscale ResNet50 features further improve saliency detection performance.The multilayer model is reported as ResNet50_ML_DSCLSTM in Table III.
  • Model-dependent effects: Multilayer features worsen results in the FCN7_DSCLSTM model.The paper suggests this may result from FCN6 and FCN7 being trained from scratch, which can degrade network training when multilayer connections are added.
  • Benchmark results: On SALICON, DSCLRCN outperforms the compared models on AUC, NSS, and CC, although DeepGaze II achieves a better sAUC score.The paper notes that sAUC can favor blurred or hazy saliency maps because it primarily rewards true positives and is less affected by false positives.
  • Benchmark results: On MIT300, DSCLRCN achieves state-of-the-art AUC and surpasses all other models on NSS and CC, including nine DNN-based models.The authors connect this result to the DSCLSTM’s incorporation of global and scene contexts.
  • Qualitative results: Qualitative MIT1003 results show DSCLRCN matching ground-truth saliency maps best, with more accurate detections and fewer false positives than the compared models.The comparison includes four DNN-based models and challenging scenes.
Loading 1610.01708v1…