PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Visual Attention Prediction

Wenguan Wang, Jianbing Shen

arXiv:1705.02544v3cs.CV

TL;DR

The paper addresses task-free prediction of human eye fixations in natural images, where CNN attention models still need more efficient use of multi-scale features. It proposes a skip-layer, deeply supervised encoder-decoder CNN that fuses saliency from multiple convolutional levels within one network. Experiments on five challenging eye-tracking datasets report more accurate saliency maps than state-of-the-art methods, with 10fps GPU inference.

  • Problem

    CNN attention models need to improve their efficient use of multi-scale features for task-free human eye-fixation prediction in natural images.

    Source: §I · pp. 1–2

  • Method

    A skip-layer, deeply supervised encoder-decoder CNN predicts and fuses multi-level saliency maps from different convolutional layers within a single network.

    Source: §I · p. 2, B. Proposed Attention Model · p. 5

  • Results

    More accurate saliency maps than state-of-the-art methods were reported across five public datasets, with 10fps inference on a GPU.

    Source: §I · p. 2, §V · p. 10

  • Takeaways & Limitations

    The model provides a memory- and time-efficient approach for multi-scale human-attention prediction using one fully convolutional network.

    Source: §I · p. 2, §V · p. 10

Abstract

from arXiv · show

In this work, we aim to predict human eye fixation with view-free scenes based on an end-to-end deep learning architecture. Although Convolutional Neural Networks (CNNs) have made substantial improvement on human attention prediction, it is still needed to improve CNN based attention models by efficiently leveraging multi-scale features. Our visual attention network is proposed to capture hierarchical saliency information from deep, coarse layers with global saliency information to shallow, fine layers with local saliency response. Our model is based on a skip-layer network structure, which predicts human attention from multiple convolutional layers with various reception fields. Final saliency prediction is achieved via the cooperation of those global and local predictions. Our model is learned in a deep supervision manner, where supervision is directly fed into multi-level layers, instead of previous approaches of providing supervision only at the output layer and propagating this supervision back to earlier layers. Our model thus incorporates multi-level saliency predictions within a single network, which significantly decreases the redundancy of previous approaches of learning multiple network streams with different input scales. Extensive experimental analysis on various challenging benchmark datasets demonstrate our method yields state-of-the-art performance with competitive inference time.

I. INTRODUCTION

The paper targets task-free bottom-up prediction of human eye fixations in natural images and proposes a CNN that combines hierarchical, multi-scale saliency within one network. Deep supervision and an encoder-decoder design support efficient prediction, while experiments report accurate maps and 10fps inference.

Motivation

  • The task is predicting human eye fixations in natural images without requiring task or scene-context knowledge.The authors frame this as task-free bottom-up visual attention prediction.Source: §I · p. 1

Proposed approach

  • The skip architecture captures local-to-global saliency responses from shallow-to-deep convolutional layers with different receptive fields.Final attention is formed by deeply fusing saliency estimations from multiple levels.Source: §I · pp. 1–2
  • Multi-scale saliency is learned within a single network, reducing the redundancy of conventional multi-stream attention models.The design captures multi-level information while remaining efficient in memory and computational time.Source: §I · p. 2

Architecture

  • The encoder-decoder network uses trainable multi-channel upsampling kernels to map low-resolution encoder features to dense full-input-resolution predictions.The encoder follows the first 13 convolutional layers of VGG16, while the decoder performs learned upsampling and dimensionality reduction.Source: §I · p. 2

Evaluation

  • 10fps on a GPU and more accurate saliency maps than state-of-the-art methods are reported across five challenging public datasets.The evaluated datasets are MIT300, MIT1003, TORONTO, PASCAL-S, and DUT-OMRON.Source: §I · p. 2

Training

  • Deep supervision feeds supervision directly into multiple layers to learn multi-level saliency features and improve their discriminativeness and robustness.This differs from supervising only the output layer and propagating supervision backward.Source: §I · p. 2

II. RELATED WORK

Related work covers traditional and deep-learning approaches to saliency detection, alongside common single-stream, multi-stream, and skip-layer architectures. The paper positions its adopted skip-layer design as a single-network way to incorporate multi-scale saliency and multi-level supervision.

Saliency Detection

  • Saliency detection includes predicting human fixation points and separately highlighting salient object regions for object-level tasks.The paper mainly reviews fixation-prediction models because its method targets human eye fixations.Source: A. Saliency Detection · p. 2
  • Classic bottom-up attention models use hand-designed intensity, color, and orientation features or center-surround heuristics.These models are stimulus-driven rather than dependent on explicit scene-context understanding.Source: A. Saliency Detection · p. 2

Deep Learning Architectures

  • Deep saliency models introduced automatic hierarchical feature extraction and end-to-end learning, including architectures that blend responses from different network layers.Examples include eDN and DeepGaze, which use multi-layer deep-network responses for saliency prediction.Source: A. Saliency Detection · p. 2
  • The adopted skip-layer architecture incorporates multi-scale saliency within a single network and applies supervision to earlier and last layers.The paper contrasts this with single-stream and multi-stream configurations and distinguishes bottom-up/top-down segmentation architectures.Source: Figure 2 · p. 3, B. Deep Learning Architectures of… · p. 3
  • Single-stream networks process inputs through a standard CNN, whereas multi-stream networks train parallel networks with multi-scale inputs and combine their responses.These configurations reflect efforts to incorporate saliency cues at different levels and scales.Source: 1) Single-stream network · p. 3

2) Multi-stream network:

The adopted attention model learns multi-scale saliency inside one primary network by combining hierarchical features and directly supervising hidden layers. Its encoder-decoder structure produces multi-scale estimates while avoiding multiple input-scale streams.

Multi-stream network

  • Multi-stream networks learn multi-scale saliency by training parallel networks with multi-scale inputs, often using different architectures for different scales.Source: 2) Multi-stream network · p. 3

Skip-layer network

Proposed architecture

Comparison

  • Compared with multi-stream and bottom-up/top-down alternatives, the model is lightweight, and deep supervision provides higher training efficiency than the difficult-to-train bottom-up/top-down network.Source: 5) The adopted network · p. 3

Proposed architecture

  • Direct supervision reaches hidden layers, encouraging robust features and multi-scale saliency estimates rather than supervising only the output layer.Source: 5) The adopted network · p. 3, Figure 3 · p. 4
  • The encoder-decoder architecture recovers spatial information while integrating global high-level and local low-level features for attention prediction.Source: A. Architecture Overview · p. 4

B. Proposed Attention Model

The proposed model is an end-to-end fully convolutional network that predicts pixel-wise saliency using encoder-decoder reconstruction, multiple decoder outputs, and learned attention fusion.

Network

Encoder

  • Convolutional layers extract hierarchical features, while pooling and nonlinearities improve translation invariance and representation capability.Source: B. Proposed Attention Model · p. 4

Decoder

  • The encoder decreases feature-map resolution, and the decoder uses transposed convolutions to upsample coarse maps back to the input resolution.Source: B. Proposed Attention Model · p. 4

Prediction

  • Each decoder output is passed through a 1 × 1 convolution with sigmoid nonlinearity to produce a saliency map matching the input size.Source: B. Proposed Attention Model · p. 5

Multi-scale prediction

  • Multiple encoder layers are selected for explicit multi-scale saliency prediction, with each selected layer connected to its own decoder.Source: B. Proposed Attention Model · p. 5

Optimization

  • The network jointly minimizes output-layer and fusion-layer classification errors using standard back-propagation stochastic gradient descent.Source: B. Proposed Attention Model · p. 5

Fusion

1) Encoder Network:

The encoder is a modified VGG16-style convolutional stack that preserves spatial detail while progressively expanding receptive fields and semantic representation capacity.

Encoder Network

  • The encoder uses the convolutional layers of VGG16 and removes its final pooling layer to preserve more spatial information.Source: 1) Encoder Network · p. 5

Hierarchical features

  • Deeper convolutional layers have larger receptive fields, while lower layers capture low-level features and higher layers encode semantic features.Source: 1) Encoder Network · p. 5

Decoder paths

  • Each selected saliency feature map is decoded through multiple deconvolution layers until reaching the original input size.Source: 2) Decoder Network · p. 5

Encoder Network

  • Successive convolutional blocks increase output channels from 64 →128 →256 →512 →512, supporting progressively richer semantic representations.Source: 2) Decoder Network · p. 5

Decoder paths

  • Decoder paths reduce channel dimensionality as they upsample, limiting memory and inference-time computational redundancy.Source: 2) Decoder Network · pp. 5–6

3) Training and Testing:

The model is initialized partly from ImageNet-pretrained VGG16 weights, trained on SALICON data, and evaluated across five saliency datasets with varied characteristics.

Training

  • The encoder’s five convolutional blocks are initialized from VGG16 trained on ImageNet, while remaining layers use zero-mean Gaussian initialization with standard deviation 0.01.Source: 3) Training and Testing · p. 6
  • The networks are trained using 10,000 images from the SALICON training set.Source: 3) Training and Testing · p. 6

Datasets

  • Evaluation covers five widely used saliency datasets with different characteristics.Source: A. Datasets · p. 6
  • MIT300 contains 300 natural images with saliency maps derived from eye-tracking data collected from 39 users.Source: 1) MIT300 [24] · p. 6
  • MIT1003 contains 1,003 Flickr and LabelMe images, with groundtruth maps created from 15 human observers.Source: 2) MIT1003 [25] · p. 6
  • TORONTO contains 120 indoor and outdoor color images presented randomly to 20 subjects for 3 seconds, separated by 2-second gray masks.Source: 3) TORONTO [26] · p. 6

3) TORONTO [26]:

This section describes the evaluation setting and metrics used to assess saliency predictions, including dataset characteristics and model-comparison context.

Evaluation metrics

  • Evaluation considers multiple metrics because no single saliency metric provides a fully fair comparison.The listed metrics include EMD, NSS, SIM, CC, and several AUC variants.Source: B. Evaluation Metrics · p. 6
  • EMD measures the minimum cost of transforming predicted saliency distribution S into ground-truth distribution G, with lower values indicating higher-quality maps.Source: 1) Earth Movers Distance (EMD) · p. 6

Model comparison

2) Normalized Scanpath Saliency (NSS):

NSS evaluates saliency at human fixation locations using a normalized prediction map, while the surrounding evaluation framework compares models with multiple statistical and distributional measures.

Metric definition

Related metrics

  • CC measures the linear relationship between continuous fixation map G and predicted saliency map S, with values ranging from -1 to +1.Scores near -1 or +1 indicate perfect alignment between the maps.Source: 3) Linear Correlation Coefficient (CC) · p. 7
  • SIM compares normalized saliency and fixation distributions as histograms by summing the minimum value at each pixel.A SIM of one means identical distributions, whereas zero means no overlap.Source: 4) Similarity Metric (SIM) · p. 7

Benchmark comparisons

5) AUC [26]:

AUC evaluates fixation discrimination through ROC analysis, while comparisons across datasets assess competing saliency models and the proposed model’s qualitative and quantitative behavior.

AUC definition

  • AUC treats fixated points as positives and non-fixated points as negatives, varying a saliency threshold to construct the ROC curve.Source: 5) AUC [26] · p. 7

AUC variants

  • The experiments use AUC-Judd, AUC-Borji, and shuffled AUC, differing in how non-fixation points are sampled.AUC-Judd and AUC-Borji use uniform non-fixation distributions, whereas shuffled AUC uses fixations from other images.Source: 5) AUC [26] · p. 7

Comparison results

  • The proposed DVA model is compared with 13 state-of-the-art models, and its reported runtime is 0.1s per frame.Source: C. Comparison Results · pp. 7–8
  • Quantitative scores are reported on MIT300, MIT1003, TORONTO, PASCAL-S, and DUT-OMRON, with promising results across the datasets despite weaker competition on MIT300 against DeeFix and SALICON.Those competing models use more complex architectures or additional training data.Source: C. Comparison Results · p. 8

Qualitative results

  • Qualitative examples show DVA capturing low-level color contrast and higher-level cues such as humans, faces, and text across varied image conditions.The maps are described as localized even with cluttered backgrounds or salient regions of different sizes.Source: C. Comparison Results · p. 8

D. Ablation Study

The ablation study finds that multi-scale layer predictions are complementary, while learned weighted fusion and deep supervision provide the strongest configuration. Learnable upsampling also improves performance, and the complete model maintains superior performance with efficient resource use.

1) Submodule:

  • Predictions from conv3-3, conv4-3, and conv5-3 capture complementary saliency information at different scales, improving the overall saliency maps.The layers represent distinct scales within the encoder-decoder architecture.Source: 1) Submodule · p. 8

2) Fusion Strategy:

  • Averaging multi-layer outputs outperforms single-layer outputs, while learned weighted fusion achieves the best performance.The fusion layer automatically learns the combination weights during training.Source: 2) Fusion Strategy · p. 8, DUT-OMRON · p. 9

3) Supervision:

  • Deep supervision improves the model beyond supervision applied only to the weighted-fusion output.The full model supervises both the fused prediction and each individual output layer.Source: 3) Supervision · p. 9

4) Upsampling:

  • Learnable multi-channel upsampling kernels outperform directly upsampling saliency outputs with fixed bilinear interpolation.Replacing learnable kernels with fixed bilinear interpolation causes a performance drop.Source: 4) Upsampling · p. 9

Overall:

  • The complete model achieves superior performance on five eye-tracking datasets while remaining memory- and time-efficient.The model is fully convolutional and runs at 10fps on a GPU.Source: §V · p. 10

Related papers

Loading 1705.02544v3…