Deep Visual Attention Prediction

Wenguan Wang; Jianbing Shen

Source-linked AI summary

Deep Visual Attention Prediction

Wenguan Wang, Jianbing Shen

arXiv:1705.02544v3May 7, 2017cs.CV

TL;DR

The paper addresses task-free prediction of human eye fixations in natural images, where CNN attention models still need more efficient use of multi-scale features. It proposes a skip-layer, deeply supervised encoder-decoder CNN that fuses saliency from multiple convolutional levels within one network. Experiments on five challenging eye-tracking datasets report more accurate saliency maps than state-of-the-art methods, with 10fps GPU inference.

Problem
CNN attention models need to improve their efficient use of multi-scale features for task-free human eye-fixation prediction in natural images.
Method
A skip-layer, deeply supervised encoder-decoder CNN predicts and fuses multi-level saliency maps from different convolutional layers within a single network.
Results
More accurate saliency maps than state-of-the-art methods were reported across five public datasets, with 10fps inference on a GPU.
Takeaways & Limitations
The model provides a memory- and time-efficient approach for multi-scale human-attention prediction using one fully convolutional network.

Abstract

from arXiv · show

In this work, we aim to predict human eye fixation with view-free scenes based on an end-to-end deep learning architecture. Although Convolutional Neural Networks (CNNs) have made substantial improvement on human attention prediction, it is still needed to improve CNN based attention models by efficiently leveraging multi-scale features. Our visual attention network is proposed to capture hierarchical saliency information from deep, coarse layers with global saliency information to shallow, fine layers with local saliency response. Our model is based on a skip-layer network structure, which predicts human attention from multiple convolutional layers with various reception fields. Final saliency prediction is achieved via the cooperation of those global and local predictions. Our model is learned in a deep supervision manner, where supervision is directly fed into multi-level layers, instead of previous approaches of providing supervision only at the output layer and propagating this supervision back to earlier layers. Our model thus incorporates multi-level saliency predictions within a single network, which significantly decreases the redundancy of previous approaches of learning multiple network streams with different input scales. Extensive experimental analysis on various challenging benchmark datasets demonstrate our method yields state-of-the-art performance with competitive inference time.

I. INTRODUCTION

The paper targets task-free bottom-up prediction of human eye fixations in natural images and proposes a CNN that combines hierarchical, multi-scale saliency within one network. Deep supervision and an encoder-decoder design support efficient prediction, while experiments report accurate maps and 10fps inference.

Motivation: The task is predicting human eye fixations in natural images without requiring task or scene-context knowledge.The authors frame this as task-free bottom-up visual attention prediction.
Proposed approach: The skip architecture captures local-to-global saliency responses from shallow-to-deep convolutional layers with different receptive fields.Final attention is formed by deeply fusing saliency estimations from multiple levels.
Proposed approach: Multi-scale saliency is learned within a single network, reducing the redundancy of conventional multi-stream attention models.The design captures multi-level information while remaining efficient in memory and computational time.
Architecture: The encoder-decoder network uses trainable multi-channel upsampling kernels to map low-resolution encoder features to dense full-input-resolution predictions.The encoder follows the first 13 convolutional layers of VGG16, while the decoder performs learned upsampling and dimensionality reduction.
Evaluation: 10fps on a GPU and more accurate saliency maps than state-of-the-art methods are reported across five challenging public datasets.The evaluated datasets are MIT300, MIT1003, TORONTO, PASCAL-S, and DUT-OMRON.
Training: Deep supervision feeds supervision directly into multiple layers to learn multi-level saliency features and improve their discriminativeness and robustness.This differs from supervising only the output layer and propagating supervision backward.

II. RELATED WORK

Related work covers traditional and deep-learning approaches to saliency detection, alongside common single-stream, multi-stream, and skip-layer architectures. The paper positions its adopted skip-layer design as a single-network way to incorporate multi-scale saliency and multi-level supervision.

Saliency Detection: Saliency detection includes predicting human fixation points and separately highlighting salient object regions for object-level tasks.The paper mainly reviews fixation-prediction models because its method targets human eye fixations.
Saliency Detection: Classic bottom-up attention models use hand-designed intensity, color, and orientation features or center-surround heuristics.These models are stimulus-driven rather than dependent on explicit scene-context understanding.
Deep Learning Architectures: Deep saliency models introduced automatic hierarchical feature extraction and end-to-end learning, including architectures that blend responses from different network layers.Examples include eDN and DeepGaze, which use multi-layer deep-network responses for saliency prediction.
Deep Learning Architectures: The adopted skip-layer architecture incorporates multi-scale saliency within a single network and applies supervision to earlier and last layers.The paper contrasts this with single-stream and multi-stream configurations and distinguishes bottom-up/top-down segmentation architectures.
Deep Learning Architectures: Single-stream networks process inputs through a standard CNN, whereas multi-stream networks train parallel networks with multi-scale inputs and combine their responses.These configurations reflect efforts to incorporate saliency cues at different levels and scales.

2) Multi-stream network:

The adopted attention model learns multi-scale saliency inside one primary network by combining hierarchical features and directly supervising hidden layers. Its encoder-decoder structure produces multi-scale estimates while avoiding multiple input-scale streams.

Multi-stream network: Multi-stream networks learn multi-scale saliency by training parallel networks with multi-scale inputs, often using different architectures for different scales.
Skip-layer network: Skip-layer networks instead learn multi-scale responses inside one primary stream from layers with different receptive fields and downsampling ratios.
Proposed architecture: The proposed network combines multi-scale and multi-level attention information from different convolutional layers through deep supervision.
Comparison: Compared with multi-stream and bottom-up/top-down alternatives, the model is lightweight, and deep supervision provides higher training efficiency than the difficult-to-train bottom-up/top-down network.
Proposed architecture: Direct supervision reaches hidden layers, encouraging robust features and multi-scale saliency estimates rather than supervising only the output layer.
Proposed architecture: The encoder-decoder architecture recovers spatial information while integrating global high-level and local low-level features for attention prediction.

B. Proposed Attention Model

The proposed model is an end-to-end fully convolutional network that predicts pixel-wise saliency using encoder-decoder reconstruction, multiple decoder outputs, and learned attention fusion.

Network: The model is a fully convolutional neural network trained end-to-end to predict pixel-wise saliency values for each input image.
Encoder: Convolutional layers extract hierarchical features, while pooling and nonlinearities improve translation invariance and representation capability.
Decoder: The encoder decreases feature-map resolution, and the decoder uses transposed convolutions to upsample coarse maps back to the input resolution.
Prediction: Each decoder output is passed through a 1 × 1 convolution with sigmoid nonlinearity to produce a saliency map matching the input size.
Multi-scale prediction: Multiple encoder layers are selected for explicit multi-scale saliency prediction, with each selected layer connected to its own decoder.
Optimization: The network jointly minimizes output-layer and fusion-layer classification errors using standard back-propagation stochastic gradient descent.
Fusion: An attention-fusion layer with a 1×1 convolution learns how to combine the decoder saliency maps into a fused prediction.

1) Encoder Network:

The encoder is a modified VGG16-style convolutional stack that preserves spatial detail while progressively expanding receptive fields and semantic representation capacity.

Encoder Network: The encoder uses the convolutional layers of VGG16 and removes its final pooling layer to preserve more spatial information.
Hierarchical features: Deeper convolutional layers have larger receptive fields, while lower layers capture low-level features and higher layers encode semantic features.
Decoder paths: Each selected saliency feature map is decoded through multiple deconvolution layers until reaching the original input size.
Encoder Network: Successive convolutional blocks increase output channels from 64 →128 →256 →512 →512, supporting progressively richer semantic representations.
Decoder paths: Decoder paths reduce channel dimensionality as they upsample, limiting memory and inference-time computational redundancy.

3) Training and Testing:

The model is initialized partly from ImageNet-pretrained VGG16 weights, trained on SALICON data, and evaluated across five saliency datasets with varied characteristics.

Training: The encoder’s five convolutional blocks are initialized from VGG16 trained on ImageNet, while remaining layers use zero-mean Gaussian initialization with standard deviation 0.01.
Training: The networks are trained using 10,000 images from the SALICON training set.
Datasets: Evaluation covers five widely used saliency datasets with different characteristics.
Datasets: MIT300 contains 300 natural images with saliency maps derived from eye-tracking data collected from 39 users.
Datasets: MIT1003 contains 1,003 Flickr and LabelMe images, with groundtruth maps created from 15 human observers.
Datasets: TORONTO contains 120 indoor and outdoor color images presented randomly to 20 subjects for 3 seconds, separated by 2-second gray masks.

3) TORONTO [26]:

This section describes the evaluation setting and metrics used to assess saliency predictions, including dataset characteristics and model-comparison context.

Evaluation metrics: Evaluation considers multiple metrics because no single saliency metric provides a fully fair comparison.The listed metrics include EMD, NSS, SIM, CC, and several AUC variants.
Evaluation metrics: EMD measures the minimum cost of transforming predicted saliency distribution S into ground-truth distribution G, with lower values indicating higher-quality maps.
Model comparison: The comparison includes characteristics of the proposed model and 13 state-of-the-art saliency models.The supplied material also notes that detailed information was not released for some models.

2) Normalized Scanpath Saliency (NSS):

NSS evaluates saliency at human fixation locations using a normalized prediction map, while the surrounding evaluation framework compares models with multiple statistical and distributional measures.

Metric definition: NSS is designed for saliency-map evaluation using a predicted saliency map S and binary fixation map Q.
Metric definition: NSS averages the scores assigned by a zero-mean, unit-standard-deviation saliency map at human eye fixations.N denotes the total number of human eye positions, and σ(·) denotes standard deviation.
Related metrics: CC measures the linear relationship between continuous fixation map G and predicted saliency map S, with values ranging from -1 to +1.Scores near -1 or +1 indicate perfect alignment between the maps.
Related metrics: SIM compares normalized saliency and fixation distributions as histograms by summing the minimum value at each pixel.A SIM of one means identical distributions, whereas zero means no overlap.
Benchmark comparisons: MIT300 and MIT1003 quantitative comparisons include multiple saliency models, while DeeFix is fine-tuned on 900 MIT1003 images and evaluated on the remaining 103.

5) AUC [26]:

AUC evaluates fixation discrimination through ROC analysis, while comparisons across datasets assess competing saliency models and the proposed model’s qualitative and quantitative behavior.

AUC definition: AUC treats fixated points as positives and non-fixated points as negatives, varying a saliency threshold to construct the ROC curve.
AUC variants: The experiments use AUC-Judd, AUC-Borji, and shuffled AUC, differing in how non-fixation points are sampled.AUC-Judd and AUC-Borji use uniform non-fixation distributions, whereas shuffled AUC uses fixations from other images.
Comparison results: The proposed DVA model is compared with 13 state-of-the-art models, and its reported runtime is 0.1s per frame.
Comparison results: Quantitative scores are reported on MIT300, MIT1003, TORONTO, PASCAL-S, and DUT-OMRON, with promising results across the datasets despite weaker competition on MIT300 against DeeFix and SALICON.Those competing models use more complex architectures or additional training data.
Qualitative results: Qualitative examples show DVA capturing low-level color contrast and higher-level cues such as humans, faces, and text across varied image conditions.The maps are described as localized even with cluttered backgrounds or salient regions of different sizes.

D. Ablation Study

The ablation study finds that multi-scale layer predictions are complementary, while learned weighted fusion and deep supervision provide the strongest configuration. Learnable upsampling also improves performance, and the complete model maintains superior performance with efficient resource use.

1) Submodule:: Predictions from conv3-3, conv4-3, and conv5-3 capture complementary saliency information at different scales, improving the overall saliency maps.The layers represent distinct scales within the encoder-decoder architecture.
2) Fusion Strategy:: Averaging multi-layer outputs outperforms single-layer outputs, while learned weighted fusion achieves the best performance.The fusion layer automatically learns the combination weights during training.
3) Supervision:: Deep supervision improves the model beyond supervision applied only to the weighted-fusion output.The full model supervises both the fused prediction and each individual output layer.
4) Upsampling:: Learnable multi-channel upsampling kernels outperform directly upsampling saliency outputs with fixed bilinear interpolation.Replacing learnable kernels with fixed bilinear interpolation causes a performance drop.
Overall:: The complete model achieves superior performance on five eye-tracking datasets while remaining memory- and time-efficient.The model is fully convolutional and runs at 10fps on a GPU.