Source-linked AI summary
Contextual Encoder-Decoder Network for Visual Saliency Prediction
Alexander Kroner, Mario Senden, Kurt Driessens, Rainer Goebel
TL;DR
Predicting human fixations in natural scenes requires semantic features that capture multiple spatial scales and contextual information. The paper proposes a pretrained-CNN encoder-decoder with parallel dilated convolutions and global scene features, achieving competitive benchmark performance with a lightweight backbone. The authors identify remaining challenges in central-bias modeling, semantic robustness, implied gaze, and motion cues.
Problem
Existing saliency models do not explicitly combine high-level multi-scale visual features with contextual information for predicting fixations in complex natural scenes.
Method
A pretrained VGG16-based encoder-decoder combines multi-level semantic features, parallel dilated convolutions, and global scene information to predict normalized saliency maps.
Results
The model achieved competitive performance on two public saliency benchmarks and significantly improved results with ASPP across five eye-tracking datasets.
Takeaways & Limitations
Its lightweight backbone and computational efficiency support saliency prediction in applications with limited hardware, including virtual robotic systems.
Takeaways & Limitations
The study does not test whether its modifications generalize to other pretrained architectures, and central bias, semantic robustness, implied gaze, and motion remain challenges.
Abstract
from arXiv · showhide
Predicting salient regions in natural images requires the detection of objects that are present in a scene. To develop robust representations for this challenging task, high-level visual features at multiple spatial scales must be extracted and augmented with contextual information. However, existing models aimed at explaining human fixation maps do not incorporate such a mechanism explicitly. Here we propose an approach based on a convolutional neural network pre-trained on a large-scale image classification task. The architecture forms an encoder-decoder structure and includes a module with multiple convolutional layers at different dilation rates to capture multi-scale features in parallel. Moreover, we combine the resulting representations with global scene information for accurately predicting visual saliency. Our model achieves competitive and consistent results across multiple evaluation metrics on two public saliency benchmarks and we demonstrate the effectiveness of the suggested approach on five datasets and selected examples. Compared to state of the art approaches, the network is based on a lightweight image classification backbone and hence presents a suitable choice for applications with limited computational resources, such as (virtual) robotic systems, to estimate human fixations across complex natural scenes.
1. Introduction
Visual saliency prediction models aim to explain where people look in complex scenes. This work uses semantic, multi-scale, and global-context representations to predict fixation patterns efficiently.
- Motivation: Human fixations balance broad visual coverage with high-resolution sampling at the fovea.Acuity decreases toward the periphery, so gaze prioritizes selected spatial locations.
- Motivation: Saliency maps represent which scene regions reliably attract human fixations.Predicting their underlying properties can clarify observed viewing behavior.
- Motivation: CNN-based saliency models replace manual feature engineering with learned representations from raw images.These representations support predictions based on semantic information rather than only low-level contrasts.
- Contribution: The model combines multi-scale contextual features with global scene information to predict visual saliency.Its performance was evaluated on public benchmarks and unseen test instances.
- Contribution: The lightweight backbone achieved state-of-the-art-level accuracy while using roughly one order of magnitude fewer processing layers.The authors therefore present it as suitable for computationally constrained robotic environments.
2. Related Work
Visual saliency research progressed from theory- and feature-engineering-based models toward learned representations. Recent approaches use pretrained CNN features, while this model explicitly combines semantic information across spatial scales with contextual information.
- Earlier approaches: Early saliency models used Bayesian, graph-based, and manually designed visual-feature formulations.These approaches represented saliency through self-information, feature dissimilarity, or selected low-, mid-, and high-level cues.
- Deep learning approaches: Pretrained ImageNet classification networks enabled saliency models to learn transferable visual representations.DeepGaze and related architectures read salient locations from selected encoding stages.
- Deep learning approaches: Saliency prediction accuracy correlates with classification performance, highlighting the relevance of object information.Other work also explored coarse-to-fine resolutions and recurrent connections for long-range dependencies.
- Proposed direction: This model differs by explicitly combining semantic representations at multiple spatial scales to include contextual information.The design extends prior multi-stage feature approaches with an explicit contextual predictive process.
3. Methods
The proposed method is a VGG16-based encoder-decoder for dense fixation-map prediction. It combines multi-level features, parallel dilated convolutions, global context, and a learned decoder while training against normalized target distributions.
- 3.1. Architecture: The architecture adapts pretrained VGG16 convolutional layers into an encoder without fully connected layers for dense prediction.Downsampling is reduced to preserve spatial information, and the decoder restores the input resolution.
- 3.1. Architecture: Outputs from encoder layers 10, 14, and 18 are concatenated to combine mid- and high-level spatial features.This multi-level representation supports fine spatial detail while retaining semantic responses.
- 3.1. Architecture: The ASPP module applies parallel convolutions with different dilation factors to capture multi-scale image information.Global average pooling over the final encoder output adds scene-level context to the representation.
- 3.1. Architecture: Three decoder upsampling blocks use bilinear scaling followed by 3 × 3 convolutions to recover the original image resolution.Each scaling operation doubles the number of rows and columns.
- 3.2. Training: The model normalizes predictions to non-negative values summing to one and minimizes KL divergence against target distributions.Training uses Adam with a maximum learning rate of 10^-6 for 10 epochs, selecting the best checkpoint.
4. Experiments
The experiments evaluate the encoder-decoder model on five eye-tracking datasets using standard saliency metrics, independent test sets, ablations, and computational-efficiency measurements. Results show competitive or state-of-the-art benchmark performance, strong gains on selected CAT2000 categories, and suitability for resource-constrained systems.
- Experimental setup: The model was evaluated on five public eye-tracking datasets using qualitative and quantitative results, including architecture ablations and comparisons with prior work.SALICON was used for initial training before fine-tuning on other datasets.
- Evaluation metrics: Nine commonly reported metrics capture different notions of saliency, while KLD was used as the training objective for fixation-density prediction.The probabilistic formulation supports model ranking on MIT1003, CAT2000, and SALICON benchmarks.
- Benchmark results: On MIT300, the model marginally missed state-of-the-art performance on every individual metric but ranked third by cumulative rank among the compared approaches.It achieved the highest performance among entries using a VGG16 backbone.
- Computational efficiency: The model combined fast inference with a low GPU-memory footprint, supporting applications constrained by computational resources.The efficiency evaluation measured inference speed, memory use, and computational operations across five trained versions.
- Benchmark results: On CAT2000, the model achieved state-of-the-art scores for AUC-J, sAUC, and KLD and competitive results on the remaining metrics.Its cumulative rank exceeded previous approaches, including those using a pre-trained VGG16 network.
- Ablation and category analysis: Multi-scale information benefited CAT2000’s Noisy, Satellite, Cartoon, and Pattern categories by changing spatial weighting, capturing central bias, and highlighting color contrast.These qualitative mechanisms were associated with improvements from the ASPP module across selected evaluation metrics.
5. Discussion
The model combines multi-scale semantic features with global context and achieves competitive saliency performance while remaining computationally lightweight. The discussion identifies persistent limitations in central-bias modeling, higher-level scene understanding, and generalization across backbones.
- The ASPP module combines multi-scale semantic features and global context, improving saliency results qualitatively and quantitatively across five eye-tracking datasets.Different dilation factors provide large receptive fields for more holistic estimation of salient regions in complex scenes.
- The model is computationally lightweight and outperformed other networks using a pre-trained VGG16 backbone under cumulative-rank evaluation.This supports potential use in computationally efficient robotic or virtual robotic systems.
- Remaining obstacles include explicit central-bias modeling, robust semantic-feature detection, and representing implied gaze and motion in static images.The model also does not test whether its modifications generalize to other pre-trained architectures.
Appendix A. Feature Concatenation Ablation Analysis
The feature-concatenation ablation was evaluated across five datasets and showed dataset-dependent effects. Improvements were generally observed outside CAT2000, supporting a narrower recommendation for complex natural-image datasets.
- Ablation setup: Each concatenation condition was independently trained 10 times, with results summarized by means and standard deviations and significance tested using one-sided paired t-tests.Table A.8 marks significant performance increases between models with and without encoder-feature concatenation.
- Ablation results: CAT2000 models did not consistently benefit from multi-level feature aggregation, whereas other evaluated cases mostly showed significant improvement across most metrics.The ablation therefore indicates that concatenation effects depend on the dataset.
- Scope: Feature concatenation may be most appropriate for datasets dominated by complex natural images, including MIT1003, DUT-OMRON, PASCAL-S, and OSIE.