PaperlayerView exact revision on arXiv

Source-linked AI summary

A Dilated Inception Network for Visual Saliency Prediction

Sheng Yang, Guosheng Lin, Qiuping Jiang, Weisi Lin

arXiv:1904.03571v2cs.CV

TL;DR

Hand-crafted saliency features and their heuristic combinations are insufficient to represent the variety of factors influencing visual saliency. DINet uses an efficient dilated inception module and linear-normalization probability-distribution losses, and achieves state-of-the-art performance on saliency benchmarks in efficiency and efficacy.

  • Problem

    Classic hand-crafted features and heuristic combinations are insufficient to represent the wide variety of factors that contribute to visual saliency.

    Source: §I · p. 1

  • Method

    DINet pairs a DCNN backbone and dilated inception module with a fully convolutional decoder, and trains end-to-end using linear normalization-based probability distribution distance losses.

    Source: §I · pp. 1–2

  • Results

    DINet with the proposed loss functions achieves state-of-the-art performance on saliency benchmark datasets in both efficiency and efficacy.

    Source: §I · p. 2, §V · p. 13

  • Takeaways & Limitations

    The proposed loss functions outperform standard regression and existing softmax-normalization probability-distribution distance losses in the reported comparisons.

    Source: §I · p. 2, §V · p. 13

Abstract

from arXiv · show

Recently, with the advent of deep convolutional neural networks (DCNN), the improvements in visual saliency prediction research are impressive. One possible direction to approach the next improvement is to fully characterize the multi-scale saliency-influential factors with a computationally-friendly module in DCNN architectures. In this work, we proposed an end-to-end dilated inception network (DINet) for visual saliency prediction. It captures multi-scale contextual features effectively with very limited extra parameters. Instead of utilizing parallel standard convolutions with different kernel sizes as the existing inception module, our proposed dilated inception module (DIM) uses parallel dilated convolutions with different dilation rates which can significantly reduce the computation load while enriching the diversity of receptive fields in feature maps. Moreover, the performance of our saliency model is further improved by using a set of linear normalization-based probability distribution distance metrics as loss functions. As such, we can formulate saliency prediction as a probability distribution prediction task for global saliency inference instead of a typical pixel-wise regression problem. Experimental results on several challenging saliency benchmark datasets demonstrate that our DINet with proposed loss functions can achieve state-of-the-art performance with shorter inference time.

I. INTRODUCTION

The paper models task-free bottom-up visual attention by predicting eye fixations on natural images, where saliency maps represent the probability of attracting attention. It proposes DINet to capture multi-scale context efficiently and trains it with linear normalization-based distribution-distance losses.

I. INTRODUCTION

  • The paper models task-free bottom-up attention by predicting eye fixations on natural images; brighter saliency-map pixels indicate higher attention probability.Source: §I · p. 1
  • Hand-crafted low-level features and their heuristic combinations are insufficient for the diverse factors affecting saliency, while multi-scale DCNN models carry high computation costs.Source: §I · p. 1
  • Linear normalization-based probability-distribution distance losses outperform softmax-based distribution losses and standard regression losses with the same architecture.The paper frames saliency prediction as probability-distribution prediction rather than pixel-wise regression.Source: §I · p. 2
  • DINet with the proposed losses achieves state-of-the-art performance across saliency benchmarks in both efficiency and efficacy.The paper describes the model as an end-to-end encoder-decoder network.Source: §I · p. 2
  • Parallel dilated convolutions with varied dilation rates let DIM capture diverse multi-scale contextual features at lower computation cost.The original inception module instead uses convolutions with different kernel sizes.Source: §I · pp. 1–2
  • Replacing deconvolution layers with a fully convolutional decoder reduces computation and supports real-time performance.Source: §I · p. 2

II. RELATED WORK

The related-work section reviews deep-learning saliency models and approaches to multi-scale feature extraction. It discusses model architectures, receptive-field and resolution trade-offs, and alternative loss formulations.

A. Deep Learning-Based Visual Saliency Prediction

B. Existing Multi-Scale Feature Extraction Deep Learning Architectures

1) Image Pyramid Network:

The paper reviews multi-scale feature approaches and their trade-offs, then introduces DINet, which combines a dilated residual encoder, a dilated inception module, a decoder, and linear-normalization losses. Dilated convolution expands receptive fields while preserving spatial resolution.

1) Image Pyramid Network:

  • Image pyramid networks merge parallel multi-scale input features but incur high computation and memory costs.Their outputs feed a decoder to generate the saliency map.Source: 1) Image Pyramid Network · p. 3

2) Skip-layer Network:

  • Skip-layer networks concatenate features from different depths more efficiently, but reduced spatial information can make saliency inference uncertain and ambiguous.The approach reuses intermediate features with increasingly larger receptive fields and output strides.Source: 2) Skip-layer Network · p. 3

3) Inception-based Network:

  • Inception-based saliency models use dilated convolutions and fewer down-sampling operations to retain spatial information, but conventional inception modules are computationally costly.DINet revises the module for more powerful multi-scale extraction with computational efficiency.Source: 3) Inception-based Network · pp. 3–4

III. OUR APPROACH

  • DINet uses a Dilated Residual Network for dense features, a dilated inception module for multi-scale features, and a decoder for saliency maps.It is trained using linear normalization-based probability-distribution distance metrics.Source: §III · p. 4

1) Dilated Convolution:

  • Dilated convolution inserts holes into kernels to enlarge receptive fields and enable dense extraction while preserving spatial resolution.A 3 × 3 kernel with dilation rate 2 samples like a 5 × 5 standard kernel, using only 3 × 3 meaningful parameters.Source: 1) Dilated Convolution · p. 4, Figure 3 · p. 4

2) Dilated Residual Network:

The encoder uses a dilated ResNet-50 backbone with output stride 8, and a compact convolutional decoder produces the saliency map with bilinear up-sampling.

2) Dilated Residual Network:

  • Dilated ResNet-50 removes down-sampling in Conv4 and Conv5 and uses dilation rates 2 and 4, yielding output stride 8.The paper selects ResNet-50 as its backbone and describes output stride 8 as a compromise between spatial resolution and computation cost.Source: 2) Dilated Residual Network · pp. 4–5

B. Decoder Network

  • The decoder uses three standard convolutional layers followed by bilinear up-sampling, avoiding deconvolutional layers.Its final 3 × 3 prediction kernel uses sigmoid activation to place saliency values in [0,1].Source: B. Decoder Network · p. 5

C. Proposed Dilated Inception Module

The proposed DIM uses parallel dilated convolutions and bottleneck design to capture multi-scale context with limited added parameters. The paper also reports that linear-normalization losses outperform softmax-normalized and unnormalized alternatives.

C. Proposed Dilated Inception Module

  • Replacing 7 × 7 and 5 × 5 standard convolutions with 3 × 3 dilated convolutions uses 56W fewer parameters while achieving similar or better results.The parallel dilated branches provide diverse fields of view; the final DIM uses dilation rates [4, 8, 16] and sum-fusion.Source: C. Proposed Dilated Inception Module · p. 6
  • 27W additional parameters and more than 50 FPS are reported for DINet with 240 × 320 input images.Among the compared models, DINet is described as offering a relatively good trade-off between validation performance and inference speed.Source: C. Proposed Dilated Inception Module · pp. 6–7

D. Loss Function

  • The loss treats saliency maps as probability distributions and replaces softmax normalization with linear regularization before measuring distribution distance.Total variation distance is selected based on validation results as an example loss.Source: D. Loss Function · p. 7
  • The linear-normalization loss functions perform better than both softmax-normalized and unnormalized alternatives.For saliency responses in [0,1], linear normalization preserves their initial proportions, whereas softmax de-emphasizes maximum values.Source: D. Loss Function · p. 7

IV. EXPERIMENTS

The experiments evaluate DINet qualitatively and quantitatively on several public saliency benchmark datasets.

IV. EXPERIMENTS

  • DINet is evaluated qualitatively and quantitatively on several public saliency benchmark datasets.Source: §IV · p. 7

A. Saliency Benchmark Datasets

The evaluation uses three public saliency benchmark datasets with different image contents and settings. The supplied dataset details describe SALICON’s size, splits, and held-out test evaluation.

A. Saliency Benchmark Datasets

  • Three public saliency benchmark datasets with different image contents and experimental settings are used.The supplied passages give dataset-specific details only for SALICON.Source: A. Saliency Benchmark Datasets · p. 7

1) SALICON [20]:

  • SALICON contains 10,000 training, 5,000 validation, and 5,000 test images drawn from Microsoft COCO.The ground truths for the test split are held out.Source: 1) SALICON [20] · p. 7
  • SALICON test results require submission through the challenge website, while evaluation protocols and code are available online.Each image has spatial resolution 480 × 640.Source: 1) SALICON [20] · p. 7

3) MIT300 [22]:

MIT300 is evaluated through the MIT Saliency Benchmark, with MIT1003 usually serving as its training and validation data. The section describes four metrics that capture different aspects of agreement with human fixation data.

3) MIT300 [22]:

  • MIT300 contains 300 indoor and outdoor images, with ground truths held out and evaluation available only by benchmark submission.MIT1003 is usually used for MIT300 training and validation.Source: 3) MIT300 [22] · p. 8

B. Evaluation Metrics for Saliency Prediction

  • Evaluation uses AUC, shuffled AUC (sAUC), Normalized Scanpath Saliency (NSS), and Linear Correlation Coefficient (CC).The metrics are presented according to their characteristics.Source: B. Evaluation Metrics for Saliency… · p. 8

1) AUC and sAUC:

  • AUC classifies fixation and non-fixation points using a threshold, while sAUC uses fixations from other images as negatives to alleviate center-bias.Both AUC-based metrics have limitations in penalizing false positives.Source: 1) AUC and sAUC · p. 8

2) NSS:

  • NSS is the mean of the unit-normalized predicted saliency values at human eye-fixation locations.N is the total number of fixations.Source: 2) NSS · p. 8
  • CC measures linear correlation between predicted saliency maps and ground-truth density maps.The maps are treated as random variables; the equation uses covariance and standard deviation.Source: 2) NSS · p. 8

C. Implementation Details

The experiments compare probability-distribution losses with regression losses, describe training and inference speed, and visualize how DIM branches contribute to fused predictions.

C. Implementation Details

  • Training uses SALICON, then evaluates generalization on MIT1003 and fine-tunes on MIT1003 for MIT300 testing.Training uses Adam with batches of 10 and a learning rate of 10^-4, reduced by a factor of 0.1 every two epochs; MIT inputs are resized to 320 × 480 with zero padding.Source: C. Implementation Details · p. 8
  • Inference takes 0.02 s per 240×320 image and 0.03 s per 320×480 image on one GTX 1080 Ti GPU.Source: C. Implementation Details · p. 8

D. Loss Function Analysis

  • Probability-distribution distance losses generally outperform standard regression losses, and linear normalization is more compatible than softmax normalization for a given statistical distance.Using NSS loss alone yields very high NSS but weaker results on the other three evaluation metrics.Source: D. Loss Function Analysis · p. 8, Table IV · p. 9
  • A single NSS loss creates a metric trade-off because NSS favors fixation-like maps, whereas CC, AUC, and sAUC favor fixation-density-like maps.The authors conclude that one loss is difficult to optimize for strong performance on NSS and the other metrics simultaneously.Source: D. Loss Function Analysis · p. 9

E. Model Visualization

  • The three DIM branches focus on different image regions, and the largest-dilation branch learns center bias that can compensate for center regions missed by the other branches.The figure presents predictions from individual branches and their summed fusion.Source: E. Model Visualization · p. 9, Figure 5 · p. 9
  • Fusion also reduces occasional low-confidence center false alarms from the largest-dilation branch, combining incomplete branch predictions into more reliable results.Source: E. Model Visualization · p. 9

F. Model Ablation Analysis

The ablations examine backbone stride, decoder design, multi-scale modules, and DIM branches. They support the selected DRN, three-layer decoder, and three-branch dilated inception design.

1) Influence of the backbone network:

2) Influence of the decoder network:

3) Effectiveness of multi-scale features:

  • Adding multi-scale features boosts saliency performance; among tested modules, inception(d) and inception(e) perform best, and (e) is chosen for efficiency.The comparison includes IPN, skip-layer, inception, and ASPP; the advantage is especially notable with a plain ResNet backbone.Source: 3) Effectiveness of multi-scale features · p. 10, Table V · p. 10

4) Ablation analysis on DIM:

  • In the trained-model ablation, removing any DIM branch sharply reduces performance, while branch-specific biases complement one another in the fused result.Branch α favors sAUC and branch β favors NSS; the additional-decoder features remain decodable by the original decoder with only a small NSS drop.Source: Table VI · p. 10, 4) Ablation analysis on DIM · p. 10
  • Across trained DINet variants, adding branches raises evaluation performance, and larger dilation rates improve single-branch results; addition versus concatenation has limited impact.The authors attribute the overall gain to collaboration among the three parallel dilated branches.Source: Table VII · p. 10, 4) Ablation analysis on DIM · pp. 10–11

5) Influence of training image size:

The section reports results across saliency benchmarks, examines training resolution and NSS-oriented ensembling, and identifies failure cases in semantically rich scenes.

5) Influence of training image size:

  • Among three training resolutions, 320×480 yields the best-performing DINet model, while averaging their outputs gives the best scores in the model ablation analysis.The 320×480 model is selected for fine-tuning on MIT1003 for MIT300 evaluation.Source: 5) Influence of training image size · p. 11

6) Ensemble learning for improving NSS:

  • The single total-variation-loss model trails two state-of-the-art models on NSS, so the final SALICON submission averages DINet outputs trained with total variation and NSS losses.The ensemble is described as a compromise between NSS and the other three evaluation metrics.Source: 6) Ensemble learning for improving NSS · p. 11

G. Comparison with state-of-the-arts

V. CONCLUSION

DINet combines a dilated-inception encoder with a fully convolutional decoder and linear-normalization probability-distribution losses. The authors report strong benchmark performance and a lightweight, time-efficient model.

V. CONCLUSION

  • DINet uses a fully convolutional encoder-decoder architecture, with a dilated inception module capturing multi-scale saliency-influential factors.The model is trained end-to-end.Source: §V · p. 12
  • Linear normalization-based probability-distribution distance losses formulate saliency prediction as distribution prediction and outperform the tested standard regression and softmax-normalization losses.The conclusion attributes this comparison to the authors’ experiments.Source: §V · p. 13
  • The authors report outstanding benchmark performance and describe DINet as lightweight and time-efficient.The comprehensive comparison is listed in Table XIII.Source: §V · p. 13, Table XIII · p. 13
Loading 1904.03571v2…