Source-linked AI summary
SalGAN: Visual Saliency Prediction with Generative Adversarial Networks
Junting Pan, Cristian Canton Ferrer, Kevin McGuinness, Noel E. O'Connor, Jordi Torres, Elisa Sayrol, Xavier Giro-i-Nieto
TL;DR
Visual saliency prediction faces inconsistent training signals because saliency metrics can disagree. SalGAN combines a BCE-trained convolutional saliency predictor with adversarial discrimination, achieving state-of-the-art performance across datasets and almost all metrics. The approach is presented as generic beyond the VGG-16-based model used here.
Problem
Choosing an optimal saliency-prediction loss remains open because different saliency metrics can disagree and yield inconsistent model comparisons.
Method
SalGAN combines a deep convolutional saliency predictor trained with BCE and a discriminator that distinguishes generated saliency maps from ground-truth maps conditioned on the input image.
Results
SalGAN achieves state-of-the-art performance on MIT300 and SALICON for almost all evaluation metrics, improving all but one metric over cross-entropy-only training.
Takeaways & Limitations
Adversarial training can complement BCE content loss in a single-tower, single-task saliency model, and the approach could extend to other saliency models.
Takeaways & Limitations
Adversarial training does not improve normalized scanpath saliency, possibly because smoother predictions increase false positives to which NSS is sensitive.
Abstract
from arXiv · showhide
We introduce SalGAN, a deep convolutional neural network for visual saliency prediction trained with adversarial examples. The first stage of the network consists of a generator model whose weights are learned by back-propagation computed from a binary cross entropy (BCE) loss over downsampled versions of the saliency maps. The resulting prediction is processed by a discriminator network trained to solve a binary classification task between the saliency maps generated by the generative stage and the ground truth ones. Our experiments show how adversarial training allows reaching state-of-the-art performance across different metrics when combined with a widely-used loss function like BCE. Our results can be reproduced with the source code and trained models available at https://imatge-upc.github.io/saliency-salgan-2017/.
1. Introduction
Visual saliency prediction uses attention maps, but selecting a training loss remains difficult because saliency metrics can disagree. SalGAN addresses this with adversarial training alongside BCE and reports state-of-the-art performance across datasets and metrics.
- Visual saliency identifies image locations that attract human attention and supports machine vision tasks such as object recognition.
- Saliency maps aggregate fixation data and smooth it with a Gaussian kernel to estimate each pixel’s probability of attracting attention.
- Choosing an optimal training loss remains open because saliency metrics define saliency differently and can produce inconsistent model comparisons.
- SalGAN combines a saliency-map generator with a discriminator that distinguishes predicted maps from ground-truth maps for the same image.
- Adversarial training with BCE achieves state-of-the-art performance on MIT300 and SALICON for almost all evaluation metrics.
- The source code and trained models are available online for reproducing the results.
2. Related work
Related work spans traditional, deep, and metric-optimized saliency models. SalGAN instead incorporates a high-level adversarial loss into conventional training and reports state-of-the-art performance on MIT300 and SALICON.
- Traditional saliency models combine low-level, mid-level, and high-level visual features or use graph-based formulations over image maps.
- Deep learning approaches introduced learned representations, multi-resolution processing, and models trained from image regions or fixation-related data.
- Researchers have optimized saliency networks with differentiable metrics such as NSS, CC, SIM, and KL divergence.
- SalGAN takes a different approach by incorporating high-level adversarial loss into conventional saliency prediction training.
- The proposed approach achieves state-of-the-art performance on both MIT300 and SALICON by a clear margin.
3. Architecture
SalGAN trains two competing convolutional networks: an encoder-decoder generator that predicts saliency maps and a discriminator that classifies generated versus real maps. The generator uses convolutional encoding and upsampling-based decoding.
- SalGAN training uses a generator network to produce saliency maps and a discriminator network to distinguish generated maps from real ones.
- The generator uses a convolutional encoder-decoder architecture with pooling in the encoder and upsampling followed by convolutions in the decoder.
- The encoder follows VGG-16’s architecture without its final pooling and fully connected layers, using ImageNet initialization while modifying only its last two convolutional groups.
- A final 1 × 1 sigmoid convolution produces a saliency map at the input image’s resolution.
- The discriminator contains six 3x3 convolutions, three pooling layers, and three fully connected layers, ending in a sigmoid output.
- Table 1 documents the generator architecture, while Table 2 documents the discriminator architecture.
4. Training
SalGAN combines pixel-wise content loss with adversarial training so predicted saliency maps match ground truth while becoming difficult for a discriminator to distinguish.
- Loss design: SalGAN combines content and adversarial losses to train the saliency prediction network.The content term compares predicted and ground-truth maps, while the adversarial term uses discriminator output on the generated map.
- Content loss: MSE serves as a baseline content loss that computes Euclidean distance between predicted and ground-truth saliency maps.The paper contrasts this baseline with BCE-based training.
- Content loss: Binary cross entropy treats each pixel prediction as an independent probability after applying an element-wise sigmoid.Ground-truth saliency values are normalized to [0, 1], and BCE averages the individual pixel-wise binary cross entropies.
- Adversarial loss: The discriminator receives the source image together with either predicted or ground-truth saliency and classifies the map as fake or real.Its loss combines BCE for real pairs with BCE for generated pairs labeled fake.
- Adversarial loss: Combining discriminator error with ground-truth cross entropy improved adversarial-training stability and convergence rate.The generator uses the adversarial objective that encourages the discriminator to classify generated saliency as real.
- Training procedure: Training begins with 15 BCE-only epochs before alternating saliency-network and discriminator updates during adversarial training.The networks are trained on 15,000 SALICON training images with batches of 32.
5. Experiments
Experiments evaluate SalGAN’s training configuration, adversarial gain, qualitative behavior, and benchmark performance. BCE and downsampling establish the baseline, while adversarial training improves most metrics and yields competitive results on SALICON and MIT300.
- 5.1. Non-adversarial training: BCE consistently improves all metrics over MSE, supporting saliency prediction as multiple binary classification problems rather than standard regression.The paper relates cross-entropy minimization to KL-divergence minimization when predictions and targets are interpreted as probabilities.
- 5.1. Non-adversarial training: Downsampling saliency maps by a factor of 1/4 reduces training and testing resources without decreasing performance, and can improve it.The selected maps are 64 × 48 in the reported architecture.
- 5.2. Adversarial gain: α = 0.005 gives the best Information Gain after a logarithmic hyperparameter search.The adversarial-loss weight was selected using Information Gain.
- 5.2. Adversarial gain: After 100 and 120 epochs, combined GAN/BCE training substantially improves BCE on five of six metrics, while BCE alone largely plateaus on AUC metrics.The single metric not improved by adversarial training is NSS, which may be sensitive to false positives from smoother, more spread-out predictions.
- 5.3. Comparison with the state-of-the-art: SalGAN improves or equals competing models on at least one metric in both the SALICON and MIT300 benchmarks.SALICON uses crowdsourced mouse clicks, whereas MIT300 uses eye trackers; SalGAN is trained on 15,000 SALICON training and validation images.
- 5.2. Adversarial gain: Adversarial training produces smoother, simpler saliency-map level sets than BCE-only training and can detect salient regions often missed by existing algorithms.The qualitative comparisons include MIT300 examples and close-up BCE-versus-adversarial outputs.
6. Conclusions
SalGAN combines adversarial training with a BCE content loss to achieve strong saliency prediction performance. The approach improves nearly all saliency metrics over continued cross-entropy training and may generalize beyond the VGG-16-based model used here.
- SalGAN achieves state-of-the-art performance with a simple encoder-decoder architecture and adversarial training.
- Adversarial training improved all but one saliency metric compared with further training on cross entropy alone.
- A BCE-based content loss initializes the saliency network and stabilizes adversarial training as a regularization term.
- The adversarial training approach is generic and could improve other saliency models beyond the VGG-16-based encoder-decoder used here.
- SalGAN detects salient regions that the BCE model often misses when multiple salient regions appear.