PaperlayerView exact revision on arXiv

Source-linked AI summary

TranSalNet: Towards perceptually relevant visual saliency prediction

Jianxun Lou, Hanhe Lin, David Marshall, Dietmar Saupe, Hantao Liu

arXiv:2110.03593v3cs.MMcs.CV

TL;DR

CNNs lack sufficient long-range context for saliency prediction intended to reflect human viewing. TranSalNet adds transformer components to CNNs, and the authors report superior benchmark performance, particularly on perception-based metrics.

  • Problem

    CNN-based saliency models have limited capacity for long-range contextual encoding, although human visual attention uses local and long-range information.

    Source: §1 · p. 2

  • Method

    TranSalNet integrates transformer encoders into a CNN-based architecture to capture long-range spatial dependencies.

    Source: §1 · p. 3

  • Results

    TranSalNet models achieved superior performance on public benchmarks and competitions, particularly on perception-based saliency metrics.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    The authors propose TranSalNet_Res and TranSalNet_Dense as human-like models for evaluating relative image-region importance in retargeting, compression, transmission, and visual quality assessment.

    Source: §4.5.4 · p. 12

    Directly adding low-level encoder features to the decoder may produce object or text shapes that are inconsistent with ground truth.

    Source: §4.5.4 · p. 12

Abstract

from arXiv · show

Visual saliency prediction using transformers - Convolutional neural networks (CNNs) have significantly advanced computational modelling for saliency prediction. However, accurately simulating the mechanisms of visual attention in the human cortex remains an academic challenge. It is critical to integrate properties of human vision into the design of CNN architectures, leading to perceptually more relevant saliency prediction. Due to the inherent inductive biases of CNN architectures, there is a lack of sufficient long-range contextual encoding capacity. This hinders CNN-based saliency models from capturing properties that emulate viewing behaviour of humans. Transformers have shown great potential in encoding long-range information by leveraging the self-attention mechanism. In this paper, we propose a novel saliency model that integrates transformer components to CNNs to capture the long-range contextual visual information. Experimental results show that the transformers provide added value to saliency prediction, enhancing its perceptual relevance in the performance. Our proposed saliency model using transformers has achieved superior results on public benchmarks and competitions for saliency prediction models. The source code of our proposed saliency model TranSalNet is available at: https://github.com/LJOVO/TranSalNet

1 Introduction

TranSalNet combines transformers with CNNs to model long-range context relevant to human visual attention. The authors report superior performance on MIT300 and SALICON benchmarks.

1 Introduction

  • Human saliency reflects interactions between local and long-range information, which may help prediction maps better reflect human perception.The paper describes peripheral vision as providing non-detailed, long-range information that modulates local saliency maxima.Source: §1 · p. 2
  • CNN kernels process local pixel subsets, limiting fully CNN-based models’ ability to obtain long-range context.The paper relates this limitation to the human visual system’s use of both foveal and peripheral vision.Source: §1 · p. 2
  • Prior approaches used multi-scale CNN representations or LSTM-based long-range modelling, while transformer use in saliency prediction remained underexplored.The paper notes challenges in optimally fusing multi-scale information and says further work was needed to close the gap with human perception.Source: §1 · p. 3
  • TranSalNet integrates transformers into a CNN-based architecture and achieves superior performance on the MIT300 benchmark and SALICON challenge.The paper presents the model as supporting perceptually more relevant saliency prediction.Source: §1 · p. 3

2 Related work

The related-work review covers deep-learning saliency models, evaluation metrics, transformers in vision, and methods for representing multi-scale and long-range information.

2 Related work

  • The review surveys deep-learning saliency methods, evaluation, vision transformers, and multi-scale and long-range approaches to saliency prediction.It frames these topics as the context for TranSalNet’s use of transformers in saliency prediction.Source: §2 · p. 3

2.1 Deep learning-based visual saliency prediction

  • Deep-learning saliency models progressed from CNN feature extractors to more sophisticated encoders and combinations of pretrained backbones.The review describes eDN, DeepGaze, EML-NET, and findings on backbone choice and concatenating pretrained networks.Source: §2.1 · p. 3

2.2 Evaluation methods for saliency models

  • Saliency metrics do not necessarily serve every application equally, and the authors evaluate single-map classical models while distinguishing perception-based from non-perception-based metrics.The perception-based group is NSS, CC, and SIM; the non-perception-based group includes sAUC, AUC, and KLD.Source: §2.2 · pp. 3–4

2.3 Transformer in visual tasks

  • Vision transformers model long-range dependencies, but salient-object detection predicts binary object masks, unlike saliency prediction’s human-fixation density maps.The review says transformer use in visual saliency prediction had not been fully explored.Source: §2.3 · p. 4

2.4 Multi-scale and long-range information in visual saliency prediction

  • Prior saliency models used multi-scale features and long-range modules; TranSalNet combines multi-scale CNN representations with transformer-enhanced long-range context.Its CNN encoder supplies three feature-map scales to transformer encoders, with the decoder fusing the enhanced representations through skip-connections.Source: §2.4 · pp. 4–5, Figure 2 · p. 5

3 The proposed model

TranSalNet combines CNN feature extraction and multi-scale transformer encoders to add long-range context before a CNN decoder predicts saliency maps.

3 The proposed model

  • TranSalNet passes multi-scale CNN features through transformer encoders for long-range context, then fuses them in a CNN decoder to predict saliency.The encoder extracts three feature-map scales, and the decoder fuses the context-enhanced maps.Source: §3 · p. 5

3.1 The CNN encoder

  • The model uses ResNet-50 or DenseNet-161 as its CNN encoder and excludes shallow feature maps after they produced undesired saliency-map artifacts.TranSalNet_Res uses ResNet-50; TranSalNet_Dense uses DenseNet-161.Source: §3.1 · pp. 5–6

3.2 The transformer encoder

  • Three CNN feature-map scales enter transformer encoders, whose attention and MLP blocks enhance their long-range contextual information.The encoders use position embeddings; the first two use 12-head attention and the third uses 8-head attention.Source: §3.2 · p. 6

3.3 The CNN decoder

  • The CNN decoder restores input resolution and fuses upsampled features with corresponding transformer outputs through element-wise products.Its convolutional blocks produce pixel-level saliency predictions.Source: §3.3 · p. 6

3.4 Loss function

  • Training combines NSS, KLD, CC, and SIM losses, with weights empirically adjusted to balance their contributions and evaluation metrics.The default weights are −1, 10, −2, and −1, respectively.Source: §3.4 · p. 7

4.1 Datasets

The study uses four benchmark datasets for training and evaluation, with different image sources and annotation or test-set availability.

4.1 Datasets

  • The study trains and evaluates on SALICON (10,000 training, 5,000 validation, and 5,000 test images), CAT2000 (2,000), MIT1003 (1,003), and MIT300 (300).SALICON and MIT300 test annotations are unpublished.Source: §4.1 · p. 7
  • SALICON records attention through mouse clicks, whereas CAT2000, MIT1003, and MIT300 use eye-tracking data or benchmarks.CAT2000 has eye-tracking data from 24 observers per image; MIT1003 has data from 15 observers per image.Source: §4.1 · p. 7
  • CAT2000 contains 20 image categories, while MIT300 is used as the test set of the MIT/Tübingen benchmark.CAT2000 includes 100 images in each category.Source: §4.1 · p. 7

4.2 Evaluation Metrics

The paper describes six common metrics for saliency agreement and separates them by ground-truth representation and their stated fit with human judgments.

4.2 Evaluation Metrics

  • CC, SIM, KLD, NSS, AUC, and sAUC measure agreement with ground truth; KLD is better nearer zero, while higher values are better for the other five.CC, SIM, and KLD are distribution-based; NSS, AUC, and sAUC are location-based.Source: §4.2 · p. 8
  • The paper groups NSS, CC, and SIM as perception-based metrics and sAUC, AUC, and KLD as non-perception-based metrics.The latter label does not mean those metrics fail to measure gaze behavior; they may assess properties such as salient-object detection.Source: §4.2 · p. 8
  • The cited metric guidance favors NSS or SIM when relative importance across image regions matters, and AUC or KL for target-detection applications.The paper focuses on perceptual relevance while also evaluating general model performance.Source: §4.2 · p. 8

4.3 Setup

Models were initialized with ImageNet-pretrained weights and trained on SALICON before testing or training on the target datasets. MIT1003 and CAT2000 used fixed 10-fold splits, with standardized input sizing and training settings.

4.3 Setup

  • ImageNet-pretrained weights initialized each model, followed by training on SALICON’s 10,000 training images before SALICON testing and MIT1003/CAT2000 training.The best model on the SALICON validation set was selected for further testing and training.Source: §4.3 · p. 8
  • MIT1003 and CAT2000 used fixed 10-fold cross-validation, with one subset for testing, one for validation, and eight for training.The reported performance aggregates ten test runs; subsets contained around 100 MIT1003 images or 200 CAT2000 images each.Source: §4.3 · p. 8
  • Inputs were resized and padded to 384×288 pixels; Adam training used a batch size of 4 for 30 epochs with early stopping patience of 5.The learning rate began at 1 ×10−5 and was multiplied by 0.1 every 3 epochs.Source: §4.3 · p. 8

4.4 Ablation study

The ablation study compares transformer encoders, skip-connections, and combined loss across nine variants. Their combination improves performance, while replacing ResNet-50 with DenseNet-161 further improves the model.

4.4 Ablation study

  • Adding a transformer encoder improves overall performance in the majority of instances, with consistent gains on CC, SIM, and NSS.The comparison is between BaseNet/BaseNet(LCB) and BaseNet+/BaseNet+(LCB).Source: §4.4 · p. 9
  • Adding skip-connections to BaseNet variants improves performance in most ablation-study instances.The comparison is between BaseNet/BaseNet(LCB) and SkipNet/SkipNet(LCB).Source: §4.4 · p. 9
  • Combining transformer encoders and skip-connections gives the decoder multi-scale features with enhanced long-range context and boosts all perception-based metric instances.Performance also improves on most non-perception-based metric instances.Source: §4.4 · p. 9
  • In the ablation study, LCB-trained model variations achieve higher performance than LBCE-trained variations on the majority of saliency metrics, and TranSalNet_Res outperforms TranSalNet_Res_BCE on all instances.Source: §4.4 · p. 9
  • Replacing TranSalNet_Res’s ResNet-50 backbone with DenseNet-161 further improves performance.TranSalNet_Dense is the DenseNet-161 variant.Source: §4.4 · pp. 9–10

4.5 Comparison with state-of-the-art methods

TranSalNet performs strongly across MIT1003/CAT2000, MIT300, and LSUN’17, especially on perception-based metrics. The discussion connects those metrics to image-region evaluation and notes architectural trade-offs.

4.5.1 On MIT1003 and CAT2000 datasets

  • Both TranSalNet variants lead on all perception-based metrics across MIT1003 and CAT2000, with competitive non-perception scores and maps closer to ground truth.The comparison uses 10-fold cross-validation and SALICON-pretrained weights; TranSalNet_Res has similar feature-extractor capacity to five comparison models, while DenseNet-161 further enhances its performance.Source: §4.5.1 · p. 10, Figure 4 · p. 11

4.5.2 On MIT300 competition

  • On MIT300, both variants rank first or second on perception-based metrics except TranSalNet_Res on NSS, while remaining competitive on sAUC and AUC.The comparison is limited to non-probabilistic classical models because probabilistic and non-probabilistic models are evaluated under different assumptions.Source: §4.5.2 · p. 11, Table 4 · p. 12

4.5.3 On LSUN’17 competition

  • On LSUN’17, both TranSalNet variants achieve superior results on perception-based metrics and promising results on other metrics.The competition uses a unified evaluation process in which each model produces one saliency map per image, without separate treatment by model type.Source: §4.5.3 · p. 12, Table 5 · p. 12

4.5.4 Discussion

  • CC and SIM are identified as perception-aligned metrics, supporting TranSalNet for evaluating relative image-region importance in retargeting, compression/transmission, and visual quality assessment.The discussion cites prior work identifying CC and SIM as especially appropriate for image-quality assessment.Source: §4.5.4 · p. 12
  • Shallow-encoder skips into decoder blocks 4 and 5 may introduce object or text shapes inconsistent with ground truth, potentially interfering with saliency prediction.The authors contrast this effect with the common use of U-Net-style skip connections in other computer-vision tasks; Figure 5 illustrates the reported artifacts.Source: §4.5.4 · p. 12, Figure 5 · p. 13
  • CC and SIM tend to increase with more MSA heads but saturate at 12, 12, and 8 heads for encoders 1, 2, and 3, the chosen compute–performance trade-off.The study reports mean results across three test runs.Source: §4.5.4 · p. 13, Figure 6 · p. 13

5 Conclusion

TranSalNet integrates transformers into CNNs to capture long-range spatial information at multiple perceptual levels. The reported ablation and benchmark results support added perceptual relevance, with potential applications in image processing.

5 Conclusion

  • TranSalNet integrates transformers into CNNs to capture long-range spatial information at multiple perceptual levels.The conclusion describes the model as predicting saliency maps intended to be perceptually close to ground truth.Source: §5 · p. 13
  • An ablation study demonstrates transformer encoders’ contribution to a CNN model, particularly their added value for perceptual relevance.Source: §5 · p. 13
  • The proposed models achieve superior performance on public benchmarks and competitions, notably on perception-based saliency metrics.Source: §5 · p. 13
  • The authors identify perceptually more relevant saliency models as having potential to advance image-processing applications.Source: §5 · p. 13

Related papers

Loading 2110.03593v3…