PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Saliency Transformer

Nian Liu, Ni Zhang, Kaiyuan Wan, Ling Shao, Junwei Han

arXiv:2104.12099v2cs.CV

TL;DR

CNN-based saliency detectors have limited access to global long-range dependencies, motivating a convolution-free alternative. VST uses a pure transformer for RGB and RGB-D SOD with multi-level tokens, transformer-based upsampling, and task-related multi-task decoding. The authors report state-of-the-art results on RGB and RGB-D benchmark datasets and present the framework as a paradigm for transformer-based dense prediction.

  • Problem

    CNN-based saliency methods are limited in learning global long-range dependencies because their local sliding-window operations make crucial global cues difficult to exploit.

  • Method

    VST is a pure-transformer sequence-to-sequence model that processes image patches and uses multi-level tokens, RT2T upsampling, and task-related tokens with patch-task-attention.

  • Results

    VST achieves state-of-the-art results on both RGB and RGB-D salient object detection benchmark datasets.

  • Takeaways & Limitations

    The framework provides a new sequence-to-sequence perspective for SOD and a proposed paradigm for transformer-based dense prediction.

Abstract

from arXiv · show

Existing state-of-the-art saliency detection methods heavily rely on CNN-based architectures. Alternatively, we rethink this task from a convolution-free sequence-to-sequence perspective and predict saliency by modeling long-range dependencies, which can not be achieved by convolution. Specifically, we develop a novel unified model based on a pure transformer, namely, Visual Saliency Transformer (VST), for both RGB and RGB-D salient object detection (SOD). It takes image patches as inputs and leverages the transformer to propagate global contexts among image patches. Unlike conventional architectures used in Vision Transformer (ViT), we leverage multi-level token fusion and propose a new token upsampling method under the transformer framework to get high-resolution detection results. We also develop a token-based multi-task decoder to simultaneously perform saliency and boundary detection by introducing task-related tokens and a novel patch-task-attention mechanism. Experimental results show that our model outperforms existing methods on both RGB and RGB-D SOD benchmark datasets. Most importantly, our whole framework not only provides a new perspective for the SOD field but also shows a new paradigm for transformer-based dense prediction models. Code is available at https://github.com/nnizhang/VST.

1. Introduction

Saliency detection methods largely use CNNs, whose local processing limits global long-range dependency modeling. VST addresses this with a pure-transformer sequence-to-sequence model for both RGB and RGB-D SOD, including dense prediction and high-resolution token processing.

  • Motivation: CNN-based SOD methods struggle to exploit global cues because convolution extracts features through local sliding windows.Existing fully connected, pooling, and non-local additions incorporate global context only in selected layers while retaining the CNN architecture.
  • Motivation: Transformer self-attention relates different sequence positions and models long-range dependencies throughout stacked encoder and decoder layers.This makes transformers a natural framework for propagating global cues in saliency detection.
  • Approach: VST reframes SOD as sequence-to-sequence modeling with a pure transformer operating on image patches for both RGB and RGB-D inputs.The transformer propagates dependencies between image patches without convolution.
  • Contributions: VST introduces multi-level token fusion and transformer-based token upsampling to address coarse tokenization and high-resolution SOD prediction.These components are among the model’s stated contributions for adapting pure transformers to dense prediction.
  • Approach: Task-related saliency and boundary tokens with patch-task-attention form a multi-task decoder for dense prediction.The design jointly performs saliency and boundary detection while reducing computational costs relative to the described workflow.

2. Related Work

Prior SOD research is dominated by CNN-based encoders and decoders using feature fusion, attention, recurrent refinement, and multi-task learning. VST instead uses a unified pure-transformer design with token-based multi-task decoding and token upsampling for RGB and RGB-D SOD.

  • Deep Learning Based SOD: CNN-based SOD methods commonly use multi-level feature fusion, attention, recurrent refinement, and auxiliary multi-task objectives.Examples include fixation prediction, image captioning, and edge detection as auxiliary tasks.
  • Deep Learning Based SOD: RGB-D SOD methods fuse RGB and depth through feature operations, attention, dynamic convolution, graph networks, knowledge distillation, and cross-attention.These approaches target cross-modal integration of appearance and depth cues.
  • Deep Learning Based SOD: VST is presented as the first unified pure-transformer model for both RGB and RGB-D SOD, replacing the CNN-based sequence-to-sequence perspective.Its multi-task decoder uses task-related tokens and patch-task-attention rather than conventional CNN components.
  • Transformers in Computer Vision: VST applies patch-task-attention between patch tokens and task tokens because a single task token cannot directly produce dense predictions.The paper presents this strategy as potentially relevant to other transformer-based dense prediction tasks.
  • Transformers in Computer Vision: Unlike hybrid segmentation models, VST avoids convolution and bilinear upsampling while producing full-resolution predictions through transformer-based processing.The related hybrid approach uses CNN features, convolution, and bilinear upsampling for full-resolution segmentation.

3. Visual Saliency Transformer

VST consists of a T2T-ViT-based transformer encoder, a token convertor, and a multi-task transformer decoder.

  • Architecture: The VST architecture includes a transformer encoder based on T2T-ViT, a convertor between encoder and decoder token spaces, and a multi-task transformer decoder.These are identified as the model’s main components.

3.1. Transformer Encoder

The transformer encoder converts image patches into multi-level tokens, models spatial and long-range dependencies, and supplies decoder-ready representations. RGB-D inputs use separate RGB and depth encoders, while reverse T2T supports token upsampling.

  • Tokens to Token: T2T-ViT iteratively restructures and softly splits patch tokens to model local structure and produce new token sequences.The re-structurization step uses a transformer layer, while soft splitting unfolds overlapping image patches into tokens.
  • Tokens to Token: Transformer layers apply multi-head self-attention and multilayer perceptrons before tokens are reshaped into a 2D image representation.The reshaping recovers spatial structure after token transformation.
  • Tokens to Token: Overlapping patch splitting introduces local correspondence between neighboring patches, while repeated T2T aggregation progressively reduces token sequence length.The reduction occurs when the stride is smaller than the patch size minus one.
  • Token Upsampling: Reverse T2T upsamples tokens by expanding each token into multiple sub-tokens, providing a transformer-based alternative to conventional upsampling.The method is designed to recover higher-resolution token representations for dense prediction.
  • Encoder Design: The encoder produces multi-level tokens, adds sinusoidal positional embeddings, and applies transformer layers to model long-range dependencies among final tokens.The final encoder tokens are projected into the decoder embedding dimension before long-range modeling.
  • RGB-D Encoding: RGB SOD uses one transformer encoder, whereas RGB-D SOD uses separate RGB and depth transformer encoders to extract modality-specific patch tokens.The RGB-D configuration follows a two-stream design at the encoder stage.

3.2. Transformer Convertor

The transformer convertor converts encoder patch tokens into decoder-space tokens and supports cross-modal long-range interaction for RGB-D SOD. For RGB SOD, standard transformer layers produce the converted patch sequence.

  • The convertor maps encoder patch tokens from encoder space into decoder space, producing converted patch tokens T C.
  • The cross-modality-attention layer propagates long-range dependencies between RGB and depth tokens by using queries from one modality with keys and values from the other.
  • The cross-modality-attention uses multi-head attention, feed-forward networks, residual connections, and layer normalization within each CMT layer.
  • After each CMT layer, a standard transformer layer separately enhances RGB and depth token sequences before the alternating process continues.
  • After L_C alternating CMT and transformer layers, RGB and depth tokens are concatenated and projected into final converted tokens; RGB SOD uses L_C standard transformer layers.

3.3. Multi-task Transformer Decoder

The decoder addresses transformer dense-prediction resolution and task-sharing challenges through reverse token upsampling, multi-level token fusion, and task-related patch attention. It jointly predicts saliency and boundaries at high resolution.

  • Token Upsampling and Multi-level Token Fusion: Direct prediction from T C is limited by its coarse token length, so the decoder first upsamples tokens before dense prediction.
  • Token Upsampling and Multi-level Token Fusion: Reverse T2T expands each token into multiple overlapping sub-tokens, providing transformer-based token upsampling instead of bilinear feature-map interpolation.
  • Token Upsampling and Multi-level Token Fusion: The decoder progressively fuses low-level T2 and T1 tokens from the RGB encoder with upsampled tokens to recover fine-grained local structure.
  • Multi-task Transformer Decoder: Patch-task-attention combines final decoder patch tokens with saliency and boundary tokens, then produces 2D saliency and boundary maps after upsampling.
  • Multi-task Transformer Decoder: Task-related saliency and boundary tokens interact with decoder patch tokens through transformer layers, learning image-dependent task embeddings.

4. Experiments

VST is evaluated across six RGB and nine RGB-D benchmark datasets using standard saliency metrics and complexity measures. Ablations support its component design, while comparisons report stronger performance than prior CNN-based methods on both tasks.

  • Datasets and Evaluation Metrics: VST is evaluated on six RGB datasets and nine RGB-D datasets using Structure-measure, maxF, Emax, MAE, MACs, and parameter counts.
  • Ablation Study: CMT improves performance over concatenation fusion, especially on the NJUD and LFSD RGB-D datasets.
  • Ablation Study: RT2T improves performance over bilinear upsampling after bilinear upsampling has already improved full-resolution saliency prediction.
  • Ablation Study: Fusing T1 and T2 further improves performance by supplying low-level fine-grained information to the decoder.
  • Ablation Study: Boundary detection improves SOD on three of four datasets, while TMD outperforms C2D on three datasets with 17.70 G versus 28.27 G computational cost.
  • Comparison with State-of-the-Art Methods: VST outperforms prior state-of-the-art CNN-based models on both RGB and RGB-D benchmarks with comparable parameters and relatively small MACs.

5. Conclusion

The paper presents VST as a unified pure-transformer, sequence-to-sequence model for RGB and RGB-D SOD. Its token upsampling, multi-level fusion, and token-based multitask decoder address dense prediction and achieve state-of-the-art results without heavy computational costs.

  • VST rethinks SOD as sequence-to-sequence modeling with a unified pure transformer for both RGB and RGB-D inputs.
  • The framework combines token upsampling, multi-level patch-token fusion, and task-related patch-task attention for joint saliency and boundary detection.
  • VST achieves state-of-the-art RGB and RGB-D SOD results without relying on heavy computational costs.

6. Supplementary materials

Ablations on RGB and RGB-D datasets support the effectiveness of VST’s components, while visual comparisons show robustness in challenging scenes. Layer studies select configurations balancing performance and computational cost.

  • RGB SOD ablation: Ablations on four RGB datasets show that RT2T improves over bilinear upsampling, multi-level token fusion improves all datasets, and TMD outperforms a conventional two-stream decoder.TMD also improves performance on all four datasets.
  • Layer number study: Reducing selected decoder layer counts produces comparable performance with lower computational cost, while the chosen configuration balances performance and acceptable computation.The best performance among tested convertor settings occurs in row VII.
  • Visual comparison: Visual comparisons report that VST handles large salient objects, cluttered backgrounds, and similar-appearance foregrounds and backgrounds, while predicting clear salient-object boundaries.The comparisons cover both RGB and RGB-D methods.
Loading 2104.12099v2…