PaperlayerView exact revision on arXiv

Source-linked AI summary

CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection

Youwei Pang, Xiaoqi Zhao, Lihe Zhang, Huchuan Lu

arXiv:2112.02363v3cs.CV

TL;DR

Bi-modal salient object detection must integrate complementary RGB and depth or thermal cues while overcoming convolutional locality and costly global attention. CAVER uses a top-down transformer path with view-mixed attention and parameter-free patch-wise token re-embedding, and extensive experiments show it outperforms recent methods across RGB-D and RGB-T benchmarks.

  • Problem

    Bi-modal SOD methods rely on convolution and complex fusion structures, while convolutional locality can limit cross-modal global information integration.

  • Method

    CAVER uses a top-down transformer-based propagation path with view-mixed attention, cross-modal integration units, and parameter-free PTRE for efficient token interactions.

  • Results

    CAVER outperforms recent methods on seven RGB-D SOD datasets and three RGB-T SOD datasets.

  • Takeaways & Limitations

    The proposed components support a simple two-stream encoder-decoder framework for global and local feature modeling in bi-modal SOD.

Abstract

from arXiv · show

Most of the existing bi-modal (RGB-D and RGB-T) salient object detection methods utilize the convolution operation and construct complex interweave fusion structures to achieve cross-modal information integration. The inherent local connectivity of the convolution operation constrains the performance of the convolution-based methods to a ceiling. In this work, we rethink these tasks from the perspective of global information alignment and transformation. Specifically, the proposed \underline{c}ross-mod\underline{a}l \underline{v}iew-mixed transform\underline{er} (CAVER) cascades several cross-modal integration units to construct a top-down transformer-based information propagation path. CAVER treats the multi-scale and multi-modal feature integration as a sequence-to-sequence context propagation and update process built on a novel view-mixed attention mechanism. Besides, considering the quadratic complexity w.r.t. the number of input tokens, we design a parameter-free patch-wise token re-embedding strategy to simplify operations. Extensive experimental results on RGB-D and RGB-T SOD datasets demonstrate that such a simple two-stream encoder-decoder framework can surpass recent state-of-the-art methods when it is equipped with the proposed components. Code and pretrained models will be available at \href{https://github.com/lartpang/CAVER}{the link}.

I. INTRODUCTION

CAVER reframes bi-modal salient object detection around transformer-based global alignment and cross-modal information propagation. It combines view-mixed attention with parameter-free token re-embedding to improve efficiency while retaining local and global cues.

  • Motivation: Bi-modal inputs such as depth or thermal imagery provide complementary structure and shape cues for challenging salient-object scenes.These modalities help distinguish objects in complex, low-contrast, or obscured scenes.
  • Motivation: Convolution-based architectures may encounter a performance bottleneck because convolution is localized and learned parameters are fixed.
  • CAVER: CAVER introduces a transformer-based top-down multi-level structure using self- and cross-attention for feature alignment and cross-modal fusion.The design can be assembled with CNN feature extractors.
  • Efficiency: PTRE aggregates pixel-wise tokens into patch-wise tokens before matrix multiplication, reducing attention computation and memory costs for multi-scale high-resolution features.The operation is parameter-free and reduces computation by p^2 and memory by p^4, where p is the patch side length.
  • CAVER: CAVER rethinks bi-modal SOD as sequence-to-sequence modeling and builds a top-down propagation path with view-mixed attention across spatial and channel views.
  • Results: Extensive experiments show that the proposed model outperforms recent methods on seven RGB-D and three RGB-T SOD datasets.

II. RELATED WORK

Related work covers visual attention, bi-modal SOD fusion strategies, and attention-based models. CAVER differs by emphasizing transformer-based decoder design for cross-modal integration while addressing long-range interaction costs.

  • Visual Attention: SOD locates and segments visually attractive regions, while related visual-attention work also studies where observers look through eye-fixation prediction.
  • Bi-modal SOD: RGB-D and RGB-T SOD add depth or thermal infrared information to provide a more comprehensive scene understanding.
  • Bi-modal SOD: Existing bi-modal methods are commonly categorized as early, intermediate, or late fusion according to their cross-modal fusion strategy.
  • Attention-Based Models: Convolution-based attention strategies improve flexibility but do not model long-range dependencies well, while prior non-local approaches remain limited by CNN architectures and computational cost.
  • Transformer-Based Models: CAVER focuses on transformer-based decoder design for RGB-D and RGB-T tasks, using a simple encoder-decoder architecture to decode and align two modalities.

III. OUR METHOD: CAVER

CAVER is a transformer-based network for integrating RGB and depth/thermal features across modalities and scales. Its architecture includes patch-wise view-mixed attention blocks and a saliency predictor that produces the final map.

  • CAVER uses a transformer-based information propagation path to integrate RGB and depth/thermal features.
  • The network includes patch-wise view-mixed self-attention and cross-attention blocks.
  • Features are converted between 2D feature maps and 1D feature sequences within the architecture.
  • A saliency predictor maps refined features to a single-channel saliency map using interpolation, convolution layers, and sigmoid activation.

B. Transformer-based Information Propagation Path (TIPP)

The TIPP integrates top-down RGB and depth/thermal features through four cascaded cross-modal integration units. It combines same-scale modality features with adjacent higher-level outputs while using compact embeddings.

  • TIPP consists of four cascaded cross-modal integration units for processing top-down multi-scale features.
  • The fourth CMIU performs only cross-modal integration, whereas the other units also integrate an adjacent higher-level output.
  • Each CMIU converts its input maps into D-dimensional pixel-wise embeddings, with D empirically set to 64.
  • The RGB and depth/thermal inputs are self-reinforced before being integrated through inter-modal cross-attention.
  • The inter-modal cross-attention output is combined with an upsampled higher-level feature when applicable.

C. Intra-Modal/Cross-Scale Self-Attention (IMSA/CSSA)

IMSA and CSSA use transformer attention with convolutional feed-forward processing, while PTRE and view-mixed attention address quadratic cost and add channel-view feature reconstruction.

  • Attention block structure: IMSA and CSSA share a structure combining PV-MHSA, Conv-FFN, normalization layers, and residual connections.
  • Attention formulation: Flattened image features contain N = H × W pixel tokens and embedding dimension D.
  • Attention formulation: MHSA aligns features by computing correlations between queries, keys, and values across the global sequence.
  • Efficiency and views: The attention matrix has quadratic complexity O(N^2), limiting the processing of multi-scale high-resolution features.
  • Efficiency and views: PTRE changes attention matrix operations from pixel-wise to patch-wise form and reduces complexity by a factor of p^2.
  • Efficiency and views: VMA executes spatial- and channel-view attention in parallel and combines their outputs using learnable weights α and β.
  • Efficiency and views: With D = 64 and Nh = 2, VMA reduces computational and memory costs when N exceeds 65.
  • Local feature processing: Conv-FFN restores local-context processing by replacing position-wise transformations with convolutional operations.

D. Inter-Modal Cross-Attention (IMCA)

IMCA uses two-stream multi-head cross-attention to associate RGB with depth/thermal information. Its outputs are fused across spatial and channel views and then combined with Conv-FFN results.

  • IMCA contains RGB and depth/thermal streams whose information sources differ for queries versus keys and values.
  • The cross-attention block associates and interacts information between modalities using inputs from separate self-attention blocks.
  • RGB-stream outputs combine spatial- and channel-view cross-attention results using learnable weights αrgb and βrgb.

IV. EXPERIMENTS

Experiments evaluate CAVER on seven RGB-D and three RGB-T benchmarks, using dataset-specific comparisons and computational measurements on a single GPU.

  • Datasets: Experiments cover seven RGB-D and three RGB-T benchmarks widely used for bi-modal salient object detection.
  • RGB-D SOD: The RGB-D evaluation includes NJUD, NLPR, SIP, STEREO1000, SSD, LFSD, and DUTRGBD datasets.
  • Evaluation Setup: Tables I–III compare recent RGB-D and RGB-T methods across the benchmark datasets, while Table IV reports FLOPs, parameters, and FPS.
  • Evaluation Setup: PR and Fβ-Threshold curves provide additional performance comparisons for RGB-D and RGB-T SOD methods.
  • RGB-T SOD: The RGB-T evaluation uses VT821, VT1000, and VT5000-TE, with VT5000 training restricted to its 2500-sample training split.

B. Evaluation Metrics

The evaluation uses complementary metrics for structural similarity, pixel error, precision-recall quality, and weighted saliency accuracy, supplemented by PR and threshold curves.

  • S-measure evaluates region-aware and object-aware structural similarity between predicted saliency maps and ground truth.
  • MAE measures average absolute pixel error, while Fβ evaluates region-based similarity from precision and recall.
  • E-measure combines image-level statistics with local pixel matching, and weighted F-measure weights precision and recall for exactness and completeness.
  • PR and Fβ-Threshold curves present a comprehensive comparison of model performance.

C. Implementation Detail

Models use ImageNet-pretrained backbones with randomly initialized remaining structures, standardized optimization, augmentation, resizing, and hybrid-loss supervision.

  • The backbone is initialized from ImageNet, while remaining structures use PyTorch’s default random initialization.
  • Training runs for 100 epochs with batch size 8, SGD momentum 0.9, weight decay 0.0005, and cosine learning-rate scheduling from 0.005.
  • Affine transforms, horizontal flipping, and color jittering are used during training to avoid over-fitting.
  • At test time, RGB and depth or thermal images are resized to 256 × 256, predictions return to original size, and hybrid loss supervises training.

D. Comparison

CAVER is compared with recent state-of-the-art methods through quantitative, qualitative, curve-based, and component-ablation analyses, including efficiency measurements.

  • Quantitative Comparison: Quantitative comparisons report CAVER as best across ten datasets, two tasks, and five metrics.
  • Quantitative Comparison: 0.912 Sm, 0.035 M, 0.947 Em, and 0.914 Fβ are reported for Ours101(II) on RGB-D SOD.
  • Quantitative Comparison: 0.98% Sm, 0.60% Fωβ, 5.43% M, 0.36% Em, and 0.77% Fβ are reported as relative gains over TriTransNet.
  • Quantitative Comparison: The RGB-T comparison also reports consistent and obvious performance gains, while PR and Fβ curves are positioned more upward for the proposed methods.
  • Ablation Analysis: Ablation analyses examine different components, channel and spatial attention branches, self-attention variants, and feed-forward network variants.
  • Qualitative Comparison: Visual comparisons cover varied object sizes, boundaries, brightness conditions, out-of-bounds objects, and challenging backgrounds.

E. Ablation Analysis

The ablation studies show that cross-modal attention, self-attention, convolutional locality, view-mixed attention, and patch re-embedding each contribute to CAVER’s effectiveness and efficiency. Visual analyses also identify failure cases involving complex scenes, ambiguous saliency definitions, and misleading annotations.

  • Inter-Modal Cross-Attention: Inter-modal cross-attention improves performance by aligning related representations and gathering global context cues between modalities.
  • Self-Attention: Cross-scale and intra-modal self-attention have positive effects, with their combination achieving the best performance and outperforming a comparable shifted-window alternative.The comparison reports higher FPS and better average performance with similar parameters and FLOPs.
  • Locality: The convolutional feed-forward network improves overall performance by introducing region-wise operations that preserve local context overlooked by global attention.
  • View-Mixed Attention: The proposed view-mixed attention combination achieves the best performance among the tested branch-weighting strategies, while fixed weights of 0.5 remain competitive.
  • Patch Re-Embedding: The patch re-embedding setting “8, 8, 8, 8” provides the best average performance with moderate computational cost, whereas standard attention is infeasible because of memory requirements.The standard setting “1, 1, 1, 1” cannot be trained on the authors’ device; the table caption defines the four values as decoding levels from top to bottom.
  • Failure Cases: Typical failures arise from uncertain objects in complex scenes, ambiguous saliency definitions, and incomplete or unaligned annotation masks.The authors state that global relationship modeling may weaken performance on uncertain objects and that annotation problems can cause evaluation errors.

V. CONCLUSION

CAVER redesigns bi-modal salient object detection around global information propagation and cross-modal alignment. Its view-mixed transformer path, patch-wise token re-embedding, and convolutional feed-forward network support multi-scale processing while retaining local details, and experiments show the method’s effectiveness.

  • Conclusion: CAVER uses a view-mixed transformer-based top-down path to model global intra-modal cues and simplify cross-modal interaction and alignment.
  • Conclusion: Patch-wise token re-embedding reduces attention matrix computation and storage intensity, enabling multi-scale high-resolution feature processing.
  • Conclusion: The convolutional feed-forward network further enhances critical local details in feature maps.
  • Conclusion: Extensive experimental comparisons demonstrate the effectiveness of the proposed method.
Loading 2112.02363v3…