Source-linked AI summary
TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network
Zhengyi Liu, Yuan Wang, Zhengzheng Tu, Yun Xiao, Bin Tang
TL;DR
RGB-D salient object detection must exploit complementary multi-level features while handling limited gains from feature aggregation and unreliable depth. TriTransNet addresses this with shared-weight triplet transformers, scale adjustment, three-stream decoding, and depth-guided fusion, achieving nearly the best metrics across six datasets.
Problem
RGB-D salient object detection is limited by diminishing benefits from aggregating U-Net features and by negative effects from poor-quality depth images.
Method
TriTransNet uses three shared-weight transformer encoders to enhance high-level multi-level features, then applies scale adjustment, three-stream decoding, and depth purification.
Results
TriTransNet achieves nearly the best evaluation metrics on six datasets, with only two S-measure values ranking second-best.
Takeaways & Limitations
The reported experiments support TriTransNet as a strong RGB-D salient object detector and show the effectiveness of its proposed modules.
Abstract
from arXiv · showhide
Salient object detection is the pixel-level dense prediction task which can highlight the prominent object in the scene. Recently U-Net framework is widely used, and continuous convolution and pooling operations generate multi-level features which are complementary with each other. In view of the more contribution of high-level features for the performance, we propose a triplet transformer embedding module to enhance them by learning long-range dependencies across layers. It is the first to use three transformer encoders with shared weights to enhance multi-level features. By further designing scale adjustment module to process the input, devising three-stream decoder to process the output and attaching depth features to color features for the multi-modal fusion, the proposed triplet transformer embedding network (TriTransNet) achieves the state-of-the-art performance in RGB-D salient object detection, and pushes the performance to a new level. Experimental results demonstrate the effectiveness of the proposed modules and the competition of TriTransNet.
1 INTRODUCTION
TriTransNet addresses the limited benefit of progressively aggregating U-Net features by enhancing high-level multi-level representations with shared-weight transformer encoders. It combines this module with scale adjustment, three-stream decoding, and depth-guided feature enhancement for RGB-D salient object detection.
- Motivation: High-level features contribute more to performance, motivating TTEM to enhance the top three feature layers.The motivation follows observed saturation when aggregating features from high-level to low-level.
- Triplet transformer embedding: TTEM uses three standard transformer encoders with shared weights to learn common information and long-range dependencies across multi-level features.The shared encoders target complementary representations from different levels of the same input image.
- Network design: TriTransNet adjusts multi-level features to a common size, enhances them with TTEM, and fuses them with lower-layer features through a three-stream decoder.The transition layer and progressively upsampling fusion module prepare inputs before TTEM processing.
- Novelty: Unlike VT-FPN's single transformer, TriTransNet shares weights across three parallel encoders to better excavate high-layer semantics and middle-layer texture or shape information.The design is presented as the first use of three shared-weight transformer encoders for this feature enhancement.
- RGB-D fusion: Depth information supplies useful cues but can introduce noise when its quality is poor, so the depth purification module uses it to purify color features.The module addresses challenging scenes such as low color contrast and cluttered backgrounds.
2 RELATED WORK
Prior RGB-D saliency methods combine color and depth through early, middle, depth-guided, or late fusion, while transformer-based methods address global dependencies alongside CNN locality. TriTransNet embeds a triplet transformer module in U-Net and uses depth-guided attention to improve RGB-D saliency detection.
- RGB-D fusion: RGB-D saliency methods use early or input fusion, two-stream middle fusion, depth-guided fusion, and late fusion.These approaches reflect different stages for combining appearance, texture, 3D layout, and spatial structure.
- Depth quality: Low-quality depth maps can introduce negative influences, motivating filtering and quality-aware methods such as D3Net, EF-Net, and DQSD.The cited methods use gating, color hints, or depth-quality weighting to manage unreliable depth.
- Proposed fusion: The paper adopts depth-guided fusion by treating depth as a supplement to color features and enhancing them through attention mechanisms.The attention mechanism uses depth information to enhance color features rather than treating both modalities identically.
- Transformer-CNN hybrids: CNNs provide low-level and local features, whereas transformers establish long-range dependencies, motivating hybrid architectures that combine both strengths.Related models use transformer branches for global dependencies and CNN branches for spatial or texture details.
- TriTransNet: TriTransNet embeds a triplet transformer module into U-Net to improve RGB-D saliency detection while combining transformer and convolutional advantages.The paper presents this integration as its approach to the related-work gap.
3 PROPOSED METHOD
TriTransNet combines multimodal fusion, scale adjustment, triplet transformer enhancement, and three-stream decoding to improve RGB-D salient object detection. Its TTEM uses shared-weight encoders to model dependencies across aligned multi-level features while preserving original information.
- Overall architecture: The proposed network consists of a multi-modal fusion encoder, feature enhancement module, and three-stream decoder.These components form the overall TriTransNet framework.
- Multi-modal fusion: Depth features are purified with channel and spatial attention before being attached to color features through residual fusion.The module concatenates depth and color features, applies channel and spatial attention sequentially, and supplements the color representation.
- Scale adjustment: A transition layer and progressive upsampling fusion module align the channels and resolutions of high-level features before transformer processing.Progressive fusion avoids direct 2× or 4× upsampling and increases spatial detail in high-level features.
- Triplet transformer embedding: Three shared-weight transformer encoders convert aligned features into embeddings, learn long-range relationships across levels, and reshape them back to feature maps.Patch embeddings receive learnable positional embeddings, while each encoder uses standard transformer layers with multi-headed self-attention and multilayer perceptrons.
- Decoder: The enhanced high-level outputs are concatenated with original features and combined with low-level features through three-stream decoding.Each stream is upsampled, convolved, and passed through a sigmoid to produce supervised saliency maps, which are then fused into the final map.
- Training: Pixel position aware loss is used for end-to-end training.The ground-truth saliency map provides the training target.
4.1 Datasets and evaluation metrics
The evaluation uses six challenging RGB-D salient-object datasets with fixed training and testing protocols. Performance is assessed using precision-recall curves, E-measure, S-measure, F-measure, and MAE.
- Datasets: Six RGB-D salient-object datasets are used, including NLPR, NJU2K, STERE, DES, SIP, and DUT.The datasets cover indoor scenes, stereo images, high-resolution salient-person images, and complex scenes.
- Datasets: The main training set contains 1,485 NJU2K images and 700 NLPR images, while remaining images and specified datasets are reserved for testing.For DUT, 800 additional pairs are used for training and 400 remaining pairs for testing.
- Evaluation metrics: Five metrics evaluate the model and competing RGB-D salient-object detectors: precision-recall curves, E-measure, S-measure, F-measure, and MAE.These metrics assess thresholded precision-recall behavior, global and local alignment, structural similarity, precision-recall balance, and pixelwise absolute error.
4.2 Implementation details
Implementation resizes RGB and depth inputs to 256×256, applies data augmentation, initializes the backbone from ResNet-50, and configures the transformer with fixed hyperparameters.
- Data preparation: RGB and depth images are resized to 256×256 and augmented through random flipping, rotation, and border clipping.These strategies are applied to all training images.
- Model configuration: The backbone is initialized with pretrained ResNet-50 parameters, while transformer settings use L=12, D=768, and N=1024.Remaining parameters use PyTorch default initialization.
4.3 Comparisons with the state-of-the-art
TriTransNet is compared with 16 state-of-the-art RGB-D salient object detection models using quantitative and qualitative evaluations. It performs near the top across six datasets and handles several challenging visual conditions with more accurate saliency maps.
- TriTransNet achieves nearly the best results across all evaluated metrics on six RGB-D salient object detection datasets.Only two S-measure values, on NLPR and STERE, are second best.
- Its precision-recall curves indicate very high recall compared with competing models.
- Qualitative comparisons show better detection in similar foreground-background scenes, complex scenes, low-quality depth maps, small-object cases, and multiple-object scenes.
- The method produces more accurate saliency maps and finer details in salient regions.
4.4 Ablation studies
Ablation studies evaluate the triplet transformer embedding module, decoder design, and depth purification module. The reported results consistently favor the proposed components over their alternatives.
- TTEM effectiveness: Compared with the baseline without TTEM, TriTransNet improves average S-measure by 0.016, F-measure by 0.021, E-measure by 0.008, and MAE by 0.007.
- TTEM effectiveness: Compared with the GRU replacement, TriTransNet improves average S-measure by 0.012, F-measure by 0.014, E-measure by 0.006, and MAE by 0.005.
- TTEM effectiveness: The triplet transformer outperforms the Siamese transformer by 0.009, 0.016, 0.006, and 0.005 on average for S-measure, F-measure, E-measure, and MAE, respectively.
- TTEM effectiveness: The triplet transformer outperforms the quadruplet transformer by 0.010, 0.009, 0.005, and 0.004 on average for S-measure, F-measure, E-measure, and MAE, respectively.
- Decoder effectiveness: The three-stream decoder improves detection performance over the single-stream decoder through fuller integration of multi-layer features.
- DPM effectiveness: The depth purification module significantly improves performance over element-wise depth-color addition and also outperforms the compared depth-enhanced module.
5 CONCLUSIONS
The paper introduces TriTransNet, which embeds a triplet transformer module into U-Net for RGB-D salient object detection. The method combines multi-level feature enhancement with depth purification and reports state-of-the-art performance.
- TriTransNet introduces a triplet transformer embedding module into U-Net to improve RGB-D salient object detection.
- The module enhances feature representation by learning long-range dependencies among different feature levels with less cost.
- Depth purification uses depth information to enhance RGB features while addressing negative influences from poor-quality depth images.
- Experimental results report state-of-the-art performance for the proposed method.