PaperlayerView exact revision on arXiv

Source-linked AI summary

HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection

Bin Tang, Zhengyi Liu, Yacheng Tan, Qian He

arXiv:2301.03036v1cs.CV

TL;DR

Two-modality SOD must combine supplementary cues with RGB while addressing modality fusion and HRFormer’s output fusion. HRTransNet injects an encoded supplementary modality into HRFormer and refines its multi-resolution outputs; the paper reports excellent performance on RGB-D, RGB-T, and light-field SOD.

  • Problem

    Two-modality SOD must fuse supplementary cues with RGB, while the approach must address both modality fusion and HRFormer output fusion.

    Source: §I · pp. 1–2

  • Method

    HRTransNet encodes a supplementary modality, injects it into the primary modality, and uses interactive transformers to fuse HRFormer output resolutions.

    Source: §I · p. 2, A. Motivation · p. 4

  • Results

    HRTransNet exhibits excellent performance on RGB-D, RGB-T, and light-field SOD.

    Source: §I · p. 2, §V · p. 12

  • Takeaways & Limitations

    The proposed model is applied across RGB-D, RGB-T, and light-field salient object detection tasks.

    Source: §V · p. 12

Abstract

from arXiv · show

The High-Resolution Transformer (HRFormer) can maintain high-resolution representation and share global receptive fields. It is friendly towards salient object detection (SOD) in which the input and output have the same resolution. However, two critical problems need to be solved for two-modality SOD. One problem is two-modality fusion. The other problem is the HRFormer output's fusion. To address the first problem, a supplementary modality is injected into the primary modality by using global optimization and an attention mechanism to select and purify the modality at the input level. To solve the second problem, a dual-direction short connection fusion module is used to optimize the output features of HRFormer, thereby enhancing the detailed representation of objects at the output level. The proposed model, named HRTransNet, first introduces an auxiliary stream for feature extraction of supplementary modality. Then, features are injected into the primary modality at the beginning of each multi-resolution branch. Next, HRFormer is applied to achieve forwarding propagation. Finally, all the output features with different resolutions are aggregated by intra-feature and inter-feature interactive transformers. Application of the proposed model results in impressive improvement for driving two-modality SOD tasks, e.g., RGB-D, RGB-T, and light field SOD.https://github.com/liuzywen/HRTransNet

I. INTRODUCTION

HRTransNet adapts HRFormer to two-modality salient object detection, addressing both modality fusion and final multi-resolution fusion. It is reported to perform well on RGB-D, RGB-T, and light-field SOD.

I. INTRODUCTION

  • Two-modality SOD is challenging when RGB scenes contain poor illumination, complex backgrounds, or indistinguishable objects, and the fusion of modalities remains an open design issue.Depth, thermal, and focal-stack inputs can contribute spatial structure, heat, and focus information, respectively.Source: §I · pp. 1–2
  • HRFormer provides a backbone for dense prediction by maintaining high-resolution representations while transformer blocks add global ability alongside local features.The paper identifies maintaining input-to-output resolution as important for SOD.Source: §I · pp. 1–2
  • An auxiliary stream encodes the supplementary modality, which is injected into the primary modality at the beginning of each multi-resolution branch.The input-level module formulates supplementary-modality weights globally and emphasizes coordinate-wise spatial positions.Source: §I · p. 2
  • Four interactive transformers form a dual-direction short-connection module that refines each HRFormer output resolution using itself and features from other resolutions.The module is intended to improve object-detail representation through final multi-resolution fusion.Source: §I · p. 2
  • HRTransNet exhibits excellent performance on RGB-D, RGB-T, and light-field SOD, supporting its use for two-modality complementary tasks.The model is presented as a unified approach for these three tasks.Source: §I · p. 2

II. RELATED WORKS

Related work describes HRNet adaptations across its stacked layers, exchange units, and final multi-resolution fusion, alongside prior multimodal fusion designs. This study combines HRFormer with transformer-based final fusion and injects supplementary features at each branch.

A. High-resolution network

  • HRNet maintains parallel resolutions and repeated multi-scale fusion for dense prediction; later variants modify its stacked layers, exchange units, or final multi-resolution fusion.Examples include transformer-block replacements and alternatives for aggregating output scales.Source: A. High-resolution network · pp. 2–3
  • The study uses HRFormer, whose stacked convolution layers are modified, and adds transformer-based final fusion to exploit intra- and inter-feature correlations.The fusion strategy is described as operating at the final multi-resolution fusion stage.Source: A. High-resolution network · p. 3
  • Prior dual-stream HRNet fusion is reported to have an insufficient receptive field, while AVT integrates audio only in a late exchange unit; this study injects an auxiliary-encoded modality at each branch.The paper notes late audio integration is not suited to supplementary image modalities such as depth, thermal images, or focal stacks.Source: A. High-resolution network · p. 3

B. Two-modality salient object detection

The related work motivates a shared two-modality SOD approach and positions HRFormer as a high-resolution transformer backbone for the task. The method combines input-level modality injection with output-level multi-resolution fusion.

B. Two-modality salient object detection

  • Depth, thermal, and focal-stack cues provide spatial, heat, and focus information, while prior methods often address only one or two SOD tasks.The paper identifies supplementary-modality purification and preserving input-to-output resolution as shared needs across these tasks.Source: B. Two-modality salient object detection · p. 3

C. Transformer-based salient object detection

  • Prior SOD work applies transformers in encoders, decoders, and backbones; this study uses HRFormer’s high-resolution representation for two-modality SOD.The approach is framed as combining high-resolution networks with transformers.Source: C. Transformer-based salient object… · pp. 3–4

A. Motivation

  • Supplementary modalities can contain noise, local blurring, and redundancy, so the proposed injection module weights them globally and emphasizes their coordinate-wise spatial positions.The cues are injected into the primary color modality.Source: A. Motivation · p. 4
  • Four TripleIT units model intra- and inter-feature correlation using HRFormer output resolutions, with dual short connections to enhance multi-scale fusion and salient-object detail.The illustrated RGB-D architecture comprises an auxiliary stream, supplementary-modality injection, HRFormer, and dual-direction short-connection fusion, and is also applied to RGB-T and light-field SOD.Source: A. Motivation · p. 4, Figure 3 · p. 4

B. Backbone network for primary color modality

HRFormer provides the primary-color backbone, while an auxiliary stream extracts supplementary-modality features for injection into the backbone’s stages. The injection module weights and purifies the modalities before combining them.

B. Backbone network for primary color modality

  • HRFormer maintains high-resolution representations and uses transformer blocks to enlarge receptive fields, with exchange units sharing information across resolutions.Its high-resolution representations suit SOD, where input and output resolutions match.Source: B. Backbone network for primary color… · pp. 4–5

C. Auxiliary stream for supplementary modality

  • ResNet18 extracts hierarchical features from the supplementary modality through an auxiliary stream.The supplementary inputs are depth images, thermal images, or light-field focal stacks for the corresponding SOD tasks.Source: C. Auxiliary stream for supplementary… · p. 5

D. Supplementary modality injection module

  • SMIM concatenates primary-color and supplementary features, generates modality weights, applies coordinate attention to purify supplementary features, and adds the modalities.The supplementary modality is injected at the beginning of each HRFormer stage.Source: D. Supplementary modality injection… · p. 5

E. Dual-direction short connection fusion module

The dual-direction short connection fusion module aggregates HRFormer’s multi-resolution outputs into decoding features. Its TripleIT units combine intra-feature processing with inter-feature connections across resolutions.

E. Dual-direction short connection fusion module

  • The module aggregates HRFormer’s multi-resolution output features into decoding features because feature concatenation alone has limited expressiveness.The authors identify this aggregation as essential for dense prediction.Source: E. Dual-direction short connection… · p. 5
  • Four TripleIT units optimize each primary feature’s long-range dependencies before combining it with associated features to generate decoding features.Each unit processes an HRFormer output at its corresponding resolution.Source: E. Dual-direction short connection… · pp. 5–6
  • High-resolution HRFormer outputs are added to lower-resolution TripleIT units, while decoding features are added back to higher-resolution units.The connections run in both directions across resolutions.Source: E. Dual-direction short connection… · p. 5
  • The first decoding feature is convolved and upsampled fourfold to form the saliency map.The feature aggregation therefore feeds the final output prediction.Source: E. Dual-direction short connection… · p. 6

F. Intra-feature and inter-feature interactive transformer

TripleIT refines each resolution feature through its relationship to itself and to associated features at other resolutions, supporting complementary multi-resolution fusion.

F. Intra-feature and inter-feature interactive transformer

  • TripleIT models intra-feature and inter-feature correlations to optimize each primary resolution feature using itself and associated features.The module comprises four TripleITs and optimizes each resolution output feature by itself and the other resolution features.Source: F. Intra-feature and inter-feature… · p. 6
  • Each TripleIT first applies self-attention to the primary feature, then cross-attention between that feature and associated features, followed by a feed-forward layer.Features are divided into 1×1×128 tokens; the attention stages model within-feature and cross-resolution relations.Source: F. Intra-feature and inter-feature… · p. 6
  • Stacking two such architectures refines the primary feature using its intra-feature relationship and information from associated features.Attention operations use residual connections and normalization; the attention layers use efficient attention for lower memory and computational costs.Source: F. Intra-feature and inter-feature… · p. 6

A. Datasets

The section reports RGB-D SOD evaluation alongside datasets, metrics, and implementation settings, with HRTransNet compared quantitatively and visually against prior methods.

A. Datasets

  • The RGB-D datasets are NLPR, NJU2K, STERE, DES, SIP, and DUT, with image counts ranging from 135 to 1,985.The section also lists RGB-T and light-field datasets.Source: A. Datasets · p. 6

B. Evaluation metrics

  • Evaluation uses precision-recall curves, S-measure, F-measure, E-measure, and mean absolute error to assess detection and similarity.The passage describes S-measure as evaluating structural similarity and MAE as measuring per-pixel absolute difference.Source: B. Evaluation metrics · p. 6

C. Implementation details

  • Training uses 2,185 paired RGB-depth images, 224×224 inputs, batch size 15, learning rate 5e-5, and up to 300 epochs.For DUT testing, the training set adds 800 DUT image pairs; training uses an NVIDIA RTX 3090 GPU.Source: C. Implementation details · p. 6

1) RGB-D SOD:

  • HRTransNet’s precision-recall curves are better than other models on all six RGB-D datasets, and most evaluation metrics are superior.The method is comparable with EBMG and SwinNet and uses fewer parameters and lower FLOPs than most compared methods.Source: 1) RGB-D SOD · pp. 6–7, Figure 6 · p. 7
  • Visual examples cover indistinguishable foregrounds, complex scenes, low-quality depth, small and multiple objects, and fine-grained objects.The paper attributes low-quality-depth robustness to filtering depth noise and fine-detail detection to multi-resolution feature fusion.Source: 1) RGB-D SOD · p. 7

2) RGB-T SOD:

The RGB-T and light-field evaluations report quantitative and visual comparisons, with HRTransNet performing strongly across the described datasets and challenging scenes.

2) RGB-T SOD:

  • The method’s precision-recall curve outperforms the other methods by a great margin, while four evaluation metrics indicate outstanding performance and robustness when applied to RGB and thermal image pairs.Source: 2) RGB-T SOD · p. 7, Figure 8 · p. 9, Table II · p. 9
  • RGB-T visual examples include indistinguishable foregrounds, clutter, weak light, low-quality thermal images, small and multiple objects, and noisy scenes.The comparisons are presented as evidence of the proposed modules’ effectiveness when combining color and thermal infrared information.Source: 2) RGB-T SOD · p. 7, Figure 9 · p. 9

3) LF SOD:

  • The light-field setup uses an all-in-focus image as the primary input and concatenated focal slices as the supplementary feature.The described focal stack contains no more than 12 slices; shorter stacks are padded with an all-zero image.Source: 3) LF SOD · p. 10
  • For light-field SOD, HRTransNet outperforms other methods on HFUT-Lytro and DUTLF-FS and is slightly better on LFSD.The reported improvement is achieved using concatenated focal slices, and the paper presents this as evidence of compatibility with two-modality SOD.Source: 3) LF SOD · p. 10, Figure 10 · p. 10, Table III · p. 10
  • Light-field visual examples cover big and small objects, multiple objects, similar foreground and background, and complex scenes.The paper describes the model as capturing long-range dependencies while maintaining fine-grained local detail.Source: 3) LF SOD · p. 10

E. Ablation studies

Ablations examine stream backbones and the supplementary-modality and output-fusion modules, with the full configuration achieving the best performance.

1) Selection of primary stream backbone:

2) Selection of auxiliary stream backbone:

3) Effectiveness of modules:

  • The model equipped with SMIM and four TripleITs achieves the best performance among the ablated configurations.TripleITs improve all evaluation metrics, while adding SMIM improves all reported values except S-measure and MAE on STERE.Source: 3) Effectiveness of modules · pp. 10–11, Table VI · p. 11
  • The module comparison finds addition inferior for incomplete or inadequate depth, while the simple decoder produces blurry boundaries.The proposed model assigns different weights to modalities to suppress depth noise and uses transformer long-range dependencies to polish details.Source: 3) Effectiveness of modules · p. 11, Figure 13 · p. 12

F. Failure cases

HRTransNet still fails on occlusion, highly similar backgrounds, and object completeness, problems also seen in other state-of-the-art RGB-D SOD methods. The conclusions describe supplementary-modality injection and interactive multi-resolution fusion, report application to RGB-D, RGB-T, and light-field SOD, and identify further tasks for future study.

F. Failure cases

  • Occlusion, extremely similar backgrounds, and object completeness cause failures; similar failed detections also occur in other state-of-the-art RGB-D SOD methods.Examples include a missed person, an erroneous result, and a glider propeller misidentified as background; the authors call for further study of these problems.Source: F. Failure cases · p. 12

V. CONCLUSIONS

  • HRTransNet injects a supplementary modality to combine two-modality cues and modifies HRFormer’s final multi-resolution fusion using intra-feature and inter-feature interactive transformers.Each transformer-like unit enhances features using both the feature itself and features from other resolutions.Source: §V · p. 12
  • Feature transmission to TripleIT forms a dual-direction short-connection flow, and each unit enhances features using themselves and features from other resolutions.The conclusions describe this as the model’s approach to modifying final multi-resolution fusion.Source: §V · p. 12
  • HRTransNet is applied to RGB-D, RGB-T, and light-field SOD, and the authors report excellent performance.Future work will attempt other two-modality SOD tasks and explore multi-modality SOD, co-saliency, and SOD in VR/AR applications.Source: §V · p. 12
Loading 2301.03036v1…