PaperlayerView exact revision on arXiv

Source-linked AI summary

DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation

Bowen Yin, Xuying Zhang, Zhongyu Li, Li Liu, Ming-Ming Cheng, Qibin Hou

arXiv:2309.09668v2cs.CV

TL;DR

Existing RGB-D methods use RGB-pretrained backbones even though RGB-D inputs and dense cross-modal finetuning interactions raise representation and computational concerns. DFormer pretrains an RGB-D backbone with tailored RGB-D blocks, and reports state-of-the-art performance across RGB-D segmentation and salient object detection with less than half the computational cost of current best methods.

  • Problem

    Existing RGB-D methods use RGB-pretrained backbones for image-depth inputs, densely interact across modalities during finetuning, and incur added cost from dual backbones.

    Source: §1 · p. 2

  • Method

    DFormer uses a pretraining framework with tailored RGB-D blocks to learn transferable representations, allocating a small portion of channels to depth features.

    Source: §1 · p. 2, §2.1 · p. 4

  • Results

    DFormer achieves state-of-the-art performance with less than half the computational cost of current best methods across two RGB-D segmentation datasets and five salient object detection datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments suggest DFormer can achieve state-of-the-art performance in RGB-D downstream tasks with far less computational cost than existing methods.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present DFormer, a novel RGB-D pretraining framework to learn transferable representations for RGB-D segmentation tasks. DFormer has two new key innovations: 1) Unlike previous works that encode RGB-D information with RGB pretrained backbone, we pretrain the backbone using image-depth pairs from ImageNet-1K, and hence the DFormer is endowed with the capacity to encode RGB-D representations; 2) DFormer comprises a sequence of RGB-D blocks, which are tailored for encoding both RGB and depth information through a novel building block design. DFormer avoids the mismatched encoding of the 3D geometry relationships in depth maps by RGB pretrained backbones, which widely lies in existing methods but has not been resolved. We finetune the pretrained DFormer on two popular RGB-D tasks, i.e., RGB-D semantic segmentation and RGB-D salient object detection, with a lightweight decoder head. Experimental results show that our DFormer achieves new state-of-the-art performance on these two tasks with less than half of the computational cost of the current best methods on two RGB-D semantic segmentation datasets and five RGB-D salient object detection datasets. Our code is available at: https://github.com/VCIP-RGBD/DFormer.

1 INTRODUCTION

RGB-D methods commonly use RGB-pretrained backbones despite differences between image-only pretraining and image-depth inputs, dense cross-modal finetuning, and the added cost of dual backbones. DFormer proposes RGB-D pretraining and tailored RGB-D blocks, and reports state-of-the-art results with lower computational cost.

1 INTRODUCTION

  • RGB-D methods use image-depth inputs after image-only pretraining, densely interact across modalities during finetuning, and incur extra computation from dual backbones.The paper identifies the unconsidered depth information during pretraining as an important reason for these issues.Source: Figure 1 · p. 1, §1 · p. 2
  • DFormer is a pretraining framework with a new interaction method for fusing RGB and depth features into transferable representations for RGB-D downstream tasks.Figure 1 contrasts this RGB-D pretraining pipeline with methods that separately encode RGB and depth using RGB-pretrained backbones.Source: Figure 1 · p. 1, §1 · p. 2
  • DFormer uses a small portion of channels for depth information relative to RGB features, which the authors describe as an effective way to reduce model size.Source: §1 · p. 2
  • DFormer achieves state-of-the-art performance with less than half the computational cost of current best methods across two RGB-D segmentation datasets and five salient object detection datasets.The introduction also reports a 57.2% mIoU on NYU Depthv2 for DFormer-L, using less than half the computations of second-best CMNext.Source: §1 · p. 2, Figure 2 · p. 2

2 PROPOSED DFORMER

DFormer uses a hierarchical RGB-D encoder pretrained on ImageNet-1K image-depth pairs, then adds a lightweight decoder for task-specific predictions.

2.3 TASK-SPECIFIC DECODER.

  • The encoder-decoder pipeline produces task-specific predictions from hierarchical features, with a lightweight decoder on top of the pretrained encoder.For semantic segmentation, the decoder aggregates RGB features from the encoder’s last three stages; it does not use depth features directly.Source: §2 · p. 3, §2.3 · p. 5

2.1 HIERARCHICAL ENCODER

  • Global awareness attention combines depth and RGB queries with RGB keys and values to model whole-scene relationships; fixed-size query pooling reduces computation.Depth fusion into Q is described as sufficient; adding depth to K or V adds computation without performance improvement.Source: §2.1 · pp. 3–4
  • Local enhancement attention uses large-kernel depth-wise convolution on depth features to reweight RGB features, complementing global attention with local detail.A separate base module transforms RGB features, and the resulting features are fused by concatenation and linear projection.Source: §2.1 · pp. 3–4
  • Four hierarchical stages generate multi-scale RGB-D features, while a small portion of channels encodes depth to reduce model complexity.DFormer-T, -S, -B, and -L share the architecture but differ in model size; T targets fast inference and L higher performance.Source: §2.1 · pp. 3–4

2.2 RGB-D PRETRAINING

  • The encoder is pretrained on ImageNet-1K image-depth pairs with a classification objective to produce transferable RGB-D representations.A classifier head is added to the encoder, and training uses cross-entropy loss.Source: §2.2 · p. 4

3 EXPERIMENTS

Experiments evaluate DFormer on RGB-D semantic segmentation and salient object detection, reporting state-of-the-art comparisons and ablations of its pretraining and block design. The analyses also test channel allocation and transfer to other modalities.

3.1 RGB-D SEMANTIC SEGMENTATION

  • 57.2% mIoU with 39.0M parameters and 65.7G Flops gives DFormer-L state-of-the-art NYUDepthv2 performance, versus CMX (MiT-B2) at 54.4% mIoU, 66.6M parameters, and 67.6G Flops.DFormer-B exceeds CMX (MiT-B2) by 1.2% mIoU with 29.5M parameters and 41.9G Flops versus 66.6M parameters and 67.6G Flops; the authors report similar advantages on SUN-RGBD.Source: §3.1 · p. 6

3.2 RGB-D SALIENT OBJECT DETECTION

  • DFormer surpasses 11 recent RGB-D salient object detection competitors across five test datasets with the least computational cost.DFormer-T achieves comparable performance to SPNet with 5.9M parameters and 4.5G versus 150.3M parameters and 68.1G.Source: §3.2 · pp. 6–7, Table 2 · p. 6

3.3 ABLATION STUDY AND ANALYSIS

  • RGB-D pretraining improves DFormer-B by 2.3% mIoU over RGB pretraining on NYU DepthV2, with finetuning modalities and model structure held constant.The RGB-pretraining comparison replaces depth maps with RGB images during pretraining and duplicates depth maps during finetuning.Source: Table 3 · p. 6, §3.3 · p. 7
  • Using only RGB-branch features as the decoder input saves computation without a performance drop for DFormer-B.The comparison tests RGB features alone against using both RGB and depth features.Source: Table 4 · p. 6, §3.3 · p. 7
  • The base, GAA, and LEA components are each essential; GAA fusion into Q suffices, and depth-derived attention weights with a Hadamard product perform best among tested LEA fusion manners.Further fusing depth into K and V adds computation with negligible improvement; GAA performs best with 7×7 fixed pooling.Source: §3.3 · pp. 7–8, Table 7 · p. 7, Table 6 · p. 7
  • When the depth-to-RGB channel ratio exceeds 1/2, improvement is trivial while computational burden rises significantly, so the default ratio is 1/2.The comparison varies depth channels while holding RGB channels fixed.Source: §3.3 · p. 8, Figure 6 · p. 8
  • RGB-D pretraining improves CMX (MiT-B2) by about 1.4% mIoU, while DFormer-L still outperforms it by a large margin under RGB-D pretraining.The authors also report a similar situation under RGB-only pretraining.Source: §3.3 · p. 8, Table 10 · p. 8, Table 9 · p. 8
  • RGB-D pretraining improves results on RGB-T MFNet and RGB-L KITTI-360, but less than on RGB-D scenes.The authors suggest scaling pretraining to other modalities as a future direction.Source: §3.3 · p. 8, Table 11 · p. 8

4 RELATED WORK

Related work covers RGB-D scene parsing and multimodal pretraining, while identifying persistent challenges in RGB-D methods. DFormer is presented as a pretraining framework that aligns RGB and depth semantic cues using classification supervision.

RGB-D Scene Parsing

  • RGB-only scene parsing lacks 3D geometric information, motivating methods that combine images with depth maps for scene understanding.The discussion frames scene parsing as a core computer-vision pursuit and notes challenges in real-world scenes [Li et al., 2023b].Source: §4 · p. 9
  • RGB-D scene-parsing methods commonly fuse features from parallel pretrained backbones, while depth-aware operators integrate RGB features with spatial information.The passage says these operators are efficient but their improvements are usually limited by insufficient extraction and use of depth geometry.Source: §4 · p. 9

Multi-modal Learning

  • Multimodal pretraining includes multi-encoder approaches that align separate modality encoders and joint-encoder approaches that model modalities together.CLIP and VATT are examples of multi-encoder methods, while MultiMAE is described as a joint encoder.Source: §4 · p. 9
  • DFormer is presented as RGB-D representation pretraining that uses explicit classification supervision to align semantic cues from RGB and depth for downstream transfer.The authors describe this as the first such attempt to their knowledge.Source: §4 · p. 9

5 CONCLUSIONS

DFormer uses a tailored RGB-D pretraining framework and block to learn transferable representations. The reported experiments show state-of-the-art downstream performance with substantially lower computation than existing methods.

5 CONCLUSIONS

  • DFormer is a pretraining framework for transferable representations in RGB-D downstream tasks.The framework uses a tailored RGB-D block for interactions between RGB and depth modalities during pretraining.Source: §5 · p. 9
  • Experiments report state-of-the-art performance for DFormer on RGB-D semantic segmentation and salient object detection.The conclusion describes the computational cost as far lower than that of existing methods.Source: §5 · p. 9
  • The reported results cover RGB-D semantic segmentation and salient object detection, with lower computation than existing methods.This conclusion is limited to the downstream tasks and experiments described in the passage.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement describes access to source code, experimental configurations, and training details. The appendix outline includes further analyses, method descriptions, experimental settings, visualizations, and future directions.

REPRODUCIBILITY STATEMENT

  • The authors describe reproducibility measures and direct readers to relevant sections in the paper, appendix, and code.They invite inquiries or requests for clarification to support transparency and reliability.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Experimental settings are referenced in the main paper, appendix configuration and training tables, and supplementary source code.The cited main-paper sections cover pretraining and the two downstream tasks.Source: REPRODUCIBILITY STATEMENT · p. 10

APPENDIX

  • The appendix outline covers depth-encoding efficiency, depth-map quality, method descriptions, experimental settings, visualizations, and future directions.Source: APPENDIX · p. 16

A MORE ANALYSIS OF DFORMER

The analysis compares RGB- and depth-pretrained backbones on depth-map segmentation and examines depth-map quality and feature-distribution changes during finetuning.

Why involving depth information in pretraining?

  • Depth-pretrained backbones produced more expressive features from depth maps than RGB-pretrained backbones, including after finetuning.Figure 7 visualizes the feature maps from the two backbones.Source: Figure 7 · p. 16, A MORE ANALYSIS OF DFORMER · p. 16
  • 15% mIoU improvement came from depth-pretrained versus RGB-pretrained backbones using the same architecture and pretraining setting for depth-map segmentation.The comparison uses depth maps as input during segmentation.Source: A MORE ANALYSIS OF DFORMER · p. 16, Table 12 · p. 16

Impact of the quality of depth maps

  • 55.6 mIoU on NYUDepth v2 was achieved using Omnidata-predicted depth maps, with nearly the same performance as using Adabins-predicted maps.The analysis concludes that depth-map quality has little effect on DFormer performance.Source: A MORE ANALYSIS OF DFORMER · p. 17, Figure 8 · p. 17

Observations towards the distribution shift

  • RGBD-pretrained DFormer showed only slight batch-normalization mean and variance changes during finetuning, while RGB-pretrained DFormer showed drastic changes.The authors interpret the difference as evidence of transferable RGBD representations and mismatched encoding, respectively.Source: A MORE ANALYSIS OF DFORMER · pp. 17–18, Figure 9 · p. 17

B DETAILS OF DFORMER

The supplement details DFormer’s RGB-D block, reports inference latency comparisons, and describes additional ablation and benchmark comparisons.

Structure

  • The RGB-D block combines GAA and LEA for global and local cross-modal interactions with a base module for RGB features.The first stage omits GAA for efficiency, and the RGB and depth features use separate MLP layers.Source: B DETAILS OF DFORMER · p. 18, Figure 11 · p. 19

Inference time analysis

More detailed ablation towards the block

  • The additional base-module ablation covers only RGB-feature module structures and uses DFormer-S pretrained for 100 epochs.The passage describes these as supplementary results to ablations of block components, GAA pooling size, and fusion manners.Source: B DETAILS OF DFORMER · pp. 18–19, Table 13 · p. 19

B.1 COMPARISON ON THE MFNET AND KITTI-360

  • The supplement compares DFormer with other state-of-the-art methods on MFNet and KITTI-360.The comparisons are presented as supplements to the main paper.Source: §B.1 · p. 19

C EXPERIMENTAL DETAILS

DFormer variants share ImageNet-1K pretraining settings except for their stochastic-depth rates.

C.1 IMAGENET PRETRAINING

  • All DFormer variants use the same ImageNet-1K pretraining settings except for stochastic-depth rate.The settings are listed in Table 17.Source: §C.1 · p. 19
  • Color-related augmentations are applied only to RGB images, while common augmentations such as random rotation are applied to both RGB images and depth maps.Source: §C.1 · p. 19

C.2 RGB-D SEMANTIC SEGMENTATION FINETUNING

The section describes the RGB-D segmentation datasets, model configuration references, and finetuning and evaluation procedure.

RGB-D semantic segmentation finetuning

  • Finetuning settings differ between NYUDepthv2 and SUN-RGBD for batch size, input size, base learning rate, epochs, and stochastic depth.The settings are listed in Table 18.Source: §C.2 · p. 19, Table 18 · p. 21

DFormer configurations

  • The DFormer configuration table defines RGB and depth channel counts by stage, blocks per stage, MLP expansion, and decoder dimension.The table denotes the RGB and depth channel counts as C = (Crgb, Cd).Source: Table 14 · p. 20

Datasets

  • NYUDepthv2 contains 1,449 RGB-D samples across 40 categories, split into 795 training and 654 test pairs; SUN-RGBD contains 10,335 images across 37 categories, split into 5,285 training and 5,050 test images.NYUDepthv2 images and depth maps use 480×640 resolution; SUN-RGBD uses 530×730 resolution.Source: §C.2 · pp. 19–20

Implementation Details

  • The segmentation network uses the lightweight Hamburger decoder, cross-entropy loss, and AdamW with an initial learning rate of 6e-5.Finetuning uses random horizontal flips and random scaling from 0.5 to 1.75; testing uses multi-scale flip inference.Source: §C.2 · p. 20

C.3 RGB-D SALIENT OBJECT DETECTION

The salient-object setup describes its finetuning and evaluation datasets, head configuration, and metrics. Decoder analysis explains why only RGB-branch features are sent to the decoder.

C.3 RGB-D SALIENT OBJECT DETECTION

  • Finetuning uses 2,195 samples (1,485 NJU2K-train; 700 NLPR-train), with evaluation reported on DES, NLPR-test, NJU2K-test, STERE, and SIP.The passage describes these as five datasets and says the model is evaluated on eight; it lists sample counts for each of the five named datasets.Source: §C.3 · p. 21
  • The salient-object head adds a one-channel Hamburger head atop the RGB-D backbone, using SPNet's data augmentation and optimizer.Evaluation uses Structure-measure (S), mean absolute error (M), max F-measure (F), and max E-measure (E).Source: §C.3 · pp. 21–22

C.4 MORE DETAILS ABOUT THE DECODER INPUT FEATURES

  • The decoder takes only RGB-branch features because RGB-D pretraining lets those features better fuse information from both modalities than RGB-pretrained features.Figure 13 contrasts RGB-only inputs with inputs from both RGB and depth branches.Source: §C.4 · p. 22, Figure 13 · p. 22

D MORE VISUALIZATION RESULTS

The visualization results report higher segmentation accuracy than CMNext on NYU Depthv2 and show salient-object detection comparisons. The paper also identifies broader applications, modalities, and lightweight deployment as future directions.

E FUTURE WORK

  • Future directions include applying DFormer to more RGB-D tasks, replacing depth with modalities such as thermal or lidar, and developing lighter models.Source: E FUTURE WORK · p. 22
  • Although DFormer is described as efficient, the paper says a gap remains for practical applications because current RGB-D methods are usually too heavy for mobile devices.Source: E FUTURE WORK · p. 22

D MORE VISUALIZATION RESULTS

Loading 2309.09668v2…