Source-linked AI summary
DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation
Bo-Wen Yin, Jiao-Long Cao, Ming-Ming Cheng, Qibin Hou
TL;DR
RGB-D segmentation methods typically encode RGB and depth with neural networks, motivating whether depth can instead serve directly as geometry information. DFormerv2 uses depth-derived geometry priors to allocate self-attention weights without extra depth-encoding layers, achieving stronger benchmark results and a better computation trade-off.
Problem
Existing RGB-D segmentation methods encode and fuse depth with RGB, while commonly overlooking the inherent differences between the two modalities.
Method
DFormerv2 derives geometry priors from depth and uses Geometry Self-Attention to allocate attention weights, without explicitly encoding depth maps with extra layers.
Results
DFormerv2 achieves better RGB-D semantic segmentation results than recent methods with far less computational cost across NYU DepthV2, SUNRGBD, and Deliver.
Takeaways & Limitations
Depth can be used as an explicit geometry prior for attention, yielding an RGB-D backbone with a stronger performance–computation trade-off.
Abstract
from arXiv · showhide
Recent advances in scene understanding benefit a lot from depth maps because of the 3D geometry information, especially in complex conditions (e.g., low light and overexposed). Existing approaches encode depth maps along with RGB images and perform feature fusion between them to enable more robust predictions. Taking into account that depth can be regarded as a geometry supplement for RGB images, a straightforward question arises: Do we really need to explicitly encode depth information with neural networks as done for RGB images? Based on this insight, in this paper, we investigate a new way to learn RGBD feature representations and present DFormerv2, a strong RGBD encoder that explicitly uses depth maps as geometry priors rather than encoding depth information with neural networks. Our goal is to extract the geometry clues from the depth and spatial distances among all the image patch tokens, which will then be used as geometry priors to allocate attention weights in self-attention. Extensive experiments demonstrate that DFormerv2 exhibits exceptional performance in various RGBD semantic segmentation benchmarks. Code is available at: https://github.com/VCIP-RGBD/DFormer.
1. Introduction
DFormerv2 treats depth as an explicit geometry prior rather than encoding it with a parallel neural network, using that prior to guide self-attention and visual features. The resulting RGB-D encoder targets stronger segmentation with substantially lower computation than existing approaches.
- Existing RGB-D methods commonly use dual encoders and fusion modules, or unified encoders, while often overlooking differences between RGB and depth.
- DFormerv2 directly uses depth to form a scene geometry prior and guide self-attention weights instead of explicitly encoding depth maps.The method models geometric and spatial relationships among image patch tokens and requires no extra depth-processing layers.
- Geometry Self-Attention provides the core mechanism for constructing DFormerv2, an RGB-D vision backbone.
- 57.7% mIoU on NYU DepthV2 matches Gemnifuision (MiT-B5) with less than half the computation costs for DFormerv2-B.
- DFormerv2-L achieves 58.4% mIoU on NYU DepthV2 with 95.5M parameters, establishing a stronger performance–computation trade-off than other methods.
- The paper reports new state-of-the-art performance with less than half the computational cost of the best current methods across three RGB-D semantic segmentation datasets.The datasets are NYU DepthV2, SUNRGBD, and Deliver.
2. Related Work
RGB-D semantic segmentation combines RGB with depth to address difficult scenes, while recent work focuses on how to fuse modalities and incorporate priors into transformer attention.
- RGB-D Semantic Segmentation: RGB-D semantic segmentation supplements RGB images with depth because depth provides 3D geometry for challenging scenes.RGB-only methods can struggle in low illumination and other difficult conditions.
- RGB-D Semantic Segmentation: Mainstream methods commonly fuse RGB and depth features produced by two parallel pretrained backbones.Interaction modules dynamically combine modality representations and aggregate them in the decoder.
- RGB-D Semantic Segmentation: DFormer uses a unified RGB-D backbone and reduces depth computation, but does not explicitly model depth’s intrinsic geometric characteristics.The paper positions geometry-prior generation from depth as an alternative to adding lower-cost depth encoding.
- Vision Transformer and Prior Knowledge: Vision Transformers model global context through attention over image patches, but vanilla self-attention has high pairwise computational cost.Sparse attention mechanisms have been proposed to reduce this burden.
- Vision Transformer and Prior Knowledge: Unlike positional priors, geometry priors derived from depth can model 3D relationships across the whole image.This motivates introducing geometry prior knowledge into self-attention.
3. Methodology
DFormerv2 builds geometry priors from depth and spatial relationships among image patches, then injects them into self-attention to guide RGB-D feature modeling. Its pyramid encoder uses decomposed geometry self-attention for efficiency and a lightweight decoder for segmentation.
- Geometry Prior Generation: Depth patches provide camera-distance information, while patch coordinates provide spatial relationships used to model geometry among all image tokens.Depth locations are obtained by average pooling each depth patch; spatial distances use Manhattan distance.
- Geometry Prior Generation: The depth and spatial relationship matrices are fused into a geometry prior that bridges scene depth with image-space structure.The method uses two learnable memories for weighted summation of the depth and spatial priors.
- Geometry Self-Attention: Geometry self-attention multiplies ordinary attention weights by β^G, suppressing distant or irrelevant key-value pairs and emphasizing geometrically relevant regions.β is a decay rate in (0, 1), and the geometry prior is applied through element-wise multiplication.
- Geometry Self-Attention: To reduce high-resolution cost, the geometry prior and attention are decomposed along horizontal and vertical directions.The decomposed priors Gx and Gy support attention along columns and rows, respectively.
- Evaluation: The method reports best results on NYU Depth V2 and SUN-RGBD, while Figure 5 compares performance against computation cost on NYU DepthV2.The supplied table caption states that the method receives the best results on both datasets; the figure caption specifies the performance-computation comparison.
- DFormerv2 Architecture: DFormerv2 uses a four-stage pyramid encoder with geometry self-attention blocks and a lightweight decoder that produces RGB-D segmentation results.The first three stages use decomposed geometry self-attention, while the final stage does not.
4. Experiments
Experiments evaluate DFormerv2 across RGB-D segmentation datasets, model scales, ablations, visualizations, latency, and modality-specific behavior. The results emphasize strong accuracy–efficiency trade-offs and the contribution of geometry priors.
- Comparisons with Other Methods: DFormerv2 achieves new state-of-the-art performance across model scales on NYU DepthV2 and SUNRGBD, with improvements also reported on Deliver.The comparison covers 17 recent RGB-D semantic segmentation methods.
- Comparisons with Other Methods: 58.4% mIoU with 95.5M parameters and 124.1G Flops lets DFormerv2-L surpass Gemnifusion by 0.7% with less than half its computations.The reported NYU DepthV2 comparison demonstrates the model’s accuracy–efficiency trade-off.
- Model Analysis: Depth and spatial priors improve vanilla self-attention accuracy, by 2.6% and 1.8% on NYU DepthV2 and 1.7% and 1.3% on SUNRGBD, respectively.These results come from the ablation roadmap toward geometry self-attention.
- Model Analysis: The geometry prior is derived from depth and represents geometric relationships between token pairs, helping identify object membership and nearby object relationships.Visualizations examine the prior, focused attention, and features with and without geometry priors.
- Model Analysis: Inference experiments assess real-time potential on identical hardware and resolution, reporting a good speed–accuracy trade-off for DFormerv2.Tests use a single RTX 3090 GPU and 480 × 640 images.
- Model Analysis: Depth mainly helps segment object shapes, while providing a smaller benefit for capturing semantic categories.This modality analysis uses classification and foreground segmentation experiments on LUSS.
5. Conclusions
The paper concludes that DFormerv2 uses depth as an explicit geometry prior within self-attention rather than as a separately encoded modality. This design yields stronger RGB-D segmentation results with substantially lower computational cost.
- Conclusions: DFormerv2 models geometric relationships between image patches from depth and uses them to allocate self-attention weights.This mechanism is called geometry self-attention.
- Conclusions: The geometry-prior design provides more effective utilization of the depth modality for RGB-D semantic segmentation.The conclusion attributes this to the tailored attention mechanism.
- Conclusions: DFormerv2 produces better results than recent methods with far less computational cost.