PaperlayerView exact revision on arXiv

Source-linked AI summary

RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

Shaohua Dong, Zexuan Meng, Haiyan Sun, Bing Fan, Cuicui Zhang, Dylan Joseph, Kewei Sha, Yunhe Feng, Heng Fan

arXiv:2609.29028v1cs.CV

TL;DR

Existing RGB-D benchmarks are limited in scale, semantic and scene coverage, and annotation quality. The paper introduces RGBD20K and the score-purified fusion (SPF) method. Its ablations report the best result with a 2-layer MLP and Sigmoid activation, reaching 46.3% mIoU.

  • Problem

    Existing RGB-D benchmarks have limited scale, category and scene coverage, and noisy labels, constraining the available data and annotation quality for RGB-D segmentation.

    Source: §I · p. 1

  • Method

    The paper introduces RGBD20K, a larger and broader annotated RGB-D dataset, and SPF, which purifies multimodal features using reliability scores before attention.

    Source: §I · pp. 1–2, B. Score-Purified Fusion · p. 5

  • Results

    46.3% mIoU is achieved by a 2-layer MLP with Sigmoid activation in the reported transformation-layer and activation comparison.

    Source: C. Ablation Studies · p. 7

  • Takeaways & Limitations

    RGBD20K provides a larger, more finely categorized and more carefully annotated benchmark, while SPF is proposed as a baseline for future RGB-D perception research.

    Source: §I · pp. 1–2

Abstract

from arXiv · show

In this paper, we propose RGBD20K, a novel dataset for facilitating the development of more robust and general RGB-D semantic segmentation by encompassing abundant categories and high-quality annotations. RGBD20K possesses several attractive properties: (1) Expanded Semantic Space. In particular, it covers 160 fine-grained categories, largely surpassing the category diversity of existing popular RGB-D benchmarks (e.g., NYUv2 with 40 classes and SUN RGB-D with 37 classes). With such enriched semantic coverage, we expect to promote the learning of more generalizable segmentation models. (2) Larger Scale. Compared with current benchmarks, RGBD20K offers 20,000 RGB-D image pairs, providing a substantially larger training resource that benefits the development of more powerful deep models. (3) High-Fidelity Annotation. We perform rigorous re-evaluation and correction of existing labels to resolve long-standing annotation noise, resulting in a clean and reliable ground-truth foundation. Furthermore, we propose a novel score-purified fusion (SPF) method, which achieves state-of-the-art performance across all evaluated benchmarks, demonstrating the effectiveness of our approach in leveraging high-quality multimodal information for RGB-D semantic segmentation. The dataset is here: https://github.com/ShaohuaDong2021/RGBD20K/.

I. INTRODUCTION

Existing RGB-D benchmarks are limited in scale, semantic coverage, scene diversity, and annotation quality. The paper introduces RGBD20K to address these dataset limitations and proposes the SPF model.

I. INTRODUCTION

  • Existing RGB-D benchmarks have limited scale, semantic coverage and scene diversity, and may contain noisy annotations that undermine evaluation and model development.NYUv1 and NYUv2 contain 2,347 and 1,449 annotated pairs, while SUN RGB-D contains 10,335 images; NYUv1, NYUv2 and SUN RGB-D have 13, 40 and 37 categories, respectively.Source: §I · p. 1
  • RGBD20K provides 20,000 high-quality RGB-D image pairs across 160 fine-grained categories and 75 scene types.The paper presents this broader scale and semantic coverage as a foundation for more comprehensive learning of complex scene structures.Source: §I · pp. 1–2
  • RGBD20K uses rigorous multi-stage manual refinement to produce more accurate pixel-level annotations with clear boundaries and consistent labels.The paper presents these refinements as a more reliable supervision signal for training.Source: §I · p. 2
  • The paper introduces score-purified fusion (SPF), following a “purify-then-attend” design principle, as a model alongside the benchmark.The authors describe releasing the dataset and SPF model as a benchmark and baseline for future RGB-D perception research.Source: §I · p. 2

II. RELATED WORK

The related work describes limitations in existing RGB-D benchmarks and segmentation architectures, alongside broader multimodal datasets. SPF is presented as a score-based purification approach before cross-modal interaction.

RGB-D Semantic Segmentation Benchmarks

  • Existing RGB-D datasets trade scale against task fit: some offer many images but target reconstruction or coarse parsing rather than precise 2D semantic masks.Matterport3D has 194,400 images across 90 buildings, while 2D-3D-S has 70,496 images; the paper says continuous scans can also bring redundancy and imprecise projected labels.Source: §II · p. 2

RGB-D Semantic Segmentation Algorithms

  • Dual-stream RGB-D methods such as CMX, TokenFusion and GeminiFusion [21] fuse features but incur backbone overhead and possible depth-stream distribution mismatch.DPLNet explores prompts to reduce trainable parameters, while DFormer [6], investigates unified RGB-D representation learning.Source: §II · p. 2
  • SPF purifies features using scores before cross-modal interaction, contrasting with interaction-heavy and unified-backbone paradigms.The text characterizes this as a more direct and efficient use of multimodal cues.Source: §II · pp. 2–3

Other Multi-modal Segmentation Benchmarks and Algorithms

  • Multimodal benchmarks beyond RGB-D include RGB-T datasets for low-illumination and nighttime settings and DeLiVER, which spans Depth, LiDAR, multiple Views, Events and RGB.These resources support research on segmentation across complementary sensing modalities.Source: §II · p. 3

III. THE PROPOSED RGBD20K

RGBD20K is constructed to provide large-scale RGB-D data, broad semantic and scene coverage, and high-precision annotations for RGB-D semantic segmentation.

A. Construction Principles

  • RGBD20K’s construction targets a large image collection, rich object categories and high-precision semantic annotations.The stated goal is to support development of more generalizable and robust RGB-D semantic segmentation methods.Source: A. Construction Principles · p. 3
  • The dataset contains 20,000 RGB-D image pairs with variation in illumination, object arrangements and scene layouts.The paper describes its scale as substantially larger than existing RGB-D benchmarks.Source: A. Construction Principles · p. 3
  • RGBD20K covers 160 fine-grained object classes and 75 scene types to increase semantic and environmental diversity.The paper connects this coverage to detailed scene understanding and real-world complexity.Source: A. Construction Principles · p. 3
  • Each RGB-D pair undergoes multiple rounds of manual inspection and refinement to improve boundary accuracy and annotation consistency while reducing noise.Source: A. Construction Principles · p. 3

B. Construction Principles

RGBD20K unifies RGB-D sources under a 160-class taxonomy and uses manual, depth-aware annotation procedures to create pixel-level masks with hierarchical and flexible category coverage.

C. Data Acquisition

  • RGBD20K integrates 20,000 depth-aligned pairs from scene-centric and tracking-oriented sources, including 8,365 pairs from DIML RGB-D.The listed sources also include SUN RGB-D, RGB-D Mirror, VidSOD, DepthTrack and ARKitTrack.Source: C. Data Acquisition · p. 4
  • Manual cleaning and re-annotation unify the sources under a 160-class taxonomy, with categories verified by domain experts and a natural long-tail distribution.The paper says this foundation can support supervised, open-vocabulary and zero-shot perception tasks.Source: C. Data Acquisition · p. 4

D. Annotation

  • Trained annotators produce pixel-level masks through a unified manual process, using hierarchical labels from coarse concepts to fine-grained classes.Source: D. Annotation · p. 4
  • Depth maps and mask geometry guide relative depth ordering to resolve overlaps and maintain foreground–background relationships.Background regions such as walls and floors are placed at the farthest level.Source: D. Annotation · p. 4
  • The annotation scheme permits new classes and labels object parts linked to parent objects, forming a lightweight hierarchy.The paper gives drawer–cabinet as an example of a part–parent link.Source: D. Annotation · p. 4

E. Dataset Split

RGBD20K uses a stratified 18,000/2,000 training-test split across 160 categories. SPF processes RGB and depth through hierarchical encoders, purifies multimodal features before attention, and decodes fused features into segmentation predictions.

E. Dataset Split

  • RGBD20K splits 20,000 pairs into 18,000 training and 2,000 test examples, preserving distributions of scene types, object categories, and depth characteristics across 160 categories.All categories appear in both subsets, with a long-tailed distribution retained.Source: E. Dataset Split · p. 4

IV. METHODOLOGY: SCORE-PURIFIED FUSION MODEL

  • SPF filters and refines Key and Value representations using cross-examined reliability scores before attention computation.The scores assess cross-modal feature consistency and adaptively suppress unreliable responses while enhancing semantically consistent regions.Source: §IV · p. 4

A. Overall architecture

  • RGB and depth pass through shared encoder layers in a four-stage hierarchy, with features fused at each stage and a SegFormer head producing segmentation predictions.The SPF module replaces GeminiFusion’s original fusion strategy.Source: A. Overall architecture · pp. 4–5, Figure 4 · p. 5

B. Score-Purified Fusion

Score-Purified Fusion estimates reliability across RGB and depth streams, then uses those scores to purify Keys and Values before attention. This “purify-then-attend” design aims to limit cross-modal noise and attention dilution.

Reciprocal Score Generation

  • SPF first projects RGB and depth features into aligned embeddings, then estimates dynamic reliability scores for each modality.The score head uses cross-modal comparison to assess feature reliability.Source: B. Score-Purified Fusion · p. 5
  • The fused importance scores are split into intra-modal and cross-modal components for RGB and depth.These components represent pixel-wise reliability scores for each stream.Source: B. Score-Purified Fusion · p. 5

Score-Guided Component Enhancement

  • Purified Alignment adaptively weights learnable modality-specific noise and cross-modal features by their reliability scores to construct Keys and Values.The formulation is described as filtering cross-modal noise, such as depth-edge artifacts, before attention.Source: B. Score-Purified Fusion · p. 5
  • Multi-Head Attention operates on the purified Key and Value manifolds, followed by residual skip connections and a Feed-Forward Network.The paper describes this as letting attention focus on context aggregation in a noise-robust latent space.Source: B. Score-Purified Fusion · pp. 5–6

A. Datasets and Implementation Details

The evaluation uses NYUv2, SUN RGB-D, and RGBD20K, which span different dataset sizes and category counts. Training follows the GeminiFusion setting with a Swin-Transformer Large backbone and a staged learning-rate schedule.

Datasets and Metrics

  • NYUv2 uses 795 training and 654 test images across 40 categories, SUN RGB-D uses 5,285 and 5,050 across 37, and RGBD20K uses 18,000 and 2,000 across 160.The paper reports mean Intersection-over-Union (mIoU), averaged across semantic categories.Source: A. Datasets and Implementation Details · p. 6
  • Inputs are evaluated at 480 × 640 for NYUv2 and RGBD20K, and 480 × 480 for SUN RGB-D.The NYUv2 resolution follows GeminiFusion for fair comparison; RGBD20K is described as having more fine-grained annotations and greater task difficulty.Source: A. Datasets and Implementation Details · p. 6

Implementation Details

  • Training uses a Swin-Transformer Large backbone for 300 epochs on four NVIDIA H100 GPUs, with three 100-epoch stages starting at learning rates of 6 × 10−5, 3 × 10−5, and 1.5 × 10−5.The learning rate follows a poly decay schedule within each stage.Source: A. Datasets and Implementation Details · p. 6

B. Comparisons with the State of the Art

The paper compares SPF with representative RGB-D segmentation methods across NYUv2, SUN RGB-D, and RGBD20K. Its annotation analysis reports that refining SUN RGB-D labels changes measured and retrained model performance.

SUN RGB-D Annotation Quality Analysis

  • Refining only SUN RGB-D test annotations raises pretrained DPLNet mIoU from 52.8 to 55.0.The original 37-category setting is retained for SUN RGB-D†.Source: B. Comparisons with the State of the Art · p. 6
  • After retraining on the refined SUN RGB-D† dataset, DPLNet achieves an mIoU of 59.0.The refined dataset retains the original 37-category setting.Source: B. Comparisons with the State of the Art · p. 6
  • All evaluated methods achieve higher performance on SUN RGB-D† than on the original SUN RGB-D, while SPF is compared with eight named methods across three benchmarks.For RGBD20K, comparisons follow each method’s original SUN RGB-D evaluation settings; the results are reported in Table III.Source: B. Comparisons with the State of the Art · p. 6

C. Ablation Studies

The RGBD20K ablations evaluate cross-modal projection, score-guided enhancement, noise selection, and Relation Arbiter design.

Effect of RGB and Depth Alignment Projections

  • Removing modality-specific projections and processing RGB and depth independently without alignment causes a noticeable performance drop.The ablation supports cross-modal projection for effective feature alignment and interaction.Source: C. Ablation Studies · pp. 6–7, Table IV · p. 7

Evaluation of the Score-Guided Enhancement Strategy

  • Jointly enhancing Key and Value representations across both modalities achieves the best results; enhancing either component or modality alone underperforms.The complete score-guided strategy refines both attention computation and feature retrieval.Source: C. Ablation Studies · p. 7, Table IV · p. 7

Effect of RGB and Depth noise selection

  • Replacing layer-specific learnable noise with simple Multiply reduces performance by 0.2% (46.1% mIoU).The best performance uses a learnable parameter defined independently for each layer.Source: C. Ablation Studies · p. 7, Table V · p. 7

Effect of Relation Arbiter Design

VI. CONCLUSIONS

RGBD20K combines a large, fine-grained RGB-D dataset with refined annotations, and SPF achieves state-of-the-art performance across the evaluated benchmarks.

VI. CONCLUSIONS

  • RGBD20K provides 20,000 RGB-D image pairs annotated across 160 fine-grained categories.Its dense, depth-aligned ground truth is presented as a high-fidelity foundation for training general-purpose perception models.Source: §VI · p. 7
  • The paper extensively evaluates representative segmentation models on RGBD20K to establish a baseline for future research.Source: §VI · p. 7
  • SPF achieves state-of-the-art performance across all evaluated benchmarks.The conclusion presents this as evidence of the method’s effectiveness.Source: §VI · p. 7
  • The released benchmark aims to advance next-generation RGB-D semantic perception for robotic and autonomous systems.Source: §VI · p. 7
Loading 2609.29028v1…