PaperlayerView exact revision on arXiv

Source-linked AI summary

DC-Scene: Data-Centric Learning for 3D Scene Understanding

Ting Huang, Zeyu Zhang, Ruicheng Zhang, Yang Zhao

arXiv:2505.15232v1cs.CV

TL;DR

3D scene learning faces high computational costs and scarce high-quality annotations. DC-Scene combines CLIP-based quality filtering with a curriculum that expands training from the top-25% to the top-75% of pairs; that subset reduces training time while maintaining or sometimes improving captioning performance.

  • Problem

    3D scene learning has high computational costs and slower training, while high-quality annotated datasets are scarce.

    Source: §1 · p. 2

  • Method

    DC-Scene uses CLIP alignment and caption-loss perplexity to filter scene–caption pairs, then progressively expands training from the top-25% to the top-75% subset.

    Source: §1 · p. 2, §3.3 · p. 5, §5 · p. 10

  • Results

    With 360 training epochs, the top-75% DIQ split reaches 86.10 CIDEr on ScanRefer and 53.60 CIDEr on Nr3D, reducing training time by two-thirds while maintaining or sometimes improving performance.

    Source: §4.3 · pp. 7–8

  • Takeaways & Limitations

    A moderate subset of high-quality data can reduce training time while preserving, and in some cases improving, overall captioning performance.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

3D scene understanding plays a fundamental role in vision applications such as robotics, autonomous driving, and augmented reality. However, advancing learning-based 3D scene understanding remains challenging due to two key limitations: (1) the large scale and complexity of 3D scenes lead to higher computational costs and slower training compared to 2D counterparts; and (2) high-quality annotated 3D datasets are significantly scarcer than those available for 2D vision. These challenges underscore the need for more efficient learning paradigms. In this work, we propose DC-Scene, a data-centric framework tailored for 3D scene understanding, which emphasizes enhancing data quality and training efficiency. Specifically, we introduce a CLIP-driven dual-indicator quality (DIQ) filter, combining vision-language alignment scores with caption-loss perplexity, along with a curriculum scheduler that progressively expands the training pool from the top 25% to 75% of scene-caption pairs. This strategy filters out noisy samples and significantly reduces dependence on large-scale labeled 3D data. Extensive experiments on ScanRefer and Nr3D demonstrate that DC-Scene achieves state-of-the-art performance (86.1 CIDEr with the top-75% subset vs. 85.4 with the full dataset) while reducing training cost by approximately two-thirds, confirming that a compact set of high-quality samples can outperform exhaustive training. Code will be available at https://github.com/AIGeeksGroup/DC-Scene.

1 Introduction

3D scene understanding is constrained by high training costs and scarce high-quality annotations. DC-Scene addresses these challenges with data-quality filtering and a curriculum that can be integrated across captioning backbones.

1 Introduction

  • Filtering out the lowest-quality 80% of samples cuts training time by 75%.This CLIP-score–based filtering result motivates the data-centric strategy.Source: Figure 1 · p. 1
  • 3D training is computationally demanding, and high-quality annotated datasets are scarce compared with those available for 2D vision.The introduction notes that ScanRefer contains annotations for only around 800 scenes.Source: §1 · p. 2
  • DC-Scene uses CLIP scene–caption alignment to prioritize high-quality pairs and gradually incorporates more challenging or noisy samples.The staged curriculum starts with clearer and simpler examples, which the authors say mitigates overfitting and improves generalization.Source: §1 · p. 2
  • DC-Scene unifies these data-centric strategies in a framework described as flexible across 3D scene-captioning backbones.The paper demonstrates it using 3D CoCa but says the methods are not limited to that architecture.Source: §1 · p. 2
  • 86.10 CIDEr@0.25 on ScanRefer and 53.60 CIDEr@0.50 on Nr3D are reported with only one-third of full-data baseline training epochs.The experiments use the state-of-the-art 3D CoCa architecture, while the framework is described as not limited to that backbone.Source: §1 · p. 2

Related Works

Prior 3D scene-captioning work ranges from proposal-based pipelines to end-to-end models, while data-centric approaches use curricula and sample selection. 3D CoCa is identified as the current state of the art.

Related Works

  • Detect-then-describe systems localize objects before generating captions, but reliance on precomputed proposals can propagate detection errors.Scan2Cap pioneered dense 3D captioning, while MORE and SpaCap3D model inter-object relationships.Source: Related Works · p. 3
  • End-to-end transformer approaches include Vote2Cap-DETR’s direct set prediction and D3Net’s shared speaker–listener representations for captioning and grounding.D3Net uses shared representations to enable semi-supervised training and more distinctive object descriptions.Source: Related Works · p. 3
  • 3D CoCa is described as the current state of the art, combining contrastive vision-language learning with direct caption generation using a frozen CLIP backbone.It aligns 3D scene features with text without relying on external object proposals.Source: Related Works · p. 3
  • Data-centric work includes easy-to-hard curricula, reliability-based pseudo-label filtering in SPSL-3D [2], and text-aligned view ranking in DiffuRank [24].The related-work discussion also notes curriculum learning in LiDAR-based 3D object detection.Source: Related Works · p. 3, Figure 2 · p. 4

3 Methodology

DC-Scene organizes 3D scene captioning around CLIP-based quality scoring, a dual-indicator filter, and a curriculum scheduler. It selects scene-caption pairs by semantic alignment and caption loss, then progressively trains on increasingly complex data.

3.1 Overview

  • DC-Scene combines quality scoring, a dual-indicator quality filter, and a curriculum scheduler to optimize 3D scene captioning.The framework is organized around these three components.Source: §3.1 · p. 4

3.2 Quality Scoring

  • CLIP scene and text embeddings are dotted to score semantic coherence; pairs within empirical score bounds form the Data of Intermediate Similarity (DIS).The bounds are typically set at the 5th and 95th percentiles of the CLIP score distribution.Source: §3.2 · p. 5

3.3 Dual-Indicator Quality (DIQ) Filter

  • The DIQ filter combines CLIP alignment scores with caption loss, using intermediate ranges on both dimensions to select samples with strong alignment and moderate linguistic complexity.Caption loss serves as a proxy for perplexity; the selected samples form the two-dimensional DIQ region.Source: §3.3 · pp. 5–6

3.4 Curriculum Scheduler

  • The curriculum divides the DIQ region into stages, progressively increasing CLIP-score and loss thresholds to introduce data of increasing complexity.Its phases are Initialization, Intermediate, and Advanced; the authors state that this progression improves robustness and generalization and reduces overfitting risk.Source: §3.4 · p. 6

4.1 Dataset and Evaluation Metrics

The evaluation uses ScanRefer and Nr3D, with dataset-specific description, object, and scene counts. Captioning performance is assessed using four standard metrics.

4.1.1 Datasets

  • ScanRefer contains 36,665 descriptions of 7,875 objects across 562 scenes, while Nr3D contains 32,919 descriptions of 4,664 objects across 511 scenes.Evaluation uses 9,508 ScanRefer descriptions and 8,584 Nr3D descriptions from the ScanNet validation split.Source: §4.1.1 · pp. 6–7

4.1.2 Evaluation Metrics

  • Captioning performance is evaluated with CIDEr, BLEU-4, METEOR, and ROUGE-L.The paper denotes these metrics C, B-4, M, and R, respectively.Source: §4.1.2 · p. 7

4.2 Implementation Details

Experiments use two captioning backbones with unchanged architectures and a three-phase DIQ curriculum. Figure 3 presents an ablation comparing DIQ regions on both benchmarks.

4.2 Implementation Details

  • Training uses 3D CoCa and Vote2Cap-DETR++ with unchanged architectures, a single RTX 4090, and DIQ pools expanding from the top 25% to 50% to the full qualified set.The phases span epochs 1–360, 361–720, and 721–1,080.Source: §4.2 · p. 7
  • Figure 3 compares ablations of different DIQ regions on ScanRefer and Nr3D.The supplied caption identifies the figure’s datasets and comparison scope.Source: Figure 3 · p. 7

4.3 Comparative Study

DC-Scene evaluates two backbones on ScanRefer and Nr3D, where the top-75% DIQ split offers a strong efficiency–accuracy trade-off.

4.3 Comparative Study

  • The evaluation uses 3D CoCa and Vote2Cap-DETR++ on the ScanRefer and Nr3D benchmarks.Source: §4.3 · p. 7
  • With 360 epochs, 3D CoCa reaches 86.10 CIDEr on ScanRefer and 53.60 CIDEr on Nr3D; the top-75% DIQ split offers the best trade-off.Training on a moderate high-quality subset reduces training time by two-thirds while maintaining, and sometimes improving, captioning performance.Source: §4.3 · pp. 7–8, Table 1 · p. 8
  • Table 1 reports ScanRefer results at IoU 0.25 and Nr3D results at IoU 0.50, without additional 2D input.It lists CIDEr, BLEU-4, METEOR, and ROUGE-L.Source: Table 1 · p. 8

4.4 Ablation Study

The ablations examine how DIQ score–loss regions relate to captioning quality and how quality-driven sampling organizes candidate pairs into a curriculum.

4.4.1 Effectiveness of DIQ

  • Regions with higher CLIP scores and moderate-to-high caption-loss values yield the strongest captioning outcomes across ScanRefer and Nr3D.The analysis partitions each dataset into nine score–loss regions and samples 7,000 scene–caption pairs per region.Source: §4.4.1 · p. 8

4.4.2 Effectiveness of quality-driven sampling

  • Figure 4 visualizes ScanRefer candidate pairs along a curriculum from compact, semantically aligned samples toward noisier, more challenging ones.The authors report faster convergence and improved generalization along this structured trajectory.Source: §4.4.2 · p. 9, Figure 4 · p. 9

4.5 Qualitative Result

The qualitative comparison presents generated captions for three ScanRefer validation scenes alongside a full-data baseline and human annotations.

4.5 Qualitative Result

  • Figure 5 compares captions from the top-75% DIQ DC-Scene model with a full-data baseline and human-annotated ground truth.The baseline captions are pink, DC-Scene captions red, and ground truth green.Source: Figure 5 · p. 9
  • The comparison covers three representative validation scenes from ScanRefer.Source: Figure 5 · p. 9
  • The DC-Scene captions are presented in comparison to a strong full-data baseline.Source: Figure 5 · p. 9, §4.5 · p. 10

5 Conclusion

DC-Scene combines a CLIP-based DIQ filter with a three-stage curriculum that expands training from the top 25% through the top 50% to the top 75% of samples. On two benchmarks, the top-75% subset outperforms the full-data baseline with one-third of the training epochs, with consistent improvements also reported for another backbone.

5 Conclusion

  • The top-75% DIQ subset achieves 86.10 CIDEr at IoU 0.25 on ScanRefer and 53.60 CIDEr at IoU 0.50 on Nr3D with 3D CoCa.It surpasses the full-data baseline with one-third of the training epochs.Source: §5 · p. 10
  • DC-Scene uses a CLIP-based DIQ filter and progressively expands its curriculum from top-25% to top-50% and top-75% DIQ subsets.Source: §5 · p. 10
  • Consistent improvements with Vote2Cap-DETR++ support the approach's effectiveness across different model architectures.Source: §5 · p. 10
Loading 2505.15232v1…