PaperlayerView exact revision on arXiv

Source-linked AI summary

RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding

Jihan Yang, Runyu Ding, Weipeng Deng, Zhe Wang, Xiaojuan Qi

arXiv:2304.00962v4cs.CVcs.AI

TL;DR

Open-world 3D scene understanding must recognize open-set objects despite scarce dense 3D annotations and limitations in existing forms of pseudo-supervision. RegionPLC builds dense regional point-language supervision from multiple 2D models and learns with a region-aware contrastive objective. It reports average gains over prior open-world methods across semantic and instance segmentation while using fewer resources.

  • Problem

    Open-world 3D recognition is important for real-world scenes, but dense semantic annotations are scarce and existing feature-distillation or caption-based supervision has resource or density limitations.

    Source: §1 · p. 1

  • Method

    RegionPLC fuses region-level captions from multiple 2D vision-language models into 3D-language pairs and learns from them with a region-aware point-discriminative contrastive objective.

    Source: §1 · pp. 1–2

  • Results

    RegionPLC outperforms prior open-world methods by average gains of 17.2% in unseen-category mIoU for semantic segmentation and 9.1% in unseen-category mAP50 for instance segmentation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RegionPLC performs across indoor and outdoor settings and can be integrated with language models for grounded 3D reasoning without task-specific training data.

    Source: §1 · p. 2, §7 · p. 8

    The training setting provides human annotations for base categories but none for novel categories, which the model must recognize alongside base categories at inference.

    Source: §3.1 · p. 2

Abstract

from arXiv · show

We propose a lightweight and scalable Regional Point-Language Contrastive learning framework, namely \textbf{RegionPLC}, for open-world 3D scene understanding, aiming to identify and recognize open-set objects and categories. Specifically, based on our empirical studies, we introduce a 3D-aware SFusion strategy that fuses 3D vision-language pairs derived from multiple 2D foundation models, yielding high-quality, dense region-level language descriptions without human 3D annotations. Subsequently, we devise a region-aware point-discriminative contrastive learning objective to enable robust and effective 3D learning from dense regional language supervision. We carry out extensive experiments on ScanNet, ScanNet200, and nuScenes datasets, and our model outperforms prior 3D open-world scene understanding approaches by an average of 17.2\% and 9.1\% for semantic and instance segmentation, respectively, while maintaining greater scalability and lower resource demands. Furthermore, our method has the flexibility to be effortlessly integrated with language models to enable open-ended grounded 3D reasoning without extra task-specific training. Code is available at https://github.com/CVMI-Lab/PLA.

1. Introduction

Open-world 3D recognition matters for real-world scenes but is constrained by scarce, hard-to-scale dense annotations and limitations in existing pseudo-supervision. RegionPLC combines region-level 3D-language pairs from multiple 2D models with region-aware contrastive learning, and reports gains over prior methods across semantic and instance segmentation.

1. Introduction

  • Dense 3D semantic annotations are scarce and difficult to gather and scale to large vocabularies, despite open-set recognition's importance in real-world applications [3] [43].Source: §1 · p. 1
  • Feature-distillation methods face 2D feature-quality limits and resource-intensive extraction, fusion, and storage, while caption-based point-language supervision is sparse [26] [9].These constraints motivate denser region-level 3D-language associations that combine strengths across foundation models.Source: §1 · p. 1
  • RegionPLC fuses diverse region-level captions from 2D vision-language models, maps them to 3D, and trains with a region-aware point-discriminative contrastive objective.The framework uses dense regional language supervision for open-world 3D learning.Source: §1 · pp. 1–2
  • Its region-aware objective limits interference from nearby points of unrelated categories and normalizes contributions across region sizes, supporting robust point features.The framework can also integrate with language models for open-ended grounded 3D reasoning without task-specific training data.Source: §1 · p. 2
  • RegionPLC gains average 17.2% in unseen-category mIoU for semantic segmentation and 9.1% in unseen-category mAP50 for instance segmentation over existing open-world methods.Experiments cover ScanNet, ScanNet200, and nuScenes, including indoor and outdoor scenarios.Source: §1 · p. 2

2. Related Work

Prior open-world 3D work either uses 2D foundation models at inference or trains 3D backbones with transferred dense features or point-language pairs. These approaches face inference overhead, training costs, inherited 2D failure modes, or coarse text supervision.

3D Scene Understanding

Open-world 3D Understanding

  • Inference-stage methods fuse 2D open-world predictions into 3D, but incur computation and storage overhead and may not adequately address 3D geometry [16] [17] [18] [26] [31] [46].Source: §2 · p. 2
  • Feature-distillation methods transfer dense 2D features into 3D backbones, but can retain high training costs and 2D prediction failure modes [26] [45].Source: §2 · p. 2
  • Ding et al. obtain point-language pairs through image captioning for scalable, large-vocabulary 3D training, but their supervision remains coarse [9] [10].Source: §2 · p. 2

3. RegionPLC

RegionPLC trains open-world 3D scene models from regional language supervision generated by multiple 2D foundation models. It studies source complementarity, fuses spatially supplementary captions, and uses point-discriminative learning for 3D representations.

3.1. Overview

  • The open-world setup trains with annotations for base categories but not novel ones, then predicts both; CLIP text embeddings replace the classification weights for category querying.A 3D encoder and vision-language adapter produce point features and semantic scores, with a localization network for instance proposals.Source: §3.1 · p. 2

3.2. Regional 3D-Language Association from 2D Foundation Models

  • Five regional-caption sources come from detector templates, prompted image captioning, and dense captioners, then projected 2D regions are associated with corresponding 3D point sets.Sliding windows trade localization precision for vocabulary openness; detector proposals localize more accurately but use predefined labels, while dense captioners can overlook small or distant objects.Source: §3.2 · pp. 3–4

3.3. Benchmark and Analysis on Regional 3D-Language Pairs

  • No regional-language source wins across settings: detector templates excel in ScanNet B12/N7 with 125K images, while dense captioners excel in the annotation-free setting.The tested settings vary novel-category and image quantities; reported fusion gains are inconsistent or incremental.Source: Table 1 · p. 4, §3.3 · p. 4

3.4. Boost Synergy of Diverse 3D-language Sources

  • SFusion selects a primary caption source and iteratively adds candidate pairs with intermediate point-set overlap, yielding spatially supplementary supervision while retaining a controlled majority of primary pairs.The overlap thresholds filter highly overlapping pairs that may be redundant or conflicting; the reported experiments show SFusion boosts multi-source combinations.Source: §3.4 · pp. 4–5

3.5. Region-aware Point-discriminative Contrastive Learning

  • Point-discriminative loss matches point-wise embeddings to captions, while region-size normalization equalizes gradient scales and conflicting captions discourage noisy optimization.Unlike pooled region features, point-wise matching lets each point’s optimization adapt to its own embedding.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate RegionPLC on ScanNet, ScanNet200, and nuScenes across base-annotated and annotation-free settings, semantic and instance segmentation, and qualitative indoor and outdoor scenes.

4.1. Basic Setups

  • The evaluation spans three datasets, base-annotated and annotation-free settings, semantic and instance segmentation, base/novel partitions, and foreground-only metrics for annotation-free semantic tests.The datasets are ScanNet, ScanNet200, and nuScenes.Source: §4.1 · p. 6

4.2. Base-annotated Open World

  • Unseen-category mIoU rises 8.3% ∼21.8% over PLA [9] on ScanNet and nuScenes; ScanNet instance segmentation gains 4.3% ∼12.3% over PLA [9] across three partitions.Source: §4.2 · p. 6, Table 3 · p. 6

4.3. Annotation-free Open World

  • 1.8% ∼ 57.5% mIoU† and 7.2% ∼72% mAcc† gains over competitive methods are reported for annotation-free semantic segmentation on ScanNet.Source: §4.3 · p. 6, Table 4 · p. 7
  • On long-tail ScanNet200, RegionPLC surpasses other counterparts by 3.2% ∼7.4% mIoU† and 7.1% ∼14.2% mAcc†.Source: Table 5 · p. 7

4.4. Qualitative Studies

  • Figure 3 examples show annotation-free recognition and base-annotated segmentation of “keyboard,” “laddar,” and “motorcycle,” plus novel-instance grouping.The examples span indoor and outdoor scenes.Source: §4.4 · p. 7

5. Ablation Study

Ablations examine regional language supervision, contrastive-loss variants, and SFusion’s caption integration choices.

  • Base-annotated ablations are measured in hIoU, mIoU_B, and mIoU_N; annotation-free results use mIoU† and mAcc†.Source: §5 · p. 7
  • Regional captions add 4% ∼10.8% across partitions; Lpdc adds 28.9% mIoU† annotation-free, and Lrpdc adds 1.1% ∼3.8% over Lpdc.Source: Table 6 · p. 7
  • SFusion adds 1.9% ∼3.5% over naive mixing; at similar source ratios, low-overlap (<0.5) pairs outperform highly overlapped pairs.Source: Table 6 · p. 7, §5 · p. 8

6. Open-ended Grounded 3D Reasoning

RegionGR integrates RegionPLC and an LLM for open-ended grounded 3D reasoning. Regional 3D-language pairs provide scene context, while RegionPLC locates and identifies objects relevant to user queries.

  • RegionGR uses regional 3D-language pairs as an LLM knowledge base and RegionPLC to coarsely locate and identify scene objects for grounded reasoning.Source: §6 · p. 8
  • Its pipeline initializes scene context, prompts the LLM with that context and the user query, then parses the response and grounds objects with RegionPLC.Source: §6 · p. 8
  • For a specified 3D region, RegionGR retains highly overlapped 3D-language pairs and supports user requests including summarization, reasoning, and planning.Source: §6 · p. 8

7. Conclusion

RegionPLC uses regional point-language contrastive learning to recognize and localize unseen categories in open-world 3D scenes. Experiments report strong performance across indoor and outdoor settings, including long-tail and annotation-free scenarios, and the framework can be integrated with an LLM for grounded 3D reasoning.

7. Conclusion

  • RegionPLC combines regional point-language pairs with a point-discriminative contrastive loss to learn distinctive, robust features from regional captions.The framework uses advanced vision-language models and SFusion to build comprehensive regional point-language pairs.Source: §7 · p. 8
  • RegionPLC outperforms prior open-world methods in indoor and outdoor scenarios and excels in challenging long-tail and annotation-free settings.Source: §7 · p. 8
  • RegionPLC can be integrated with an LLM for grounded 3D visual reasoning.Source: §7 · p. 8

Outline

The supplementary file provides implementation details, category partitions, baseline information, and additional experimental results, alongside qualitative results and RegionGR prompts.

Outline

  • The supplement covers implementation, category partitions, baseline methods, per-class performance, zero-shot domain transfer, error-bar analysis, qualitative results, and RegionGR prompts.Source: Outline · p. 11
  • The supplement adds analyses and results omitted from the main paper due to page-length limits.Source: Outline · p. 11

A. Implementation Details

A.1. Implementation Details of RegionPLC

The supplement details implementation and evaluation choices, compares open-world methods, and reports per-category, repeatability, and transfer results.

A.1. Implementation Details of RegionPLC

  • ScanNet uses SparseUNet16 and ScanNet200 and nuScenes use SparseUNet32; semantic training runs for 512 epochs on ScanNet datasets and 50 on nuScenes.ScanNet instance segmentation runs for 640 epochs; regional captions are extracted from 125K ScanNet frames and all nuScenes images.Source: §A.1 · p. 11

A.2. Category Partition for Base-annotated Results

  • The base-annotated setup partitions categories into base and novel; ScanNet200 uses random splits with 30 or 50 novel categories and assigns 11 validation-absent categories to base.NuScenes excludes the ambiguous “otherflat” category and partitions the remaining 15 categories.Source: §A.2 · pp. 11–12, Table 9 · p. 12, Table 10 · p. 12

A.3. Implementation of Baseline Methods

  • The reproduced baselines adapt MaskCLIP and PointCLIP for 3D segmentation, train OpenScene-3D under matched conditions, and omit PLA’s binary encoder in annotation-free evaluation.PointCLIP-Seg uses depth images for 2D feature extraction, unlike MaskCLIP’s RGB images.Source: §A.3 · p. 12

A.4. Comparisons of 3D Open-world Scene Understanding Methods

  • RegionPLC incorporates advanced 2D models for regional 3D-language association while retaining PLA’s low-storage, scalable training, addressing its sparse and rough language supervision.The comparison with ConceptFusion, OpenScene, and PLA is summarized in Table 11.Source: §A.4 · p. 12, Table 11 · p. 13

B.1. Per-category Results

  • RegionPLC improves PLA’s ScanNet novel-category IoU across all categories and different base-annotated partitions.Table 12 presents the per-class comparison.Source: §B.1 · p. 13, Table 12 · p. 13

B.2. Error Bar

  • Repeated experiments show generally stable RegionPLC performance on ScanNet open-world segmentation across base-annotated and annotation-free settings.The reported error-bar analysis reproduces each experiment three times; Table 13 identifies the metrics for both settings.Source: §B.2 · p. 13

B.3. Zero-shot Domain Transfer

  • 6.8%–37.1% boosts over PLA in mIoU† are reported across ScanNet-to-S3DIS splits without fine-tuning.The authors report that more ScanNet base categories can hinder transfer and that semantic-rich caption supervision achieves out-of-domain generalization.Source: §B.3 · p. 13

C. Qualitative Results for Annotation-free Open World

Qualitative examples illustrate annotation-free segmentation and tail-category discovery. The supplementary RegionGR prompts guide an LLM to use human queries and regional captions for grounded 3D reasoning.

C. Qualitative Results for Annotation-free Open World

D. Prompts for RegionGR

  • RegionGR filters regional captions by overlap with the queried 3D region and supplies them as environmental context for answering the user.The prompt instructs the LLM to ground its response in the supplied context and keep it short.Source: D. Prompts for RegionGR · pp. 14–15

E. Limitation and Future Works

The discussion identifies two improvement directions: more advanced integration of language, 3D, and image features, and adaptive visual prompting. Combining RegionPLC with OpenScene also suggests potential for auxiliary 2D image supervision.

E. Limitation and Future Works

  • Combining RegionPLC with OpenScene demonstrates potential for introducing 2D image features as auxiliary supervision.The current loss combination is described as straightforward; more advanced strategies integrating language, 3D, and image features may lead to better performance.Source: E. Limitation and Future Works · pp. 15–16
  • Visual prompts are predefined before training and remain unchanged, while more adaptive prompting might improve language-supervision quality.Source: E. Limitation and Future Works · p. 16
  • The authors plan to develop an open-world 3D scene-understanding framework that addresses both the feature-combination and prompting limitations.Source: E. Limitation and Future Works · p. 16
Loading 2304.00962v4…