PaperlayerView exact revision on arXiv

Source-linked AI summary

PLA: Language-Driven Open-Vocabulary 3D Scene Understanding

Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, Xiaojuan Qi

arXiv:2211.16312v2cs.CV

TL;DR

3D models trained on human-annotated datasets struggle with unseen categories, while large-scale 3D-text pairs are inaccessible. PLA distills vision-language knowledge through captioned multi-view images, hierarchical point-language pairs, and contrastive learning, outperforming baselines in open-vocabulary segmentation and transferring across datasets.

  • Problem

    Human-annotated 3D models recognize only dataset categories, while large-scale 3D-text pairs are inaccessible for transferring 2D open-vocabulary advances.

  • Method

    PLA captions images aligned with 3D data, constructs hierarchical point-language pairs using geometric constraints, and applies contrastive learning to connect 3D and text.

  • Results

    PLA surpasses baselines by 25.8% ∼44.7% hIoU and 14.5% ∼50.4% hAP50, while transferring from ScanNet to S3IDS under data-distribution and category shifts.

  • Takeaways & Limitations

    PLA enables language-aware 3D models to localize and recognize novel categories in in-domain open-vocabulary and out-of-domain zero-shot transfer tasks.

  • Takeaways & Limitations

    The framework remains limited by over-confident base-class predictions and incomplete novel-class instance masks during proposal grouping.

Abstract

from arXiv · show

Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. The recent breakthrough of 2D open-vocabulary perception is largely driven by Internet-scale paired image-text data with rich vocabulary concepts. However, this success cannot be directly transferred to 3D scenarios due to the inaccessibility of large-scale 3D-text pairs. To this end, we propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D, which allows explicitly associating 3D and semantic-rich captions. Further, to foster coarse-to-fine visual-semantic representation learning from captions, we design hierarchical 3D-caption pairs, leveraging geometric constraints between 3D scenes and multi-view images. Finally, by employing contrastive learning, the model learns language-aware embeddings that connect 3D and text for open-vocabulary tasks. Our method not only remarkably outperforms baseline methods by 25.8% $\sim$ 44.7% hIoU and 14.5% $\sim$ 50.4% hAP$_{50}$ in open-vocabulary semantic and instance segmentation, but also shows robust transferability on challenging zero-shot domain transfer tasks. See the project website at https://dingry.github.io/projects/PLA.

1. Introduction

3D open-vocabulary understanding seeks to recognize unseen categories despite limited 3D-text data and costly annotation. PLA bridges 3D scenes and language through captioned multi-view images, hierarchical geometric associations, and contrastive learning, improving segmentation and transfer.

  • 3D scene understanding supports robot manipulation, virtual reality, and human-machine interaction but conventional models recognize only annotated categories.
  • 22.3 minutes of annotation for one 20-class scene makes manually covering all real-world categories infeasible.
  • 17.8% mIoU with 20-fold higher latency shows the limitations of applying MaskCLIP to projected 2D ScanNet images.
  • PLA captions images aligned with 3D point data to distill vocabulary-rich textual descriptions and explicitly associate 3D content with language.
  • Hierarchical scene-, view-, and entity-level point-caption pairs use multi-view geometric relationships to support coarse-to-fine visual-semantic learning.
  • 25.8% ∼44.7% hIoU and 14.5% ∼50.4% hAP50 gains over baselines span in-domain semantic and instance segmentation, with transfer to S3IDS.

3. Method

PLA converts a close-set 3D scene network into an open-vocabulary model by matching point features with text-derived category embeddings and calibrating base-versus-novel predictions. It further supplies caption supervision through geometrically aligned, hierarchical point-language associations from multi-view images.

  • 3.1. Preliminary: Open-vocabulary inference divides categories into base and novel classes: training uses base annotations, while inference predicts across both category sets.The model accesses point clouds and base annotations during training but must localize and classify both base and novel categories at inference.
  • 3.2.1 Text-Embedded Semantic Classifier: The text-embedded semantic classifier replaces the learnable classifier with category embeddings from a frozen text encoder and scores projected point features by similarity.A vision-language adapter matches 3D feature dimensions to text embeddings; training uses base-category embeddings, while inference can use desired base and novel categories.
  • 3.2.2 Semantic Calibration with Binary Head: The binary calibration module predicts whether points belong to novel categories and uses that probability to rectify semantic scores that are over-confident toward base classes.The binary head is trained with binary cross-entropy on base-versus-novel labels and corrects semantic predictions during inference.
  • 3.3. Image-Bridged Point-Language Association: PLA generates captions for multi-view images with a pretrained image-captioning model, then associates each caption with 3D points using image-to-scene geometric projection.This image bridge provides language supervision without human annotation, using images aligned with the corresponding 3D scene.
  • 3.3.2 Associate Point Cloud with Language: Hierarchical scene-, view-, and entity-level associations provide progressively finer point-caption correspondence for coarse-to-fine language-aware 3D representation learning.Scene captions supervise all points, view captions map to points inside image frustums, and entity captions are formed from extracted entity words and associated point sets.

4. Experiments

Experiments evaluate PLA on ScanNet and S3DIS across open-vocabulary semantic and instance segmentation, including category shifts, self-training, and zero-shot domain transfer.

  • Datasets and Metrics: Experiments use ScanNet and S3DIS for semantic and instance segmentation, with metrics computed separately on base and novel classes.hIoU and hAP50 summarize performance across the base/novel category partition.
  • Category Partitions: Open-vocabulary benchmarks use multiple base/novel partitions while excluding ScanNet’s “otherfurniture” and S3DIS’s “clutter” classes.These classes lack exact semantic meanings and may contain arbitrary categories.
  • Semantic Segmentation: PLA exceeds LSeg-3D by 51.3% ∼65.3% hIoU on ScanNet and 34.5% ∼38.5% hIoU on S3DIS across semantic-segmentation partitions.It also surpasses 3DGenZ and 3DTZSL on ScanNet and improves over PLA without caption supervision.
  • Instance Segmentation: 29.2% ∼50.4% hAP50 and 14.5% ∼14.9% hAP50 gains are reported on ScanNet and S3DIS, respectively, for instance segmentation.The smaller S3DIS improvement is attributed to fewer training scenes and fewer point-caption pairs.
  • Self-Bootstrap: Self-training with novel category names yields around 2% ∼12% gains across semantic and instance segmentation on both datasets without human annotation.The variant extends the model’s vocabulary through self-bootstrap.
  • Zero-Shot Domain Transfer: Training on ScanNet and testing on all S3DIS classes without fine-tuning evaluates transfer under both category and data-distribution shifts.S3DIS includes four categories absent from ScanNet; PLA consistently outperforms LSeg-3D by 7.7% ∼18.3% mIoU.

5. Ablation Studies

Ablations identify binary calibration, fine-grained caption associations, diverse entity words, CLIP text encoding, and stronger captioning foundations as important contributors to PLA’s performance.

  • Study Setup: Ablations are conducted on the ScanNet B15/N4 partition, with the default configuration marked in gray.The component analysis includes zero-shot domain-transfer results and caption-supervision variants.
  • Binary Calibration: 39.8% hIoU and 15.9% hAP50 improvements over LSeg-3D result from adding the binary calibration head for semantic and instance segmentation.The gains on both base and novel classes support its role in rectifying semantic scores.
  • Caption Supervision: 14.8% ∼23.8% hIoU and 31.8% ∼35.6% hAP50 improvements arise from point-caption association variants, with entity-level supervision performing best.The study links the strongest results to fine-grained language-point correspondence.
  • Caption Composition: Keeping only entity phrases can outperform full captions, while restricting entities to exact category names causes over 13% mIoU degradation on novel categories.Diverse entity words expand the semantic space, whereas foundation-model-generated captions remain slightly stronger than manually composed labels.
  • Text Encoder Selection: CLIP’s vision-language text encoder achieves over 7% higher mIoU than BERT and GPT2 text-only encoders.The comparison supports using vision-aware language embeddings for 3D-language tasks involving texture, shape, and RGB information.
  • Captioning Foundation Model: OFA consistently surpasses GPT-ViT2 across three partitions, indicating that stronger image-captioning foundation models can further boost PLA.GPT-ViT2 is the default captioning model in the main experiments.

6. Qualitative Analysis

Qualitative results show PLA recognizing synonymical classes, abstract scene concepts, and categories that were unannotated in the dataset.

  • Synonymical Novel Classes: Replacing “sofa” with “couch” or “refrigerator” with “freezer” still produces high-quality segmentation masks.These substitutions test recognition of synonymical novel classes at inference.
  • Abstract Novel Classes: Adding “bathroom” or “kitchen” enables the model to roughly cover corresponding room regions using abstract concepts.The bathroom example removes object categories such as “shower curtain,” “toilet,” “sink,” and “bathtub.”

7. Conclusion

PLA enables language-driven 3D scene understanding for localizing and recognizing novel categories. It uses hierarchical point-language pairs and contrastive learning, with strong in-domain and out-of-domain transfer results.

  • PLA constructs hierarchical point-language pairs by combining 2D vision-language foundation models with geometric constraints between 3D scenes and images.Contrastive learning brings associated 3D and language features closer, introducing semantic concepts into the 3D network.

Outline

The supplementary file provides additional benchmark details, analyses, experiments, examples, qualitative results, and limitation discussions for PLA.

  • Supplementary contents: S1 details the open-vocabulary scene understanding benchmark.
  • Supplementary contents: S2 analyzes PointCLIP limitations for scene understanding tasks.
  • Supplementary contents: S3 reports additional partition, per-class, error-bar, fully supervised, caption-supervision, and supervision-combination experiments.
  • Supplementary contents: S4 presents image-caption and hierarchical point-caption examples.
  • Supplementary contents: S5 provides qualitative open-vocabulary scene-understanding results.
  • Supplementary contents: S6 discusses limitations and open problems.

S1. Implementation Details

The implementation extends a sparse-convolutional 3D network into an open-vocabulary learner using VL adapters, fixed text embeddings, and class-agnostic instance components.

  • Benchmark and network: The benchmark uses ScanNet and S3DIS with multiple base/novel category partitions for semantic and instance segmentation.ScanNet contains 1,613 scenes and 19 retained classes; S3DIS contains 271 scans and 12 retained categories.
  • Benchmark and network: PLA uses a sparse-convolutional UNet backbone, replacing the close-set semantic head with a VL adapter and fixed text-encoder category embeddings.The category embeddings function as classifier weights for open-vocabulary prediction.
  • Benchmark and network: The open-vocabulary instance head modifies segmentation and scoring modules to be class-agnostic and uses a classification design that does not require novel-category labels.
  • Baselines and training: LSeg-3D, 3DGenZ, and 3DTZSL are reproduced with matched UNet and CLIP text-embedding components for fair comparison.
  • Baselines and training: Training uses AdamW with cosine learning-rate decay and batch size 32 across ScanNet and S3DIS experiments.Experiments run on 8 NVIDIA V100 or A100 GPUs.
  • Caption supervision: Caption supervision uses scene-, view-, and entity-level pairs, with entity filtering and dataset-specific loss weights; S3DIS scenes without images receive no language supervision.

S2. Analysis of PointCLIP for Scene Understanding

PointCLIP’s projection-based approach performs poorly for 3D scene semantic segmentation and is computationally expensive, motivating direct 3D processing.

  • Performance and efficiency: 2.2% mIoU is obtained with depth-image input on 5,436 ScanNet validation samples, while assembled 3D predictions reach only 1.7% mIoU on 312 samples.Using RGB images raises 2D performance to 17.8% mIoU, but projection still loses 3D information.
  • Limitations of projection: Projection from 3D to 2D causes information loss and prevents direct learning from information-rich 3D data.
  • Performance and efficiency: PointCLIP requires 1667ms per 3D scene, compared with 83ms for PLA’s 3D network.The reported processing-time comparison makes the 3D network 20 times more efficient, excluding PointCLIP postprocessing.
  • Limitations of projection: Poor zero-shot performance, projection information loss, and heavy computation costs make this approach unsuitable for 3D scene understanding.

S3. Additional Experimental Results

Additional experiments assess robustness across category splits, class types, repeated runs, supervision settings, and caption-loss combinations.

  • Robustness: 53.1% ∼65.3% hIoU improvement over LSeg-3D is reported across four resampled B15/N4 ScanNet semantic-segmentation splits.The resampling procedure tests robustness to different novel classes.
  • Per-class performance: Novel classes generally perform worse than base classes, and ‘bookshelf’ drops 7.4% mIoU when enlarging the novel-category set from B15/N4 to B12/N7.The paper attributes this degradation to insufficient seen-category data for tuning.
  • Repeatability: Repeated experiments show lower standard deviations for base classes than novel classes, while ScanNet results are more stable than S3DIS results.The reported variability indicates higher confidence uncertainty for novel-class predictions.
  • Supervision comparison: Caption supervision neither hinders nor enhances fully supervised performance, whose access to annotations for all categories already provides complete category supervision.This comparison is presented as a fairness check for the fully supervised baseline.
  • Caption-supervision ablation: Combining three captions increases hIoU by 0.6%, but the required loss-weight balance is dataset-dependent and is therefore omitted for generalization.The paper leaves more effective caption-supervision combinations for future work.

S4. Caption Examples

Caption examples illustrate rich semantic descriptions and hierarchical associations, while qualitative results show recognition and localization of novel categories.

  • Caption examples: Generated captions describe entities, room types, textures, colors, and spatial relationships, including uncommon classes such as ‘buddha statue’.These descriptions provide a broad vocabulary of semantic clues.
  • Hierarchical associations: Hierarchical point-caption pairs use geometric constraints to associate scene-level descriptions with whole areas and view-level descriptions with local 3D details.The examples contrast global scene supervision with more localized view-frustum supervision.
  • Semantic segmentation: The framework recognizes novel categories with accurate semantic masks, whereas LSeg-3D confuses unseen classes with seen classes.The qualitative comparison attributes the semantic improvement to point-caption association injecting rich semantic concepts.
  • Instance segmentation: The framework produces accurate instance masks, while LSeg-3D misses novel objects or predicts incomplete masks.The comparison reflects stronger generalized localization in the framework’s instance predictions.

S6. Limitation and Open Problems

The supplementary discussion identifies calibration, data-scale, and caption-supervision limitations that leave several directions open.

  • Calibration: The semantic binary calibration head fails to rectify out-of-domain transfer predictions because it is trained on dataset-specific base/novel partitions.The paper calls for more transferable score-calibration modules.
  • Calibration: Novel-class proposal grouping can remain poorly calibrated, producing incomplete instance masks despite fine-grained point-caption pairs improving localization.The paper leaves this instance-calibration issue as an unresolved challenge.
  • Dataset scale: S3DIS performs slightly worse than ScanNet because it has less sample size, diversity, and fewer point-caption associations.The paper suggests pre-training on a larger semantically rich dataset before fine-tuning small datasets as future work.
  • Caption supervision: Combining caption losses requires sophisticated trade-offs that are not universally applicable across datasets, so the scene-level caption is excluded for generalization.More effective combinations are left for future investigation.
Loading 2211.16312v2…