Source-linked AI summary
GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting
Thodoris Betsas, Anastasios Doulamis, Andreas Georgopoulos
TL;DR
GoDeep addresses the reliance of open-vocabulary 3D segmentation on CLIP’s joint vision-language space and on 3D training resources. It translates posed images into structured descriptions, lifts them into a language-only embedding space, and matches strong annotation-free baselines while showing sharper behavior on corrected and genuinely unseen vocabulary.
Problem
Open-vocabulary 3D segmentation commonly lifts CLIP features into 3D, despite CLIP’s bag-of-words behavior and the frequent need for 3D training corpora or dedicated 3D encoders.
Method
GoDeep uses a vision-language model as a translator, grounds structured entity descriptions in posed images, projects them onto points, and aggregates them in a language-only sentence-embedding space without 3D training.
Results
GoDeep matches strong annotation-free ScanNet++ baselines and, after two independent vocabulary corrections, language-space embeddings respond more strongly than CLIP while separating genuinely unseen-object presence from absence more sharply.
Takeaways & Limitations
Keeping representations as discrete text makes predictions traceable to point-level evidence and supports aggregation that favors precise observations and finer-element localization.
Takeaways & Limitations
The pipeline adds substantial per-image and per-scene computation, uses a less reproducible closed-source heritage VLM, and relies on an internal heritage comparison.
Abstract
from arXiv · showhide
Open vocabulary 3D semantic segmentation methods typically lift CLIP features into 3D. This embeds points in a joint vision-language space known to behave like a bag-of-words on compositional tasks. Furthermore, even annotation free variants often require a large 3D training corpus and a dedicated 3D encoder per domain. Instead we use a vision-language model purely as a translator. It produces structured, entity-level descriptions of each posed image. These descriptions are grounded, projected, and aggregated directly in a general-purpose, language-only embedding space, with no 3D training corpus or encoder required. On ScanNet++, our pipeline is competitive with strong annotation free baselines trained on ScanNet. On a 5-building cultural heritage benchmark, raw scores initially favor a CLIP-based variant, but a single systematic vocabulary correction reverses this ranking. An effect confirmed by a second, independent correction on a different class, indicating that language-space embeddings track physical content more faithfully. This fidelity extends to genuinely out-of-vocabulary (OOV) objects on ScanNet++ proving that language-space embeddings separate presence from absence objects far more sharply than CLIP-based embeddings do. GoDeep also localize these OOV objects within the scene, all without any 2D-3D annotation. Because every representation remains discrete text, predictions are also explainable at the point level. Finally, exploiting both a heuristic weighting, that favors precise over merely frequent observations and GoDeep's explainability property, we propose an aggregation strategy, as a proof of concept, that favors finer elements localization.
1 Introduction
GoDeep addresses the limits of CLIP-based 3D representations and domain-specific 3D training by lifting structured VLM descriptions into a language-only embedding space. Its evaluations show competitive annotation-free performance, vocabulary-correction reversals favoring language-space embeddings, and point-level explainability.
- Motivation: Human-annotated point clouds are costly and often omit categories, while closed-set classes absorb finer elements into coarse structural labels.Examples include capitals, shafts, and friezes being absorbed into categories such as columns.
- Motivation: CLIP-based open-vocabulary methods lift joint vision-language features into 3D, but CLIP can behave like a bag-of-words on relational and attribute-binding tasks.This limitation is especially relevant when open-vocabulary segmentation must distinguish compositional properties.
- Motivation: Existing annotation-free approaches still require large 3D training corpora, dense 2D or textual supervision, and a dedicated 3D encoder for each target domain.Their queries also remain mediated by the same CLIP text encoder.
- Approach: GoDeep uses a VLM as a translator, grounds structured entities with an open-vocabulary segmenter, projects them into 3D, and aggregates them in a general-purpose sentence-embedding space.The pipeline uses frozen off-the-shelf models and requires neither a 3D training corpus nor a dedicated 3D encoder.
- Results: On ScanNet++, GoDeep is competitive with strong annotation-free baselines trained on ScanNet, while a single vocabulary correction reverses the heritage benchmark ranking between language-space and CLIP-based embeddings.The correction supports the claim that language-space embeddings track physical content more faithfully when the evaluation vocabulary is precise.
- Interpretability: GoDeep retains discrete textual evidence for point-level traceability, and its weighting favors precise observations over merely frequent ones.This makes predictions explainable and connects the weighting design to the evidence used for final labels.
2 Related Work
Related 3D scene-understanding methods progress from fixed-vocabulary segmentation toward open-vocabulary systems, but most still rely on CLIP features or trained 3D encoders. GoDeep instead motivates language-space representations for preserving relational semantics.
- Closed-set segmentation: Closed-set 3D segmentation methods use diverse point, dimensionality-reduction, discretization, graph, and hybrid representations but remain confined to fixed training vocabularies.Their transferability is also sensitive to the acquisition modality of the training data.
- Open-vocabulary segmentation: Open-vocabulary 3D methods commonly distill CLIP features or caption-based supervision into point clouds using a dedicated 3D encoder.OpenScene, PLA, CLIP-FO3D, Mosaic3D, and SceneSplat exemplify this recipe, while OpenMask3D uses a trained class-agnostic proposal network with CLIP embeddings at inference.
- Language-space representations: CLIP’s contrastive image-text objective can be satisfied by recognizing salient keywords without resolving relations among words, motivating sentence encoders trained for inference and semantic similarity.GoDeep’s language-space choice follows this distinction between keyword matching and relational semantic parsing.
3 Method
GoDeep converts posed images into structured textual entities, grounds and projects them into the point cloud, weights observations by reliability, and robustly aggregates them into per-point sentence embeddings.
- Scene description: A VLM produces dense descriptions of each posed image, and a lightweight language model converts them into JSON entities with labels, prompts, and optional attributes.The structured representation filters task-irrelevant content and records defects or conditions separately.
- Entity grounding: Each entity prompt is grounded to image pixels with SAM3, and overlapping masks allow a point to accumulate evidence from multiple entities.This supports simultaneous evidence for structural elements and surface defects.
- Multi-view lifting: Visible point-camera-mask triples are selected after projection and occlusion handling using calibrated camera geometry and a point-cloud depth buffer.Candidate views are filtered before observation weights are assigned.
- Reliability weighting: Observation weights favor closer, more central, and more precisely segmented masks through depth, image-center distance, and relative mask area.The weighting measurably reorders which evidence dominates a point’s final label.
- Language-space aggregation: Mean-inliers pooling computes a weighted mean, removes observations below a cosine-similarity threshold, and recomputes the normalized embedding from the remaining inliers.This reduces the influence of occasional hallucinated descriptions or mis-grounded masks.
4 Experiments
The experiments evaluate GoDeep on ScanNet++ and a five-building cultural heritage benchmark using argmax predictions and standard segmentation metrics. The language-only variant performs competitively without 3D training, while vocabulary corrections substantially improve its heritage results.
- Datasets: The evaluation covers 50 ScanNet++ validation scenes with a top-100 vocabulary and five historic buildings annotated with 10 ARCH classes.The heritage buildings combine terrestrial laser scanning and photogrammetry.
- Evaluation protocol: Predictions assign each point the class whose text embedding has maximum cosine similarity with its aggregated embedding, without per-class thresholds.ScanNet++ reports f-mIoU and f-mAcc excluding wall, floor, and ceiling; the heritage benchmark reports mIoU, mAcc, and mPrec across all 10 classes.
- ScanNet++ results: GoDeep performs on par with CLIP and in several cases matches or exceeds ScanNet-trained methods without any 3D training of its own.Table 1 reports f-mIoU and f-mAcc under the argmax protocol, with qualitative predictions shown in Figure 2.
- ScanNet++ results: MiniLM slightly outperforms CLIP at 16.83 versus 16.17 f-mIoU under otherwise identical settings, isolating the text encoder effect.GoDeep also attains substantially higher f-mAcc than every ScanNet-only baseline, while trailing methods trained on larger multi-dataset corpora or directly on ScanNet++.
- Cultural heritage results: On the raw heritage vocabulary, CLIP outperforms MiniLM on all three metrics, but correcting floor to floor/grass dataset-wide reverses the mIoU ranking.A second correction, vault to vault/ceiling, raises MiniLM’s mIoU to 35.96%.
5 Ablation Studies
The ablations test vocabulary sensitivity, out-of-vocabulary recognition, VLM description properties, aggregation strategies, and point-level explainability. Together, they show stronger language-space responsiveness to precise vocabulary and useful gains from richer, traceable evidence aggregation.
- 5.1 Semantic Truth vs. Annotation Convention:: Correcting floor raises MiniLM’s column IoU by +29.00 while leaving vault/ceiling unchanged, whereas correcting vault raises arch by +5.56 while leaving column nearly unchanged.The floor correction reclaims 96.3% of 3.04M misclassified floor/grass points for MiniLM.
- 5.2 Out-of-Distribution Evaluation:: MiniLM separates present from absent OOV objects with a +0.143 margin versus +0.061 for CLIP, while localization mean IoU is 0.204 versus 0.189.MiniLM wins 72/125 class-threshold comparisons, but localization remains category-dependent.
- 5.3 VLM Description Richness:: Gemini descriptions are 4× longer, with 2.1× higher MTLD, 3.4× more unique adjectives, and 3.1× more unique nouns than Qwen2-VL-2B descriptions.The comparison uses the same 748 images from scene 1_SC.
- 5.4 VLM Description Consistency:: Qwen2-VL-2B produces 100% exact-match descriptions across four runs and mean pairwise Jaccard similarity of 1.000 on 360 images.The authors attribute this observed reproducibility to quantization narrowing the sampling distribution, but did not verify that mechanism directly.
6 Discussion
GoDeep decouples 3D scene understanding from CLIP by lifting VLM-generated descriptions into sentence-embedding space without 3D training. Its vocabulary sensitivity, OOV discrimination, and text traceability support more faithful and inspectable representations, but computational, memory, reproducibility, and evaluation-scope constraints remain.
- Core finding: GoDeep matches strong annotation-free ScanNet++ baselines without 3D training and uses a controlled internal comparison for the heritage benchmark.No comparable prior method reports results under the same heritage evaluation protocol.
- Semantic fidelity: Two independent vocabulary corrections show MiniLM responding more strongly than CLIP, while OOV evaluation shows sharper presence-versus-absence discrimination but category-dependent localization.The discussion links these findings to vocabulary precision rather than uniform superiority across localization categories.
- Explainability: Discrete text at every intermediate stage makes predictions traceable to specific evidence and motivates scale-aware multi-vector aggregation.The traceability connects point labels to descriptions, masks, weighted observations, and final embeddings.
- Limitations: GoDeep requires substantial computation and memory, including approximately 6.8 GB per 5M-point scene for mean-inliers and 24.3 GB for scale-aware multi-vector aggregation.The heritage results also rely on an internal comparison, and aggregation experiments use a single scene.
7 Conclusion
GoDeep lifts structured VLM descriptions into pure sentence-embedding space without a 3D training corpus or dedicated 3D encoder. It matches strong ScanNet-trained annotation-free baselines, while vocabulary corrections reverse the cultural-heritage ranking and support stronger physical-content fidelity, including for out-of-vocabulary objects.
- GoDeep requires no 3D training corpus or dedicated 3D encoder while lifting structured VLM descriptions into sentence-embedding space.
- On ScanNet++, GoDeep matches strong annotation-free baselines trained on ScanNet.
- Two independent vocabulary corrections on a five-building cultural-heritage benchmark reverse the raw ranking and show stronger physical-content fidelity for language-space embeddings than CLIP-based embeddings.
- Language-space fidelity extends to genuinely out-of-vocabulary objects and generalizes across five aggregation strategies.
- GoDeep predictions remain traceable to the natural-language evidence that produced them, supporting explanation, diagnosis, and pipeline improvement.