Source-linked AI summary
Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection
Youbin Kim, Jinho Park, Hogun Park, Eunbyung Park
TL;DR
Multi-view open-vocabulary 3D detection can suffer irreversible association errors when geometry-only merging uses incomplete, view-dependent evidence. Group3D adds MLLM-derived semantic compatibility groups as merge-time constraints alongside geometry, achieving strong state-of-the-art performance and zero-shot generalization with RGB inputs.
Problem
Geometry-dominated instance construction can over-merge distinct objects or fragment one instance before semantic labeling, because multi-view evidence is incomplete and view-dependent.
Method
Group3D builds a scene-adaptive MLLM vocabulary, organizes categories into compatibility groups, and merges 3D fragments only when semantic and geometric constraints agree.
Results
Group3D achieves state-of-the-art multi-view open-vocabulary 3D detection performance and strong zero-shot generalization across ScanNet and ARKitScenes.
Takeaways & Limitations
Semantic compatibility in fragment merging supports more reliable open-vocabulary 3D instance construction from multi-view RGB inputs under incomplete geometric evidence.
Takeaways & Limitations
The framework uses multi-view RGB observations and does not use category-specific 3D supervision from evaluated benchmarks in zero-shot results.
Abstract
from arXiv · showhide
Open-vocabulary 3D object detection aims to localize and recognize objects beyond a fixed training taxonomy. In multi-view RGB settings, recent approaches often decouple geometry-based instance construction from semantic labeling, generating class-agnostic fragments and assigning open-vocabulary categories post hoc. While flexible, such decoupling leaves instance construction governed primarily by geometric consistency, without semantic constraints during merging. When geometric evidence is view-dependent and incomplete, this geometry-only merging can lead to irreversible association errors, including over-merging of distinct objects or fragmentation of a single instance. We propose Group3D, a multi-view open-vocabulary 3D detection framework that integrates semantic constraints directly into the instance construction process. Group3D maintains a scene-adaptive vocabulary derived from a multimodal large language model (MLLM) and organizes it into semantic compatibility groups that encode plausible cross-view category equivalence. These groups act as merge-time constraints: 3D fragments are associated only when they satisfy both semantic compatibility and geometric consistency. This semantically gated merging mitigates geometry-driven over-merging while absorbing multi-view category variability. Group3D supports both pose-known and pose-free settings, relying only on RGB observations. Experiments on ScanNet and ARKitScenes demonstrate that Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection, while exhibiting strong generalization in zero-shot scenarios. The project page is available at https://ubin108.github.io/Group3D/.
1 Introduction
Group3D addresses geometry-driven errors in multi-view open-vocabulary 3D instance construction by enforcing semantic compatibility during fragment merging. It uses MLLM-derived scene memories and supports pose-known and pose-free RGB-only detection.
- Fixed-taxonomy 3D detectors require costly new 3D-box annotations to recognize object types outside their training vocabulary.
- Multi-view pipelines often construct class-agnostic instances and merge fragments mainly by geometric consistency, despite incomplete and view-dependent geometry.
- Geometry-only merging can irreversibly fuse fragments from different categories before later semantic reasoning can disentangle them.
- Group3D enforces semantic compatibility alongside geometric consistency during instance construction to mitigate over-merging under geometric ambiguity.
- Group3D builds a scene vocabulary and category-aware 3D fragment memory using MLLM queries, multi-view masks, and geometric lifting.
- Group3D reports strong open-vocabulary and zero-shot 3D detection performance using only multi-view RGB inputs, without ground-truth depth or 3D supervision.
2 Related Works
Related work spans point-cloud and multi-view image-based 3D detection, including open-vocabulary methods that transfer semantics through language alignment or aggregate lifted image predictions.
- Point-cloud 3D detectors initially extended 2D paradigms, but sparsity caused computational waste and bottlenecks in detection speed and accuracy.
- Multi-view image-based detectors construct 3D representations from multiple RGB observations using approaches including BEV projections and DETR-based frameworks.
- Open-vocabulary point-cloud methods use CLIP-style semantic transfer or pseudo supervision from 2D priors and cross-modal alignment, typically with target-domain training.
- Open-vocabulary multi-view methods lift 2D predictions into 3D, aggregate object hypotheses, and assign semantics through pseudo boxes or vision-language similarity.
3 Group3D
Group3D constructs open-vocabulary 3D instances from multi-view RGB by combining scene-adaptive semantic information with geometric fragment consistency. Its MLLM-derived compatibility groups constrain merging, while accumulated evidence determines each instance’s category and 3D bounding box.
- Scene Memory Construction: The method builds a Scene Vocabulary Memory from normalized MLLM category predictions aggregated across views.The resulting scene-level vocabulary is used to induce semantic compatibility groups.
- Scene Memory Construction: Category-aware 2D masks are lifted into a shared 3D coordinate system to form fragments storing point clouds, category hypotheses, and confidence scores.Depth maps and camera poses provide the back-projection geometry, with reliability filtering suppressing extreme depth outliers.
- Semantic Compatibility Grouping: MLLM-generated compatibility groups capture plausible cross-view category variability while excluding structural attachments, supporting structures, and part–whole relationships.Examples of compatible categories include chair–sofa and desk–table, whereas wall–window and table–cup are excluded.
- Group-Gated 3D Fragment Merging: Fragments merge only when their categories share a semantic group and their voxel representations satisfy geometric overlap or containment criteria.Overlap uses voxel IoU or a containment ratio, and merged clusters receive final labels from confidence-weighted multi-view support before bounding-box construction.
4 Experiments
Experiments evaluate Group3D across ScanNet and ARKitScenes in pose-known, pose-free, and zero-shot settings. Results show strong performance across vocabulary scales, datasets, and semantic-grouping ablations.
- Experimental setup: Evaluation uses ScanNetV2 and ARKitScenes validation splits, reporting mAP at 3D IoU thresholds of 0.25 and 0.50.All evaluated results use zero-shot settings without category-specific 3D supervision.
- Experimental setup: Group3D is compared with point cloud-based and multi-view image-based open-vocabulary detectors in pose-known and pose-free regimes.The comparisons focus exclusively on multi-view RGB input while including stronger point-cloud baselines.
- Results and comparisons: Group3D establishes a clear multi-view state of the art on ScanNet20 and remains competitive on ScanNet60 and ScanNet200 as the vocabulary becomes larger and more fine-grained.The method also preserves strong performance in pose-free settings, where reconstruction noise makes geometry-only association brittle.
- Results and comparisons: Group3D improves across a broad set of ScanNet20 categories and generalizes to ARKitScenes with distinct capture conditions and scene statistics.These results suggest that the semantic compatibility prior transfers across domains.
- Ablation study: K = 5 and K = 10 category hypotheses per view yield comparable performance, motivating K = 5 for improved efficiency without sacrificing accuracy.The experiments adopt K = 5 in the reported pipeline.
- Ablation study: Replacing the MLLM with a smaller 8B-scale model moderately reduces performance, while alternative reconstruction and segmentation models preserve the overall pipeline trend.Depth Anything 3 maintains strong results in a strictly zero-shot setting, whereas VGGT is trained on ScanNet.
- Ablation study: Removing category information causes geometry-driven over-merging, while strict same-category merging remains sensitive to cross-view label variability.Semantic compatibility grouping achieves the best performance by permitting consistent labels to merge while preventing incompatible associations.
5 Conclusion
The conclusion presents Group3D as a framework that integrates semantic constraints into multi-view 3D instance construction. Its results suggest that semantic compatibility improves instance reliability under incomplete RGB evidence while supporting broader open-vocabulary perception.
- Conclusion: Group3D incorporates semantic constraints directly into instance construction through scene-adaptive category groups and merge-time semantic gating.The framework targets geometry-driven over-merging and cross-view category variability.
- Conclusion: The findings suggest that language-driven semantic priors can complement geometric reasoning in multi-view 3D perception.The proposed direction uses multi-view RGB inputs without relying on dense 3D supervision or explicit geometry sensors.
A Additional Implementation Details
The implementation constructs scene-level semantic priors with MLLM-generated compatibility groups, then combines them with voxel-based geometric consistency to merge 3D fragments.
- Scene Vocabulary Memory: The MLLM aggregates per-view object category hypotheses into a scene vocabulary used throughout the pipeline.The query focuses on prominent objects, uses simple singular nouns, avoids modifiers, and returns a comma-separated list.
- Semantic Compatibility Grouping: Semantic compatibility groups collect categories that could refer to the same physical object across views despite taxonomy variation.The grouping instruction excludes co-occurrence, structural, opening, and part–whole relationships.
- Semantic Compatibility Grouping: Group membership is used as a semantic merge prior for voxel-based 3D fragment merging.Categories may appear in at most one multi-category group; unmentioned categories become singleton groups.
- Voxelization and Geometric Overlap: Voxelization maps each fragment point cloud into discrete voxel indices by flooring each 3D point coordinate divided by voxel size.For a fragment F, vox(F) consists of the voxel indices obtained from points p in F using voxel size s.
- Voxelization and Geometric Overlap: Experiments use a 5 cm voxel size, with geometric overlap evaluated using thresholds τiou = 0.01 and τcont = 0.10.The overlap predicate is defined in the main paper’s Eq. (6).
B.1 More Ablation Studies
The ablations examine voxel resolution and input-frame count in the challenging pose-free setting, finding practical trade-offs between accuracy, completeness, and efficiency.
- Experimental Setting: Ablation studies use the pose-free setting because it represents the more challenging scenario.They analyze voxel resolution for fragment merging and the number of input frames.
- Voxel Size: 5 cm voxelization is adopted as a practical accuracy–efficiency trade-off, while 10 cm significantly degrades fragment association.The difference between 1 cm and 5 cm is marginal, whereas smaller voxels provide slightly more precise alignment.
- Input Frames: Increasing input frames improves scene coverage and fragment completeness, while reducing frames gradually degrades detection performance.The final configuration uses 128 frames.
B.2 More Qualitative Results
Additional qualitative results illustrate Group3D across multiple datasets and vocabulary settings.
- Qualitative Results: Qualitative examples span ScanNet20, ScanNet200, and ARKitScenes with diverse vocabulary settings.The examples include both pose-known and pose-free settings.
B.3 Qualitative Analysis of Semantic Compatibility Grouping
The semantic grouping analysis presents MLLM-generated compatibility examples and qualitative results across datasets and camera-pose settings.
- Semantic Compatibility Grouping: MLLM-generated groups combine lexical variants and semantically related categories that may denote the same physical object across views.These groupings provide semantic priors for subsequent fragment merging.
- Qualitative Results: Qualitative grouping results are reported on ScanNet20, ScanNet200, and ARKitScenes under pose-known and pose-free settings.The figures provide examples across the three datasets.
B.4 Open-Vocabulary 3D Instance Segmentation
Group3D also supports instance-level 3D segmentation by constructing instances from reconstructed multi-view RGB geometry. Evaluation transfers predicted labels to ScanNet mesh vertices, showing consistent segmentation, with lower performance in the pose-free setting due to reconstruction noise.
- Group3D's fragment merging process produces instance-level 3D point sets that enable additional segmentation evaluation.
- Unlike conventional methods, Group3D predicts instances on a newly reconstructed 3D point set derived from multi-view RGB observations.This supports both pose-known and pose-free operation using RGB inputs alone.
- Predicted instance labels are transferred to ground-truth mesh vertices using nearest-neighbor association, a 5 cm distance threshold, and bounding-box containment.Vertices failing either condition remain unassigned.
- Group3D produces consistent instance-level segmentations despite operating on reconstructed geometry rather than the original scene point cloud.The results are evaluated with the standard ScanNet protocol using AP25 and AP50.
- Segmentation performance drops in the pose-free setting because reconstruction noise affects the reconstructed geometry.