Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

• Fixed-taxonomy 3D detectors require costly new 3D-box annotations to recognize object types outside their training vocabulary.Source: §1 · p. 2
• Multi-view pipelines often construct class-agnostic instances and merge fragments mainly by geometric consistency, despite incomplete and view-dependent geometry.Source: §1 · pp. 2–3
• Geometry-only merging can irreversibly fuse fragments from different categories before later semantic reasoning can disentangle them.Source: §1 · p. 3
• Group3D enforces semantic compatibility alongside geometric consistency during instance construction to mitigate over-merging under geometric ambiguity.Source: §1 · p. 3
• Group3D builds a scene vocabulary and category-aware 3D fragment memory using MLLM queries, multi-view masks, and geometric lifting.Source: §1 · p. 3
• Group3D reports strong open-vocabulary and zero-shot 3D detection performance using only multi-view RGB inputs, without ground-truth depth or 3D supervision.Source: §1 · p. 4

• Point-cloud 3D detectors initially extended 2D paradigms, but sparsity caused computational waste and bottlenecks in detection speed and accuracy.Source: §2.1 · p. 4
• Multi-view image-based detectors construct 3D representations from multiple RGB observations using approaches including BEV projections and DETR-based frameworks.Source: Multi-view image-based detection · p. 4
• Open-vocabulary point-cloud methods use CLIP-style semantic transfer or pseudo supervision from 2D priors and cross-modal alignment, typically with target-domain training.Source: Point cloud-based detection · p. 4
• Open-vocabulary multi-view methods lift 2D predictions into 3D, aggregate object hypotheses, and assign semantics through pseudo boxes or vision-language similarity.Source: Point cloud-based detection · p. 5

Scene Memory Construction

• The method builds a Scene Vocabulary Memory from normalized MLLM category predictions aggregated across views.The resulting scene-level vocabulary is used to induce semantic compatibility groups.Source: §3.1 · p. 5
• Category-aware 2D masks are lifted into a shared 3D coordinate system to form fragments storing point clouds, category hypotheses, and confidence scores.Depth maps and camera poses provide the back-projection geometry, with reliability filtering suppressing extreme depth outliers.Source: 3D Fragment Memory · pp. 6–7

Semantic Compatibility Grouping

• MLLM-generated compatibility groups capture plausible cross-view category variability while excluding structural attachments, supporting structures, and part–whole relationships.Examples of compatible categories include chair–sofa and desk–table, whereas wall–window and table–cup are excluded.Source: §3.2 · p. 7

Group-Gated 3D Fragment Merging

• Fragments merge only when their categories share a semantic group and their voxel representations satisfy geometric overlap or containment criteria.Overlap uses voxel IoU or a containment ratio, and merged clusters receive final labels from confidence-weighted multi-view support before bounding-box construction.Source: §3.3 · pp. 7–8, §3.4 · p. 9

Experimental setup

• Evaluation uses ScanNetV2 and ARKitScenes validation splits, reporting mAP at 3D IoU thresholds of 0.25 and 0.50.All evaluated results use zero-shot settings without category-specific 3D supervision.Source: §4.1 · p. 9, Zero-shot setting · p. 10
• Group3D is compared with point cloud-based and multi-view image-based open-vocabulary detectors in pose-known and pose-free regimes.The comparisons focus exclusively on multi-view RGB input while including stronger point-cloud baselines.Source: Table 1 · p. 10, §4.3 · pp. 11–12

Results and comparisons

• Group3D establishes a clear multi-view state of the art on ScanNet20 and remains competitive on ScanNet60 and ScanNet200 as the vocabulary becomes larger and more fine-grained.The method also preserves strong performance in pose-free settings, where reconstruction noise makes geometry-only association brittle.Source: §4.3 · p. 12
• Group3D improves across a broad set of ScanNet20 categories and generalizes to ARKitScenes with distinct capture conditions and scene statistics.These results suggest that the semantic compatibility prior transfers across domains.Source: §4.3 · p. 12

Ablation study

• K = 5 and K = 10 category hypotheses per view yield comparable performance, motivating K = 5 for improved efficiency without sacrificing accuracy.The experiments adopt K = 5 in the reported pipeline.Source: §4.4 · p. 13, Table 6 · p. 13
• Replacing the MLLM with a smaller 8B-scale model moderately reduces performance, while alternative reconstruction and segmentation models preserve the overall pipeline trend.Depth Anything 3 maintains strong results in a strictly zero-shot setting, whereas VGGT is trained on ScanNet.Source: Table 4 · p. 12, §4.4 · p. 13
• Removing category information causes geometry-driven over-merging, while strict same-category merging remains sensitive to cross-view label variability.Semantic compatibility grouping achieves the best performance by permitting consistent labels to merge while preventing incompatible associations.Source: §4.4 · pp. 13–14, Table 5 · p. 13

Conclusion

• Group3D incorporates semantic constraints directly into instance construction through scene-adaptive category groups and merge-time semantic gating.The framework targets geometry-driven over-merging and cross-view category variability.Source: §5 · p. 14
• The findings suggest that language-driven semantic priors can complement geometric reasoning in multi-view 3D perception.The proposed direction uses multi-view RGB inputs without relying on dense 3D supervision or explicit geometry sensors.Source: §5 · p. 14

Scene Vocabulary Memory

• The MLLM aggregates per-view object category hypotheses into a scene vocabulary used throughout the pipeline.The query focuses on prominent objects, uses simple singular nouns, avoids modifiers, and returns a comma-separated list.Source: §A.1 · p. 19

Semantic Compatibility Grouping

• Semantic compatibility groups collect categories that could refer to the same physical object across views despite taxonomy variation.The grouping instruction excludes co-occurrence, structural, opening, and part–whole relationships.Source: §A.2 · pp. 19–20
• Group membership is used as a semantic merge prior for voxel-based 3D fragment merging.Categories may appear in at most one multi-category group; unmentioned categories become singleton groups.Source: §A.2 · p. 20

Voxelization and Geometric Overlap

• Voxelization maps each fragment point cloud into discrete voxel indices by flooring each 3D point coordinate divided by voxel size.For a fragment F, vox(F) consists of the voxel indices obtained from points p in F using voxel size s.Source: §A.3 · p. 20
• Experiments use a 5 cm voxel size, with geometric overlap evaluated using thresholds τiou = 0.01 and τcont = 0.10.The overlap predicate is defined in the main paper’s Eq. (6).Source: §A.3 · p. 20

Experimental Setting

• Ablation studies use the pose-free setting because it represents the more challenging scenario.They analyze voxel resolution for fragment merging and the number of input frames.Source: §B.1 · p. 21

Voxel Size

• 5 cm voxelization is adopted as a practical accuracy–efficiency trade-off, while 10 cm significantly degrades fragment association.The difference between 1 cm and 5 cm is marginal, whereas smaller voxels provide slightly more precise alignment.Source: §B.1 · p. 21

Input Frames

• Increasing input frames improves scene coverage and fragment completeness, while reducing frames gradually degrades detection performance.The final configuration uses 128 frames.Source: §B.1 · p. 21

Qualitative Results

• Qualitative examples span ScanNet20, ScanNet200, and ARKitScenes with diverse vocabulary settings.The examples include both pose-known and pose-free settings.Source: §B.2 · p. 21

Semantic Compatibility Grouping

• MLLM-generated groups combine lexical variants and semantically related categories that may denote the same physical object across views.These groupings provide semantic priors for subsequent fragment merging.Source: §B.3 · p. 21, Table 9 · p. 23

Qualitative Results

• Qualitative grouping results are reported on ScanNet20, ScanNet200, and ARKitScenes under pose-known and pose-free settings.The figures provide examples across the three datasets.Source: Figure 5 · p. 22, Figure 6 · p. 22, Figure 7 · p. 23

• Group3D's fragment merging process produces instance-level 3D point sets that enable additional segmentation evaluation.Source: §B.4 · p. 24
• Unlike conventional methods, Group3D predicts instances on a newly reconstructed 3D point set derived from multi-view RGB observations.This supports both pose-known and pose-free operation using RGB inputs alone.Source: §B.4 · p. 24
• Predicted instance labels are transferred to ground-truth mesh vertices using nearest-neighbor association, a 5 cm distance threshold, and bounding-box containment.Vertices failing either condition remain unassigned.Source: §B.4 · p. 24
• Group3D produces consistent instance-level segmentations despite operating on reconstructed geometry rather than the original scene point cloud.The results are evaluated with the standard ScanNet protocol using AP25 and AP50.Source: §B.4 · p. 24
• Segmentation performance drops in the pose-free setting because reconstruction noise affects the reconstructed geometry.Source: §B.4 · p. 24