Source-linked AI summary
VGGT-Det: Mining VGGT Internal Priors for Sensor-Geometry-Free Multi-View Indoor 3D Object Detection
Yang Cao, Feize Wu, Dave Zhenyu Chen, Yingji Zhong, Lanqing Hong, Dan Xu
TL;DR
Existing indoor 3D detectors depend on costly sensor geometry, motivating SG-Free detection from images alone. VGGT-Det integrates VGGT into a transformer pipeline with AG and QD, and outperforms the best SG-Free method by 4.4 and 8.6 mAP@0.25 on ScanNet and ARKitScenes, respectively.
Problem
Current multi-view indoor 3D detectors rely on costly sensor-derived camera poses or depth, motivating detection without these geometric inputs.
Method
VGGT-Det integrates a pretrained VGGT encoder into a transformer pipeline and uses AG for attention-guided queries plus QD for query-driven aggregation of multi-level geometric features.
Results
4.4 and 8.6 mAP@0.25 points are the gains over the best-performing SG-Free method on ScanNet and ARKitScenes, respectively.
Takeaways & Limitations
Ablations confirm that AG and QD effectively leverage the semantic and geometric priors learned inside pretrained VGGT.
Takeaways & Limitations
VGGT introduces noticeable runtime and memory overhead, and current SG-Free pipelines use dataset scales to denormalize its normalized predictions.
Abstract
from arXiv · showhide
Current multi-view indoor 3D object detectors rely on sensor geometry that is costly to obtain (i.e., precisely calibrated multi-view camera poses) to fuse multi-view information into a global scene representation, limiting deployment in real-world scenes. We target a more practical setting: Sensor-Geometry-Free (SG-Free) multi-view indoor 3D object detection, where there are no sensor-provided geometric inputs (multi-view poses or depth). Recent Visual Geometry Grounded Transformer (VGGT) shows that strong 3D cues can be inferred directly from images. Building on this insight, we present VGGT-Det, the first framework tailored for SG-Free multi-view indoor 3D object detection. Rather than merely consuming VGGT predictions, our method integrates VGGT encoder into a transformer-based pipeline. To effectively leverage both the semantic and geometric priors from inside VGGT, we introduce two novel key components: (i) Attention-Guided Query Generation (AG): exploits VGGT attention maps as semantic priors to initialize object queries, improving localization by focusing on object regions while preserving global spatial structure; (ii) Query-Driven Feature Aggregation (QD): a learnable See-Query interacts with object queries to 'see' what they need, and then dynamically aggregates multi-level geometric features across VGGT layers that progressively lift 2D features into 3D. Experiments show that VGGT-Det significantly surpasses the best-performing method in the SG-Free setting by 4.4 and 8.6 mAP@0.25 on ScanNet and ARKitScenes, respectively. Ablation study shows that VGGT's internally learned semantic and geometric priors can be effectively leveraged by our AG and QD.
1. Introduction
VGGT-Det targets SG-Free multi-view indoor 3D object detection, removing reliance on sensor-derived camera poses or depth. It mines VGGT’s internal semantic and geometric priors through AG and QD, achieving substantial gains on two indoor benchmarks.
- Multi-view indoor 3D object detection supports applications in robotics and augmented reality.
- Existing methods rely on precisely calibrated camera poses or depth, which limits scalability and real-world deployment.
- SG-Free detection removes sensor-derived geometric inputs, using multi-view images without sensor-provided poses or depth.
- VGGT-Det integrates a pretrained VGGT encoder into a transformer pipeline instead of merely consuming VGGT predictions.
- Attention-Guided Query Generation uses VGGT attention as a semantic prior to focus object queries on object regions while preserving global spatial structure.
- Query-Driven Feature Aggregation uses a learnable See-Query to dynamically aggregate multi-level geometric features from VGGT layers.
- 4.4 and 8.6 mAP@0.25 points separate VGGT-Det from the best-performing SG-Free method on ScanNet and ARKitScenes, respectively.
2. Related Work
The related work frames VGGT-Det as mining internal VGGT priors rather than merely consuming predictions, alongside broader developments in multi-view detection and generalizable 3D reconstruction.
- Multi-view Outdoor 3D Object Detection: Outdoor multi-view 3D object detection includes methods that project 3D queries into 2D images to aggregate visual knowledge.
- Multi-view Outdoor 3D Object Detection: DETR3D uses sparse 3D object queries to index multi-view 2D features through camera parameters.
- Multi-view Outdoor 3D Object Detection: PETR methods encode 3D coordinate information into multi-view image features through position embedding transformation.
- Generalizable 3D Reconstruction: VGGT-Det mines internal semantic and geometric priors from VGGT rather than merely consuming its predictions.
- Generalizable 3D Reconstruction: Generalizable 3D reconstruction estimates scene geometry and camera poses from unposed multi-view images, replacing traditional multi-stage reconstruction pipelines.
3. Method
VGGT-Det integrates a VGGT encoder into an encoder-decoder transformer that detects 3D objects from multi-view images without sensor-provided poses or depth. AG initializes queries using semantic attention and spatially diverse point sampling, while QD dynamically aggregates geometric features across VGGT layers.
- Overview: VGGT-Det takes multi-view images as input and outputs 3D detection results without sensor-provided multi-view poses or depth.
- Basic Backbone: The backbone concatenates VGGT tokens from all views into a unified representation that object queries access through transformer self-attention and cross-attention.VGGT produces M tokens per view with token dimension C; concatenation yields (V · M) × C tokens.
- Attention-Guided Query Generation: AG addresses background-heavy VGGT point clouds by using encoder attention as semantic guidance while preserving spatial diversity during query-point sampling.Its priority combines normalized attention with normalized minimum distance, weighted by λ_dist.
- Attention-Guided Query Generation: AG selects the first query at the highest-attention point and subsequently favors points combining semantic importance with distance from previously selected samples.This encourages object-focused queries while maintaining coverage of the 3D space.
- Query-Driven Feature Aggregation: QD uses a learnable See-Query to dynamically aggregate multi-level VGGT feature maps, whose layers encode progressively different levels of geometric information.The See-Query produces softmax weights over L feature maps, which are combined as a weighted sum.
4. Experiments
Experiments evaluate VGGT-Det through ablations, loss dynamics, efficiency, frame-count sensitivity, qualitative comparisons, and benchmark results on ScanNet and ARKitScenes.
- Ablation Studies: AG improves the basic backbone by +2.8 points, while QD adaptively aggregates relevant geometric features through See-Query interactions.The ablation isolates the contribution of AG and describes QD’s feature-selection mechanism.
- Method Analysis: Validation losses support both designs: AG lowers GIoU loss versus the baseline, and AG+QD becomes lower than AG after a few epochs.The dynamics indicate that See-Query learns to interact with object queries and aggregate encoded geometric features.
- Efficiency Analysis: Efficiency analysis includes VGGT computation and reports 13.81 GB for the MVSDet variant, while VGGT-Det achieves a 5.6-point improvement with 40 input frames.All methods are evaluated with 40 frames on a 1×H800 GPU, including shared VGGT costs for adapted baselines.
- Frame Sensitivity: More input frames provide richer information, with performance saturating around 80 frames.The models are evaluated on ScanNet with varying numbers of input frames.
- Feature Aggregation: Multi-level feature aggregation improves performance: Vanilla-v2 exceeds Vanilla-v1, while QD outperforms Vanilla-v2 by 2.7 points.Vanilla-v2 uses encoded features from the 4th, 11th, 17th, and 23rd VGGT layers.
- Qualitative Comparison: Qualitatively, VGGT-Det detects more objects with higher precision than MVSDet.The qualitative comparison is performed in the SG-Free setting.
- Quantitative Comparison: VGGT-Det reaches 46.9 mAP@0.25 on ScanNet, exceeding MVSDet by +4.4 points and FCAF3D by 6.3 points.Competitive methods are retrained with VGGT-predicted geometry for the SG-Free comparison.
- Quantitative Comparison: On ARKitScenes, VGGT-Det surpasses MVSDet by 8.6 points in mAP@0.25.The comparison uses SG-Free adaptations of competitive methods.
5. Conclusion
The conclusion frames VGGT-Det as an SG-Free detector that mines VGGT’s internal semantic and geometric priors through AG and QD, outperforming strong methods on two datasets.
- 5. Conclusion: VGGT-Det removes dependence on sensor-derived geometric inputs by using a VGGT-based transformer backbone for SG-Free 3D detection.The framework targets multi-view indoor detection without sensor-provided poses or depth.
- 5. Conclusion: AG uses VGGT attention mechanisms to generate object queries focused on object regions while preserving global spatial structure.This component mines internal semantic priors rather than merely consuming VGGT predictions.
- 5. Conclusion: QD dynamically interacts with object queries to aggregate multi-level geometric features from VGGT.The conclusion identifies AG and QD as the two key innovations.
- 5. Conclusion: Experiments and ablations show that VGGT-Det outperforms strong SG-Free methods on ScanNet and ARKitScenes while leveraging VGGT’s learned priors.The reported evidence supports both benchmark performance and the effectiveness of AG and QD.
6. More Ablation Studies
Additional studies examine AG’s spatial-balance parameter, robustness to noisy VGGT point clouds, frame count, sensor-geometry gaps, efficiency, and query localization.
- Parameter Sensitivity: AG performs best around λdist = 0.8, which balances attention guidance and spatial dispersion.Too-small values cause excessive focus on specific high-attention regions.
- Noise Robustness: At noise level 0.01, VGGT-Det reaches 47.0 mAP@0.25 versus FCAF3D’s 18.7, a 28.3-point advantage.VGGT-Det maintains 34.1 mAP@0.25 even at noise level 0.3, while degradation increases with noise for both methods.
- Noise Robustness: AG improves robustness by using semantic priors to focus queries on relevant regions, reducing reliance on precise VGGT point-cloud geometry.FCAF3D depends more heavily on geometric integrity and degrades more rapidly under noise.
- Input Frames: Performance rises with more input frames and saturates around 80 frames, indicating that this count captures most needed scene information.The frame-count study is conducted on ScanNet.
- Sensor Geometry: VGGT-Det achieves the highest performance in both SG-Free and SG-Based settings, although sensor geometry still provides strong detection cues.A noticeable performance gap remains between the two settings.
- Efficiency: Time and GPU-memory costs increase with the number of input views, a common challenge in multi-view 3D detection.The efficiency analysis studies varying input counts.
- Query Localization: Attention maps highlight object regions, and AG places more generated queries inside those regions than the baseline, improving query distribution.The visualization connects this behavior to a 2.8-point mAP@0.25 gain.
7. Comparison with Alternatives
VGGT-Det is qualitatively compared with MVSDet in the SG-Free setting, where it detects more objects with higher accuracy.
- Qualitative Comparison: VGGT-Det detects more objects with higher accuracy than MVSDet in qualitative comparisons.MVSDet is trained and tested using VGGT-predicted multi-view poses for a fair SG-Free comparison.
8. Training and Testing Time
The ablation experiments use eight H800 GPUs, with ScanNet training taking approximately two days and testing taking about one minute.
- Approximately 2 days are required to train the model on ScanNet using eight H800 GPUs.
- About 1 minute is required to process the entire ScanNet testing set.
9. Limitation and Future work
VGGT-Det and current SG-Free pipelines face runtime, memory, and scale-related limitations. A lighter VGGT-like model with metric-scale predictions is identified as a direction for future work.
- Limitation: Current SG-Free pipelines incur noticeable runtime and memory overhead from VGGT.
- Limitation: VGGT’s normalized predictions require dataset scales to denormalize predictions across current SG-Free pipelines.
- Future work: A lighter VGGT-like model with metric-scale predictions could further advance SG-Free indoor 3D detection.