VGGT-Det: Mining VGGT Internal Priors for Sensor-Geometry-Free Multi-View Indoor 3D Object Detection

1. Introduction

• SG-Free detection removes sensor-derived camera poses and depth, addressing the cost and limited availability that constrain current multi-view indoor detectors.The introduction identifies multi-view indoor 3D object detection as a task with applications in robotics and augmented reality.Source: §1 · p. 1
• Feed-forward reconstruction models can infer 3D structure from unposed images and encode learned priors that support 3D reasoning.This motivates using VGGT [41] beyond consuming its predicted geometry.Source: §1 · pp. 1–2
• Attention-Guided Query Generation uses semantic information in VGGT encoder attention to initialize and steer object queries toward object regions while preserving global spatial structure.The introduction notes that VGGT attention maps capture rich semantic information despite no explicit semantic training.Source: §1 · p. 2
• Query-Driven Feature Aggregation uses a learnable See-Query to interact with object queries and dynamically aggregate geometric features across VGGT layers.Because VGGT progressively lifts 2D features into 3D, its layers encode distinct geometric abstraction levels that QD combines hierarchically.Source: §1 · p. 2
• VGGT-Det outperforms the best-performing SG-Free method by 4.4 mAP@0.25 points on ScanNet and 8.6 on ARKitScenes.The introduction reports these gains for the proposed approach.Source: §1 · p. 2

Multi-view Outdoor 3D Object Detection

• Outdoor multi-view detectors either project 3D queries into image features or lift 2D features into BEV, whereas indoor camera mobility makes reliable poses costly and motivates SG-Free detection.The projection line includes DETR3D [44] and PETR [24] [25]; the lifting line includes LSS [31], with BEVDepth [17] and BEVStereo addressing depth through explicit supervision.Source: §2 · p. 2

Multi-view Indoor 3D Object Detection

• Existing multi-view indoor detectors leverage pose-dependent geometric clues from voxels or splats, but their reliance on sensor-derived poses or depth limits real-world applicability.Examples include ImVoxelNet [34], NeRF-Det [46], and MVSDet [47]; the paper addresses this limitation with SG-Free indoor detection.Source: §2 · p. 2

Generalizable 3D Reconstruction

• For unposed multi-view reconstruction, DUSt3R [43] and MASt3R [15] use direct prediction but remain pair-wise, while Fast3R [50] handles long sequences in one pass without coordinate alignment.VGGT [41] predicts camera poses and other 3D-relevant attributes, and VGGT-Det leverages its internal priors rather than merely consuming its predictions.Source: §2 · p. 3
• VGGT-Det mines VGGT’s internal semantic and geometric priors through Attention-Guided Query Generation and Query-Driven Feature Aggregation.AG uses encoder attention to initialize object-focused queries while preserving global spatial structure; QD uses a learnable See-Query to guide dynamic aggregation of multi-level geometric features.Source: §2 · p. 3, Figure 2 · p. 3

3.1. Overview

• VGGT-Det takes multi-view images as input and outputs 3D detections without sensor-provided camera poses or depth.Source: §3.1 · p. 3

3.2. Basic Backbone

• The transformer backbone extracts 3D-aware VGGT features, initializes object queries by farthest point sampling, and decodes them against concatenated multi-view tokens to predict classes and 3D boxes.Source: §3.2 · pp. 3–4

3.3. Attention-Guided Query Generation

• Attention-Guided Query Generation selects points using VGGT attention and spatial dispersion, steering queries toward semantic object regions while retaining spatial coverage.It addresses background queries caused by uniformly applying farthest point sampling to dense VGGT reconstructions.Source: §3.3 · pp. 4–5

3.4. Query-Driven Feature Aggregation

• Query-Driven Feature Aggregation uses a shared learnable See-Query to weight VGGT’s multi-level features, interact with object queries, and iteratively guide context-aware aggregation across decoder layers.The weighted feature sum supplies keys and values for cross-attention with the unified See-Query and object-query set.Source: §3.4 · pp. 5–6

4.1. Experimental Setup

• Evaluation uses mAP@0.25 on ScanNet and ARKitScenes [1]; training freezes the pretrained VGGT encoder and uses 256 object queries.ScanNet has 18 standard categories and ARKitScenes has 17; training uses AdamW, cosine annealing, and the 3DETR loss setting.Source: §4.1 · p. 6

4.2. Method Analysis

• Attention-Guided Query Generation improves the basic backbone by +2.8 points, using VGGT attention to focus queries on object regions while preserving scene-wide spatial structure.The gain is reported against the basic backbone with FPS Sampling Generation.Source: §4.2 · p. 6, Table 2 · p. 7
• Query-Driven Feature Aggregation outperforms Vanilla-v2 by 2.7 points, using a See-Query to interact adaptively with object queries across decoder layers and select and aggregate relevant encoded geometric features.Source: §4.2 · p. 7
• With 40 input frames, VGGT-Det uses 3.57 GB versus MVSDet’s 13.81 GB, with comparable time consumption and a 5.6-point performance improvement.The comparison was conducted on ScanNet using one H800 GPU; timing includes shared VGGT computation for both methods.Source: Table 2 · p. 7, §4.2 · p. 7
• On ScanNet, performance increases with more input frames and saturates at around 80, which the study identifies as sufficient to capture most scene information.The frame-count evaluation was performed at inference time.Source: Table 2 · p. 7, §4.2 · p. 7
• Validation losses support both designs: AG lowers GIoU loss versus the baseline, while AG+QD loss becomes significantly lower than AG after See-Query learns to aggregate features.The AG and AG+QD losses are initially similar; the latter separates after a few epochs.Source: Figure 5 · p. 7, §4.2 · p. 7

4.3. Comparison with Alternatives

• Qualitative comparisons show VGGT-Det detects more objects with higher precision than MVSDet [47], attributed to effective use of VGGT priors through AG and QD.The comparison is shown on validation examples.Source: Figure 6 · p. 8, §4.3 · p. 8
• VGGT-Det reaches 46.9 mAP@0.25 on ScanNet, 4.4 higher than MVSDet [47] and 6.3 points above FCAF3D [33]; on ARKitScenes, it surpasses MVSDet [47] by 8.6 points.For fair SG-Free comparisons, competing methods were retrained using VGGT-predicted poses, or predicted point clouds for FCAF3D.Source: Table 3 · p. 8, §4.3 · p. 8

Conclusion

• The framework removes dependence on sensor-derived geometric inputs and uses the VGGT encoder in a transformer-based backbone.It introduces Attention-Guided Query Generation (AG) and Query-Driven Feature Aggregation (QD).Source: §5 · p. 8
• AG uses VGGT attention mechanisms’ semantic priors to focus object queries on object regions while maintaining global spatial structure.This supports query generation that draws on VGGT’s internal semantic priors.Source: §5 · pp. 8–9
• VGGT-Det significantly outperforms strong methods in the SG-Free setting on both ScanNet and ARKitScenes.Ablations confirm that AG and QD effectively leverage learned priors from pretrained VGGT.Source: §5 · p. 9
• QD dynamically interacts with object queries to adaptively aggregate multi-level geometric features.Ablations confirm that QD effectively leverages learned priors from pretrained VGGT.Source: §5 · p. 9

6. More Ablation Studies

• AG performs best around λdist = 0.8, which balances attention guidance against spatial dispersion.Too little λdist over-focuses queries on high-attention regions, while too much reduces the influence of attention guidance.Source: §6 · p. 11, Table 4 · p. 11
• At noise level 0.01, VGGT-Det scores 47.0 mAP@0.25 versus FCAF3D’s 18.7, a 28.3-point advantage, and retains 34.1 mAP@0.25 at noise level 0.3.Both methods degrade as Gaussian noise is added to VGGT-predicted point clouds, but FCAF3D begins degrading significantly at noise level 0.001 and reaches 0.0 mAP@0.25 at 0.1; VGGT-Det starts degrading at 0.1. The evaluation uses ScanNet and gives both methods noisy point-cloud inputs.Source: Table 5 · p. 11, §6 · p. 11
• Across different input-frame counts, VGGT-Det consistently surpasses the strongest competing method, MVSDet.The comparison follows the 80-frame practice used in the official MVSDet implementation, while also evaluating other frame counts.Source: Table 6 · p. 11, §6 · p. 11
• VGGT-Det achieves the highest performance in both the SG-Free and SG-Based settings, while a gap remains between the settings.The passage attributes the gap to the strong detection cues provided by sensor geometry and describes SG-Free as practical where sensor geometry is unavailable or impractical.Source: §6 · pp. 11–12, Table 7 · p. 12
• Under the SG-Free comparison, VGGT-Det has inference time comparable to +MVSDet and the lowest GPU memory usage among the methods.For fairness, SG-Free variants are trained and evaluated with multi-view RGB images processed by VGGT to estimate camera poses, so the comparison includes VGGT’s runtime and memory costs.Source: §6 · p. 12, Table 8 · p. 12
• Time and GPU memory costs increase as the number of input views grows.Source: §6 · p. 12, Table 9 · p. 12
• AG places more generated queries within object regions than unguided query generation and improves mAP@0.25 by 2.8 points.The attention maps highlight object regions; the passage attributes AG’s query distribution and performance improvement to this semantic guidance.Source: §6 · p. 12

Qualitative Comparison

• 4.4 points of performance improvement over MVSDet [47] are reported alongside qualitative detections of more objects with higher accuracy.The passage says the qualitative comparison is consistent with the improvement reported in Tab. 1 of the main paper.Source: §7 · p. 12
• For a fair SG-Free comparison, MVSDet [47] is trained and tested using multi-view poses predicted by VGGT [41].Source: §7 · p. 12
• The passage attributes VGGT-Det’s higher performance to its use of internal VGGT priors through Attention-Guided Query Generation (AG) and Query-Driven Feature Aggregation (QD).Source: §7 · p. 12

• Training on ScanNet takes approximately 2 days, and processing its entire testing set takes about 1 minute.Source: §8 · p. 12
• All ablation experiments are conducted on eight H800 GPUs.Source: §8 · p. 12

Limitation and Future work

• Current SG-free pipelines incur VGGT runtime and memory overhead, rely on dataset scales to denormalize its predictions, and could benefit from a lighter model with metric-scale predictions.The dataset-scale denormalization requirement applies across all current SG-free pipelines.Source: §9 · p. 12