RelationVGGT: Visual Geometry Transformers for 3D Spatial Relation Segmentation

1 Introduction

• Open-vocabulary 3D understanding has moved from scene-optimized representations toward feed-forward multi-view semantics, but existing methods remain object-centric.Spatial relations are presented as important to holistic perception, spatial question answering, and embodied agents.Source: §1 · p. 1
• The task segments, across target views, the object satisfying a 3D spatial relation given a reference-view subject mask and relational text query.It requires semantic identity, directional relation reasoning, and cross-view geometric consistency.Source: §1 · pp. 1–2
• RelationField relies on scene-specific optimization and calibrated poses, while video-language models lack a 3D representation for maintaining correspondences across large viewpoint changes.These limitations motivate a feed-forward, pose-free multi-view formulation.Source: §1 · p. 2
• RelationVGGT fuses DINOv2 semantics with Pi3 geometry-aware features and uses a relation transformer for subject-conditioned, cross-view prediction.The framework reformulates volumetric relation search as per-pixel prediction and requires neither per-scene optimization nor known camera poses.Source: §1 · p. 2
• The paper provides a fully automated ScanNet++ annotation pipeline using VLMs and LLMs to generate scalable training data.Its contributions also include the task formulation and a feed-forward architecture combining 2D semantic and 3D geometry-aware features.Source: §1 · p. 2

3D Scene Understanding with Foundation Models

• Foundation-model scene understanding supports open-vocabulary segmentation, but remains object-centric rather than reasoning about inter-object relations.The paper builds on this line by combining DINOv2 semantics with geometry-aware representations.Source: §2 · p. 2

3D Spatial Relation and Scene Graph

• Scene graph methods predict object nodes and relation edges, while RelationField requires calibrated poses and scene-specific optimization for 3D relation search.RelationVGGT takes a relation-first approach, directly segmenting a target conditioned on a visually specified subject and relation query.Source: §2 · p. 2

Visual Grounding

• Visual grounding methods predict masks from language but do not explicitly model 3D structure, making sparse, wide-baseline multi-view relational grounding challenging.The cited methods are primarily designed for image or temporally continuous video observations.Source: §2 · p. 3
• Unlike 3D referring segmentation, this task gives a subject mask and relation text, leaving the target identity and category to be inferred.Referring expressions may identify targets through category, attributes, or relations.Source: §2 · p. 3

3.1 3D Spatial Relation Segmentation

• The task localizes, in each target view, the object satisfying a 3D spatial relation to a subject specified by a reference-view mask and relational query.The method replaces volumetric search with per-pixel relation prediction conditioned on the reference subject.Source: §3.1 · pp. 3–4

3.2 Architecture

• DINOv2 semantic features and Pi3 geometry-aware tokens are fused, then a relation transformer uses masked subject tokens and cross-view attention before decoding pixel-aligned relation features.The resulting relation features are matched with the language embedding of the query to localize the target.Source: §3.2 · pp. 4–5

3.3 Training

• Training supervises each target mask with multiple compatible relational queries using sigmoid focal loss, combined with Dice loss for spatial mask alignment.The multiple-query strategy addresses valid alternative expressions for the same spatial relation.Source: §3.3 · p. 5

3.4 Inference

• Inference matches per-pixel relation features with the text-query embedding, applies sigmoid, and thresholds scores at 0.3 without additional post-processing.The same threshold is used for all experiments.Source: §3.4 · p. 5

3.5 Dataset

• The ScanNet++ annotation pipeline collects VLM relation proposals, requires agreement across at least three viewpoints, filters geometrically implausible pairs, and uses an LLM to diversify validated relation wording.Retained relations are propagated to co-visible frames to complete the multi-view annotations.Source: §3.5 · p. 6

4.1 Quantitative results

• 0.382 average mIoU and 0.549 mAcc; RelationVGGT substantially outperforms RelationField across the three benchmark splits.The method uses relational reasoning without target-object queries at test time.Source: §4.1 · p. 7, Table 1 · p. 8
• On Replica- and ScanNet++-based splits, RelationVGGT consistently outperforms all baselines despite whole-scene camera trajectories that make open-vocabulary segmentation challenging.The trajectories broadly cover indoor scenes rather than focusing on individual objects.Source: §4.1 · p. 8
• On LERF’s cluttered tabletop scenes, RelationVGGT remains comparable to RelationField and exceeds RelationField†; unlike UniPixel, its performance stays stable across camera-baseline ranges.UniPixel is competitive at small baselines but generally degrades as viewpoint changes widen; the authors attribute RelationVGGT’s stability to 3D foundation-model features.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 Qualitative results

• RelationVGGT produces cleaner target masks than feature-field baselines and maintains view-consistent predictions across substantial camera-pose changes without explicit 3D representations.The authors attribute consistency to the 3D foundation-model backbone, which lets the relation transformer capture view-invariant relational semantics from multi-view input.Source: §4.2 · pp. 8–9, Figure 4 · p. 9

4.3 Ablations

• Raw pointmaps underperform learned features; removing DINOv2 hurts, and removing Pi3 hurts more because DINOv2 alone lacks multi-view awareness and geometric correspondence.The authors hypothesize that raw coordinates lack semantic cues embedded in learned features.Source: §4.3 · p. 9
• Cross-attention-only and alternative alternating-attention decoders perform worse, as does replacing multi-query with single-label focal loss for many-to-one relations.The multi-query objective lets multiple valid predicates supervise each target mask, avoiding penalties for semantically plausible alternatives.Source: §4.3 · p. 9
• Removing geometry-based edge filtering lowers performance because it admits physically implausible supervision, such as contact relations between distant objects.Source: §4.3 · p. 9
• SAM2 single-click or box masks preserve essentially ground-truth-mask performance; predicate paraphrases, including rare or unseen expressions, cause only limited degradation.Automatic subject grounding retains most performance with Qwen3-VL-8B+SAM2 but degrades more with GroundingDINO+SAM2; the authors attribute this mainly to instance ambiguity.Source: §4.3 · p. 10, Table 4 · p. 10, Table 5 · p. 10

5 Conclusion

• RelationVGGT integrates 3D geometry priors with 2D visual foundation models for subject-conditioned relational segmentation across views, without per-scene optimization or known camera poses.The task uses a visually specified subject and a spatial relation to localize the target using the underlying 3D scene structure.Source: §5 · p. 10
• A fully automated ScanNet++ data-generation pipeline uses VLMs and LLMs to produce scalable, multi-view-consistent annotations.Source: §5 · p. 10
• The pipeline depends on datasets with instance-level annotations, and extending it to unannotated or in-the-wild scenes remains future work.Source: §5 · p. 10
• The framework models object-object spatial relations; functional relations and joint scene-graph node-and-edge prediction are proposed extensions.The authors also identify more accurate and efficient VLM annotation strategies as future work.Source: §5 · p. 10

A.1 Relation distribution

• The final training predicate distribution is strongly long-tailed, with frequent relations coexisting alongside many low-frequency expressions.Source: §A.1 · p. 14
• The lexical diversity of open-vocabulary annotations motivates multi-query supervision instead of a fixed predicate classifier.Source: §A.1 · p. 14
• Figure 5 plots predicate occurrence counts on a logarithmic horizontal axis and groups 54 additional low-frequency predicates for visualization.Source: Figure 5 · p. 14

A.2 Human audit of annotation quality

• Two authors independently audited 4,695 directed proposals across 25 ScanNet++ scenes, excluding ambiguous judgments from quality-rate denominators.The reported 95% confidence intervals use a scene-level cluster bootstrap with 20,000 resamples.Source: §A.2 · p. 14

A.2.1 How many proposals survive filtering?

• Stage 2 removes 3,925 of 4,695 aggregated proposals, conservatively discarding relations supported by fewer than three frames.A false relation that survives becomes direct training supervision, whereas removing a correct relation primarily reduces coverage.Source: §A.2.1 · p. 14, Table 6 · p. 14

A.2.2 Human judgment versus filtering decision

• The two annotators marked 33 and 409 proposals as ambiguous, respectively, and those judgments are excluded from Table 7 and derived rates.Source: §A.2.2 · p. 15, Table 7 · p. 15
• The filter removes the large majority of proposals judged incorrect but also discards many correct proposals under its K = 3 multi-view agreement requirement.Relations visible or confidently recognized in only one or two frames are rejected.Source: §A.2.2 · p. 15

A.2.3 Annotation quality

• 36.4% to 67.2%: filtering increases proposal precision from Stage 1 to Stage 2.Source: §A.2.3 · p. 15
• 92.0% of proposals judged incorrect are removed, while the 28.5% correct-retention rate applies only within the Stage-1 proposal pool, not to all true scene relations.Source: §A.2.3 · p. 15
• Annotators agree on 83.1% of proposals (Cohen’s κ = 0.670; κ = 0.816 excluding ambiguous judgments), while the retained set remains noisy and sacrifices coverage for reliability.Source: §A.2.3 · p. 15

B.1 Training details

• Compatible phrasings share a target mask and receive independent binary supervision, with multi-query focal and Dice losses averaged over queries.Source: §B.1 · pp. 15–16

B.2 Inference efficiency

• 26 ms for RelationVGGT inference versus 288 s per query and more than 3 hours of scene optimization for RelationField on an NVIDIA L40S.RelationVGGT uses feed-forward inference on unseen scenes without scene-level optimization, while RelationField constructs a scene-specific representation.Source: Table 9 · p. 16, §B.2 · p. 16

B.3 Annotation prompts

• Visual extraction prompts seek directed, open-vocabulary relations among visible tagged instances; text-only refinement preserves semantic intent and relation family without inferring layout.The VLM is instructed to avoid unsupported or ambiguous relations, while the LLM receives no image or scene geometry.Source: §B.3 · p. 16, Figure 6 · p. 17, Figure 7 · p. 18

• The experiments compare RelationVGGT with closely related 3D referring-segmentation methods.Source: C Additional Experiments · p. 19
• A separate analysis tests robustness when the reference subject is absent from the target view.Source: C Additional Experiments · p. 19
• Predicate controls deliberately change the meaning of the relation query.Source: C Additional Experiments · p. 19