Source-linked AI summary
Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, Ivan Laptev
TL;DR
3D object grounding requires localizing objects referred to by natural language while distinguishing spatially related instances, despite scarce training data. ViL3DRel addresses this with language-conditioned spatial self-attention and teacher-student training, significantly outperforming state-of-the-art methods across Nr3D, Sr3D, and ScanRefer, including a 9.3% gain on Nr3D.
Problem
3D object grounding must localize language-referred objects and distinguish similar instances using spatial relations, while available training data remains scarce.
Method
ViL3DRel combines spatial self-attention encoding pairwise object distances and orientations with teacher-student training that transfers relation knowledge from ground-truth labels to point-cloud inputs.
Results
ViL3DRel significantly outperforms state-of-the-art methods on Nr3D, Sr3D, and ScanRefer, with a 9.3% absolute gain on Nr3D.
Takeaways & Limitations
The approach supports language-conditioned spatial relation reasoning for 3D object grounding and may interest multimodal transformer designs that incorporate priors into self-attention.
Abstract
from arXiv · showhide
Localizing objects in 3D scenes based on natural language requires understanding and reasoning about spatial relations. In particular, it is often crucial to distinguish similar objects referred by the text, such as "the left most chair" and "a chair next to the window". In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations. To this end, we design a spatial self-attention layer that accounts for relative distances and orientations between objects in input 3D point clouds. Training such a layer with visual and language inputs enables to disambiguate spatial relations and to localize objects referred by the text. To facilitate the cross-modal learning of relations, we further propose a teacher-student approach where the teacher model is first trained using ground-truth object labels, and then helps to train a student model using point cloud inputs. We perform ablation studies showing advantages of our approach. We also demonstrate our model to significantly outperform the state of the art on the challenging Nr3D, Sr3D and ScanRefer 3D object grounding datasets.
1 Introduction
3D object grounding must resolve language that describes relative locations and viewpoints while coping with limited 3D scene-language training data. ViL3DRel addresses these challenges with spatially informed attention and teacher-student relation learning, and reports gains across established benchmarks.
- 3D object grounding localizes sentence-referred objects in point clouds to support language-based interaction with real-world environments.
- Spatial expressions such as “closest to the piano” and “on the left when facing them” require distance comparisons, viewpoint inference, and same-class object disambiguation.
- 3D grounding has less scene-language training data than 2D grounding, increasing the difficulty of learning spatial relations.
- ViL3DRel uses spatial self-attention encoding pairwise distances and orientations, rotation augmentation, and teacher-student distillation from labeled-object inputs to point-cloud inputs.
- 9.3 and 8.3 absolute gains on Nr3D and Sr3D, respectively, and 4.47 points on ScanRefer are reported against previous work under the stated proposal settings.
2 Related Work
Prior 3D object-grounding methods use one- or two-stage architectures, object graphs, and multimodal transformers. These approaches differ in how they represent object relations and whether they transfer information from 2D data.
- One-stage methods fuse text with patch- or point-level visual features to regress bounding boxes directly, while two-stage methods use pre-detected object proposals.
- Object-graph approaches connect objects to nearest neighbors using Euclidean distance, which provides a structured representation of local relations.
- Recent methods adopt transformers for 3D grounding, including multimodal models and approaches that convert grounding into language modeling.
- Some methods transfer 2D semantics to assist 3D training, whereas other approaches rely on predicted object labels or point-level visual representations.
3 Method
ViL3DRel grounds language-referred objects through multimodal transformer processing that explicitly models 3D spatial relations. Its teacher-student training transfers relation knowledge from ground-truth semantic inputs to a point-cloud student.
- 3.1 Architecture Overview: The task uses object proposals from 3D point clouds or annotations to localize the object referred to by a sentence.The grounding target is a 3D bounding box, and the system follows a detection-then-matching framework.
- 3.1 Architecture Overview: ViL3DRel combines text encoding, object encoding, multimodal fusion, and a grounding head.Text is encoded with BERT, objects with PointNet++, and the grounding head scores proposals before softmax selection.
- 3.2 Spatial Self-Attention: The spatial self-attention layer complements standard self-attention by modeling pairwise distances, orientations, absolute locations, and object appearances.It uses language-conditioned spatial relevance and multi-head fusion to capture relations described in the text.
- 3.2 Spatial Self-Attention: A sigmoid-softmax fusion combines language-conditioned spatial relevance with standard self-attention to produce spatially informed object representations.The resulting attention explicitly considers 3D relative spatial locations, absolute spatial locations, and object appearances.
- 3.3 Teacher-Student Training: Teacher-student training uses ground-truth object semantics for the teacher and point clouds for the student, distilling attention and hidden-state relation knowledge.The approach is intended to reduce the cross-modal gap caused by incorrect object-class estimation without additional training data.
- 3.3 Teacher-Student Training: Training combines grounding, sentence-classification, object-classification, attention-distillation, and hidden-state-distillation losses.The attention and hidden-state terms are weighted by λ_a and λ_h in the overall objective.
4 Experiments
Experiments evaluate spatial relation modeling, textual encoders, attention mechanisms, and state-of-the-art performance across Nr3D, Sr3D, and ScanRefer. Results include ablations, qualitative attention analysis, and proposal-setting comparisons.
- Datasets: Nr3D contains 37,842 human-written sentences across 641 ScanNet scenes, while Sr3D uses template-generated spatial descriptions.Nr3D includes easy/hard and view-dependent/view-independent subsets; Sr3D sentences distinguish same-class objects using spatial relations.
- Evaluation setting: Ground-truth proposal evaluation selects the target box, whereas ScanRefer also evaluates regressed boxes using acc@0.25 and acc@0.5.The ScanRefer detected-proposal setting uses automatically generated object proposals.
- Ablations: Rotation augmentation improves Nr3D accuracy from 55.1% to 62.4%, with larger gains for view-independent than view-dependent sentences.The reported gains are +8.7% for view-independent sentences and +4.5% for view-dependent sentences.
- Ablations: Explicit orientation modeling improves orientation-related sentences by 10.5%, while distance modeling contributes most to distance-only sentences.Distance modeling has little influence on orientation-related samples.
- Qualitative analysis: The teacher’s cross-attention shifts from object-word alignment toward relation words, while self-attention initially emphasizes nearby objects.Figure 4 visualizes these learned attention patterns across multimodal fusion layers.
- Attention design: Multi-head spatial attention outperforms the corresponding single-head variant, supporting multiple heads for learning spatial relations.The comparison is reported against full model R9.
- State-of-the-art comparison: ViL3DRel gains 9.3% on Nr3D and 8.3% on Sr3D over the previous best method using ground-truth object proposals.The model also outperforms the cited state of the art on ScanRefer with both ground-truth and detected proposals.
5 Conclusion
The paper concludes that ViL3DRel combines spatial self-attention with teacher-student training to improve language-conditioned 3D relation reasoning. It reports strong performance across three benchmarks but identifies proposal, orientation, and dataset-diversity limitations.
- Contributions: ViL3DRel explicitly models relative distances and orientations in transformer self-attention and transfers relation knowledge from a labeled teacher to a point-cloud student.The teacher uses ground-truth object labels, while the student uses point cloud inputs.
- Results: The model significantly outperforms the state of the art on Nr3D, Sr3D, and ScanRefer.
- Limitations: The two-stage framework remains limited by imperfect object proposals in the first detection stage.The paper also notes unaddressed explicit object-orientation extraction and limited environmental diversity in existing datasets.
Checklist
The checklist reports that the paper addresses contributions, limitations, societal impacts, asset licensing, training details, reproducibility plans, error bars, and compute reporting.
- Disclosure: The authors report that they described their contributions and scope and discussed limitations in the conclusion.
- Ethics: The authors report discussing potential negative societal impacts and conforming to ethics review guidelines.
- Theory: The authors report that theoretical-results assumptions and proofs are not applicable because the paper does not present theoretical results.
- Reproducibility: Code, data, and reproduction instructions were not yet released, but the authors state they will release them upon acceptance.
- Reporting: The authors report training details, error bars, compute resources, asset citations, and licenses in the paper or supplementary material.
- Human subjects: The checklist marks human-subject consent, personally identifiable information, offensive content, participant risks, and compensation as not applicable.
A Implementation Details
The implementation computes pairwise spatial relations from object-center coordinates using distance and angular features, and applies discrete rotations during augmentation.
- Spatial features: Pairwise Euclidean distance and horizontal and vertical angles are computed from the centers of two objects.The horizontal angle is the azimuth from one object to the other, while the vertical angle is the elevation.
- Augmentation: Rotation augmentation randomly rotates the entire point cloud by 0, 90, 180, or 270 degrees.
A.3 Training losses
Training uses auxiliary cross-entropy losses for sentence-level and object-level class prediction, with different loss handling for teacher and student models.
- Lsent predicts the target object class from the sentence, while L*obj predicts each object token’s class.
- The teacher model uses all auxiliary losses during training.
- The student model’s L*obj loss does not affect weights because PointNet is fixed.
B.1 Ablations Studies on ScanRefer Dataset
ScanRefer ablations evaluate the proposed spatial self-attention and teacher-student training, showing that knowledge distillation improves student grounding with noisy object features.
- ScanRefer ablations separately evaluate the proposed spatial self-attention and teacher-student training.
- Over 2% boost is achieved by knowledge distillation over the baseline student model on ScanRefer.The comparison uses ground-truth object proposals.
- Both attention-weight and hidden-state distillation improve student training with noisy object features.
- Hidden-state distillation slightly outperforms attention distillation on ScanRefer.The passage contrasts this result with the Nr3D results and attributes the difference as a possible consequence of dataset language patterns.
B.2 Robustness to Random Seeds
The proposed ViL3DRel model is reported to remain stable across five random seeds, with low deviations in grounding accuracy.
- Five random seeds are used to measure average grounding accuracy and standard deviations.
- The proposed model is stable across seeds, with low deviations in grounding accuracy.
C Qualitative Results
Qualitative comparisons show ViL3DRel handling object perception and spatial relations, while failure cases identify errors associated with missing object proposals.
- The proposed model is reported to improve object perception and reasoning about relative distances and orientations.
- One reported failure involves missing object proposals for an outlet and another object.
- ViL3DRel is compared with a baseline lacking spatial self-attention and knowledge distillation.
- Figure 5 visualizes ground-truth targets and same-class distractors alongside predictions from ViL3DRel and the baseline.
- Figure 6 presents failure cases for the proposed model.