Source-linked AI summary
3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection
Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, Si Liu
TL;DR
3D visual grounding is constrained by two-stage detection and matching over irregular, large-scale point clouds. 3D-SPS instead uses language-guided progressive keypoint selection to localize targets in one stage, achieving state-of-the-art performance on ScanRefer and Nr3D/Sr3D.
Problem
Two-stage 3D visual grounding can miss targets with sparse proposals, confuse matching with dense proposals, and sample few target points because detection is language-irrelevant.
Method
3D-SPS performs single-stage 3D visual grounding through coarse language-relevant keypoint sampling with DKS followed by fine target-oriented mining with TPM.
Results
3D-SPS achieves state-of-the-art performance on both ScanRefer and Nr3D/Sr3D datasets.
Takeaways & Limitations
Progressive language-guided keypoint selection bridges detection and matching by directly localizing the target at a single stage.
Takeaways & Limitations
3D-SPS remains challenged by numerous and noisy points, complex 3D scenes, and the sparse, unordered structure of point clouds.
Abstract
from arXiv · showhide
3D visual grounding aims to locate the referred target object in 3D point cloud scenes according to a free-form language description. Previous methods mostly follow a two-stage paradigm, i.e., language-irrelevant detection and cross-modal matching, which is limited by the isolated architecture. In such a paradigm, the detector needs to sample keypoints from raw point clouds due to the inherent properties of 3D point clouds (irregular and large-scale), to generate the corresponding object proposal for each keypoint. However, sparse proposals may leave out the target in detection, while dense proposals may confuse the matching model. Moreover, the language-irrelevant detection stage can only sample a small proportion of keypoints on the target, deteriorating the target prediction. In this paper, we propose a 3D Single-Stage Referred Point Progressive Selection (3D-SPS) method, which progressively selects keypoints with the guidance of language and directly locates the target. Specifically, we propose a Description-aware Keypoint Sampling (DKS) module to coarsely focus on the points of language-relevant objects, which are significant clues for grounding. Besides, we devise a Target-oriented Progressive Mining (TPM) module to finely concentrate on the points of the target, which is enabled by progressive intra-modal relation modeling and inter-modal target mining. 3D-SPS bridges the gap between detection and matching in the 3D visual grounding task, localizing the target at a single stage. Experiments demonstrate that 3D-SPS achieves state-of-the-art performance on both ScanRefer and Nr3D/Sr3D datasets.
1. Introduction
3D-SPS replaces isolated detection and matching with language-guided progressive keypoint selection for single-stage 3D visual grounding. It uses DKS for coarse language-relevant sampling and TPM for fine target selection, achieving state-of-the-art performance on ScanRefer and Nr3D/Sr3D.
- 1. Introduction: Sparse proposals may omit the target, whereas dense proposals can confuse matching in traditional two-stage 3D visual grounding.These issues arise because detection and matching are isolated and 3D point clouds are large-scale and irregular.
- 1. Introduction: DKS focuses on language-relevant objects, and TPM progressively mines the target before regressing its bounding box from the selected keypoints.The method combines self- and cross-attention with global scene features during progressive target selection.
- 1. Introduction: 3D-SPS directly performs 3D visual grounding at a single stage, bridging the gap between detection and matching.The paper identifies this as its first single-stage 3D visual grounding formulation.
- 1. Introduction: 3D-SPS treats visual grounding as keypoint selection and progressively selects target-related points with language guidance.DKS samples coarse language-relevant keypoints, while TPM mines cross-modal relationships to distinguish the target.
- 1. Introduction: 3D-SPS achieves state-of-the-art performance on both ScanRefer and Nr3D/Sr3D datasets.The method is evaluated against existing 3D visual grounding approaches.
2. Related Work
Prior 3D visual grounding methods mainly use detection followed by cross-modal matching, while 2D one-stage strategies do not transfer directly to irregular 3D point clouds. 3D-SPS instead uses progressive keypoint selection to address this mismatch.
- 2. Related Work: 2D one-stage visual grounding methods cannot be directly transferred to 3D because point clouds are numerous, noisy, sparse, unordered, and irregular.Treating every 3D point as a candidate is computationally unacceptable, and sliding-window strategies rely on regular 2D inputs.
- 2. Related Work: The proposed framework avoids separating detection and matching by selecting language-relevant points before concentrating on the referred target.The framework then predicts referring confidence scores and regresses the target box from the highest-scoring keypoint.
- 2. Related Work: Existing 3D visual grounding methods generally follow a detection-then-matching framework that does not use language during detection.Their detection stage samples keypoints and generates proposals before cross-modal matching.
3. Method
3D-SPS replaces detection-then-matching with language-guided progressive keypoint selection, using DKS for coarse language-relevant sampling and TPM for fine target mining before direct box regression.
- Overview: The method extracts seed points and word features, then applies DKS, TPM, confidence scoring, and box regression in sequence.DKS produces language-relevant keypoints; TPM progressively reduces them while updating language features.
- Overview: 3D-SPS treats 3D visual grounding as selecting target keypoints from point clouds and directly regressing the target bounding box.The highest referring-confidence keypoint supplies the predicted box center and size.
- Description-aware Keypoint Sampling: DKS first selects object-centered points, then ranks them by description relevance to retain keypoints from language-relevant objects.The two scores respectively identify points near object centers and points related to the description context.
- Target-oriented Progressive Mining: TPM uses a two-stream cross-modal transformer with scene-location modeling and language-points attention to discard irrelevant keypoints layer by layer.Average-pooled cross-attention scores determine which keypoints continue to the next layer.
- Training Objectives: Training combines visual-grounding, DKS, detection, and language-classification losses, with multi-hot supervision for feasible target keypoints.The visual-grounding loss supervises referring confidence, while DKS uses focal-loss supervision for object confidence and description relevance.
4. Experiments
Experiments evaluate 3D-SPS on ScanRefer and Nr3D/Sr3D, then analyze progressive selection, module ablations, and qualitative behavior. Results show strong benchmark performance and language-guided concentration on relevant and target keypoints.
- Quantitative Comparison: 3D-SPS outperforms existing methods on ScanRefer, improving over prior state of the art by +1.96% at Acc@0.5 and +1.75% at Acc@0.25 with 3D-only input.With 2D+3D input, the improvements are 2.31% at Acc@0.5 and 1.25% at Acc@0.25.
- Effectiveness Validation: Progressive language-relevant selection increases the target-keypoint ratio after each selection and keeps it above language-irrelevant sampling, while 3D-SPS performance improves as sampled seed points increase.The two-stage baseline instead improves initially and then declines as the sampled point number grows.
- Quantitative Comparison: 3D-SPS improves grounding accuracy by +2.3% on Nr3D and +4.7% on Sr3D, with a large margin over LanguageRefer on Sr3D Hard samples.For Nr3D/Sr3D, DKS is removed because ground-truth object boxes are provided, leaving TPM to verify progressive selection.
- Ablation Study: Using both object-confidence and description-relevance scores in DKS produces promising results by selecting description-related keypoints near object centers.The ablation compares FPS, each score separately, and their joint use.
- Ablation Study: More TPM layers improve accuracy, but the default T = 4 avoids potentially leaving out target keypoints and missing the best bounding box.The evaluated layer counts are T ∈ {1, 2, 3, 4, 5}.
- Qualitative Comparison: Qualitative comparisons show that DKS and TPM progressively focus on the target, whereas ScanRefer’s scattered attention fails to locate it.For the same target, 3D-SPS changes selected keypoints when descriptions mention different contextual objects such as windows, desks, beds, or armchairs.
5. Conclusion and Discussion
3D-SPS is a coarse-to-fine, language-guided framework that progressively selects keypoints and directly localizes targets in 3D visual grounding. It achieves new state-of-the-art performance on ScanRefer and Nr3D/Sr3D, while view-dependent descriptions and ambiguous queries remain challenging.
- 3D-SPS progressively selects language-guided keypoints in a coarse-to-fine pattern and directly localizes the target at a single stage.The framework is designed to bridge detection and matching within one grounding process.
- 3D-SPS outperforms existing 3D visual grounding methods by a large margin on both ScanRefer and Nr3D/Sr3D, establishing new state-of-the-art performance.
- View-dependent descriptions and ambiguous queries can confuse 3D-SPS because 3D point clouds and free-form descriptions remain complex.The authors identify these issues as limitations and directions for future work.