3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection

1. Introduction

• Sparse proposals may omit the target, whereas dense proposals can confuse matching in traditional two-stage 3D visual grounding.These issues arise because detection and matching are isolated and 3D point clouds are large-scale and irregular.Source: §1 · pp. 1–2, Figure 1 · p. 1
• DKS focuses on language-relevant objects, and TPM progressively mines the target before regressing its bounding box from the selected keypoints.The method combines self- and cross-attention with global scene features during progressive target selection.Source: §1 · p. 2
• 3D-SPS directly performs 3D visual grounding at a single stage, bridging the gap between detection and matching.The paper identifies this as its first single-stage 3D visual grounding formulation.Source: §1 · p. 2
• 3D-SPS treats visual grounding as keypoint selection and progressively selects target-related points with language guidance.DKS samples coarse language-relevant keypoints, while TPM mines cross-modal relationships to distinguish the target.Source: §1 · p. 2
• 3D-SPS achieves state-of-the-art performance on both ScanRefer and Nr3D/Sr3D datasets.The method is evaluated against existing 3D visual grounding approaches.Source: §1 · p. 2

2. Related Work

• 2D one-stage visual grounding methods cannot be directly transferred to 3D because point clouds are numerous, noisy, sparse, unordered, and irregular.Treating every 3D point as a candidate is computationally unacceptable, and sliding-window strategies rely on regular 2D inputs.Source: §2 · p. 2
• The proposed framework avoids separating detection and matching by selecting language-relevant points before concentrating on the referred target.The framework then predicts referring confidence scores and regresses the target box from the highest-scoring keypoint.Source: Figure 2 · p. 3
• Existing 3D visual grounding methods generally follow a detection-then-matching framework that does not use language during detection.Their detection stage samples keypoints and generates proposals before cross-modal matching.Source: §2 · p. 3

Overview

• The method extracts seed points and word features, then applies DKS, TPM, confidence scoring, and box regression in sequence.DKS produces language-relevant keypoints; TPM progressively reduces them while updating language features.Source: §3.1 · pp. 3–4
• 3D-SPS treats 3D visual grounding as selecting target keypoints from point clouds and directly regressing the target bounding box.The highest referring-confidence keypoint supplies the predicted box center and size.Source: §3.1 · p. 4

Description-aware Keypoint Sampling

• DKS first selects object-centered points, then ranks them by description relevance to retain keypoints from language-relevant objects.The two scores respectively identify points near object centers and points related to the description context.Source: §3.2 · p. 4, Figure 3 · p. 4

Target-oriented Progressive Mining

• TPM uses a two-stream cross-modal transformer with scene-location modeling and language-points attention to discard irrelevant keypoints layer by layer.Average-pooled cross-attention scores determine which keypoints continue to the next layer.Source: §3.3 · pp. 4–5, Figure 4 · p. 4

Training Objectives

• Training combines visual-grounding, DKS, detection, and language-classification losses, with multi-hot supervision for feasible target keypoints.The visual-grounding loss supervises referring confidence, while DKS uses focal-loss supervision for object confidence and description relevance.Source: §3.4 · p. 5

Quantitative Comparison

• 3D-SPS outperforms existing methods on ScanRefer, improving over prior state of the art by +1.96% at Acc@0.5 and +1.75% at Acc@0.25 with 3D-only input.With 2D+3D input, the improvements are 2.31% at Acc@0.5 and 1.25% at Acc@0.25.Source: §4.3 · p. 6, Table 1 · p. 6

Effectiveness Validation

• Progressive language-relevant selection increases the target-keypoint ratio after each selection and keeps it above language-irrelevant sampling, while 3D-SPS performance improves as sampled seed points increase.The two-stage baseline instead improves initially and then declines as the sampled point number grows.Source: Figure 5 · p. 7, §4.3 · p. 7

Quantitative Comparison

• 3D-SPS improves grounding accuracy by +2.3% on Nr3D and +4.7% on Sr3D, with a large margin over LanguageRefer on Sr3D Hard samples.For Nr3D/Sr3D, DKS is removed because ground-truth object boxes are provided, leaving TPM to verify progressive selection.Source: §4.3 · pp. 6–7, Table 2 · p. 6

Ablation Study

• Using both object-confidence and description-relevance scores in DKS produces promising results by selecting description-related keypoints near object centers.The ablation compares FPS, each score separately, and their joint use.Source: §4.4 · p. 7, Table 3 · p. 7
• More TPM layers improve accuracy, but the default T = 4 avoids potentially leaving out target keypoints and missing the best bounding box.The evaluated layer counts are T ∈ {1, 2, 3, 4, 5}.Source: §4.4 · p. 7, Table 4 · p. 7

Qualitative Comparison

• Qualitative comparisons show that DKS and TPM progressively focus on the target, whereas ScanRefer’s scattered attention fails to locate it.For the same target, 3D-SPS changes selected keypoints when descriptions mention different contextual objects such as windows, desks, beds, or armchairs.Source: §4.5 · pp. 7–8, Figure 6 · p. 8, Figure 7 · p. 8

• 3D-SPS progressively selects language-guided keypoints in a coarse-to-fine pattern and directly localizes the target at a single stage.The framework is designed to bridge detection and matching within one grounding process.Source: §5 · p. 8
• 3D-SPS outperforms existing 3D visual grounding methods by a large margin on both ScanRefer and Nr3D/Sr3D, establishing new state-of-the-art performance.Source: §5 · p. 8
• View-dependent descriptions and ambiguous queries can confuse 3D-SPS because 3D point clouds and free-form descriptions remain complex.The authors identify these issues as limitations and directions for future work.Source: §5 · p. 8