Source-linked AI summary
InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, Shuguang Cui
TL;DR
Natural-language 3D grounding on point clouds is challenging because point clouds are sparse, irregular, and unordered, and proposal-based methods use redundant candidates with incomplete attributes and under-modeled relations. InstanceRefer filters panoptic-segmentation instances by language-predicted category, then matches the remaining candidates using cooperative attribute, relation, and global-localization perceptions. It achieves state-of-the-art results on ScanRefer, Nr3D, and Sr3D.
Problem
Point-cloud visual grounding is challenging because sparse, irregular, unordered data and redundant proposals limit attribute capture and contextual relation modeling.
Method
InstanceRefer predicts the target category from language, filters panoptic-segmentation instances, and matches candidates through attribute, relation, and global-localization perceptions.
Results
InstanceRefer achieves state-of-the-art results on the ScanRefer benchmark and Nr3D/Sr3D datasets.
Takeaways & Limitations
The framework formulates 3D visual grounding as instance matching with cooperative holistic scene-language understanding.
Abstract
from arXiv · showhide
Compared with the visual grounding on 2D images, the natural-language-guided 3D object localization on point clouds is more challenging. In this paper, we propose a new model, named InstanceRefer, to achieve a superior 3D visual grounding through the grounding-by-matching strategy. In practice, our model first predicts the target category from the language descriptions using a simple language classification model. Then, based on the category, our model sifts out a small number of instance candidates (usually less than 20) from the panoptic segmentation of point clouds. Thus, the non-trivial 3D visual grounding task has been effectively re-formulated as a simplified instance-matching problem, considering that instance-level candidates are more rational than the redundant 3D object proposals. Subsequently, for each candidate, we perform the multi-level contextual inference, i.e., referring from instance attribute perception, instance-to-instance relation perception, and instance-to-background global localization perception, respectively. Eventually, the most relevant candidate is selected and localized by ranking confidence scores, which are obtained by the cooperative holistic visual-language feature matching. Experiments confirm that our method outperforms previous state-of-the-arts on ScanRefer online benchmark and Nr3D/Sr3D datasets.
1. Introduction
Point-cloud visual grounding is harder than image grounding because point clouds are sparse, irregular, and unordered, while proposal-based 3D methods introduce redundancy and miss attribute and contextual relations. InstanceRefer addresses these issues by filtering segmentation instances using language cues and matching candidates through attributes, local relations, and global localization, achieving state-of-the-art results on ScanRefer, Nr3D, and Sr3D.
- 3D point clouds are sparse, irregular, and unordered, making 2D visual-grounding schemes deficient in real 3D scenarios.
- Proposal-based 3D grounding generates hundreds of redundant proposals, creating complex inter-proposal relationships that introduce noise and ambiguity.
- Sparse and incomplete object point clouds hinder attribute extraction, while relations among proposals and between proposals and background remain insufficiently studied.
- InstanceRefer filters panoptic-segmentation instances using predicted language categories, preserving original geometric and attribute details while reducing candidate count.
- Its three components jointly select candidates using instance attributes, local relations, and global localization, reframing grounding as instance matching.
- InstanceRefer achieves state-of-the-art performance on the ScanRefer benchmark and Nr3D/Sr3D datasets.
2. Related Work
Prior visual grounding commonly selects a matching proposal after proposal generation, while point-cloud methods address irregular 3D representations through point-based learning and related perception tasks. InstanceRefer instead builds on instance-level candidates and multi-level contextual referring for point-cloud grounding.
- 2D visual grounding typically generates candidate proposals first and then selects the best matching proposal for a language query.
- InstanceRefer extracts semantic instances, filters them using language-predicted categories, and applies attribute, relation, and global-localization perception modules.
- Point-cloud learning methods include graph-based learning and convolution-like operations for irregular 3D representations.
3. Method
InstanceRefer converts point-cloud visual grounding into instance matching by filtering panoptic instances with language-derived categories, then progressively comparing candidates through attribute, relational, and global-localization context.
- Instance candidate generation: InstanceRefer extracts foreground instances through panoptic segmentation and reformulates point-cloud visual grounding as instance matching.The framework uses instance masks and semantic labels to obtain instance-level point clouds from the scene.
- Language-guided filtering: Language encoding uses GloVe embeddings, bidirectional GRUs, attention pooling, and target-category prediction to filter candidate instances.The predicted category guides selection of candidates belonging to the queried class.
- Attribute perception: The AP module applies four-layer SparseConv and average pooling to produce a global representation of each candidate’s attributes.SparseConv operates only on non-empty voxels after voxelizing the candidate point cloud.
- Relation perception: The RP module searches nearest instance-level neighborhoods and aggregates candidate-neighbor information using relative positions and semantic relationships.Its graph-based aggregation encodes relations between each candidate and surrounding instances.
- Global localization perception: The GLP module predicts a candidate’s location in a 3×3 BEV plane, interpolates area probabilities onto candidate points, and aggregates them with candidate features.The module uses whole-scene point-wise features to incorporate background localization information.
- Cooperative matching: A contrastive objective treats instances with IoU above threshold Γ as positive examples and trains matching scores to distinguish positive from negative candidates.All candidates participate in optimization when a query has at least one positive example.
4. Experiments
Experiments evaluate InstanceRefer’s implementation, datasets, metrics, benchmark performance, qualitative behavior, and module contributions. Results show strong performance across ScanRefer and ReferIt3D benchmarks, with gains linked to filtered instance candidates and multi-level perception.
- 4.1. Implementation: Experiments use PointGroup for panoptic segmentation, GloVE and BiGRU language encoding, 30 training epochs, Adam optimization, and a single NVIDIA 1080Ti GPU.Language features use 256 channels; the learning rate starts at 0.0005 and decays by 0.9 every 10 epochs.
- 4.2. Dataset and Metrics: ScanRefer evaluation reports 3D bounding-box IoU through Acc@mIoU at m∈{0.25, 0.5}, separating unique and multiple object categories.The study compares both validation and online test-set performance.
- 4.3. Quantitative Comparisons: InstanceRefer achieves the highest scores on both the ScanRefer validation set and online benchmark, including 11.8% in unique and 6.7% in multiple at Acc@0.5.The comparison includes methods using different input modalities and language encoders.
- 4.3. Quantitative Comparisons: InstanceRefer achieves approximately 10% improvement on Acc@0.5 over ScanRefer despite ScanRefer using additional multiview features.Offline segmentation results also provide shorter training time and lower memory consumption than other methods.
- 4.3. Quantitative Comparisons: InstanceRefer achieves top-ranked referring-object identification on both Nr3D and Sr3D, demonstrating effective instance-matching performance.The results are reported across the datasets’ referring-object identification task.
- 4.4. Comprehensive Analysis: Ablations show that AP performs best alone, while adding RP and GLP produces a marked improvement, with RP contributing slightly more than GLP.The visual-language matching method also outperforms simple cosine-similarity ranking.
5. Conclusion
InstanceRefer performs 3D visual grounding by filtering panoptic-segmentation instances and jointly modeling attributes, relations, and global localization. Experiments show it outperforms previous methods by a large margin.
- InstanceRefer predicts the target category from language and filters a small candidate set using panoptic segmentation.This reduces grounding to instance-level candidate matching.
- The framework jointly uses instance attributes, instance-to-instance relations, and instance-to-background global localization.These form its multi-level contextual referring strategy.
- InstanceRefer selects and localizes the most relevant candidate through cooperative holistic scene-language understanding.
- InstanceRefer outperforms previous methods by a large margin in experimental evaluations.