PaperlayerView exact revision on arXiv

Source-linked AI summary

Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud

Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang, Ajmal Mian

arXiv:2103.16381v1cs.CV

TL;DR

Free-form 3D object grounding requires connecting complex descriptions to targets in raw point-cloud scenes, where language structure and scene context matter. The paper proposes language and visual graphs that model those relationships and match phrases to proposals. It reports state-of-the-art performance on ScanRefer and Nr3D.

  • Problem

    Grounding complex, multi-sentence descriptions in point clouds is challenging, and prior methods underuse long-range language relations, proposal context, and phrase–object structure.

    Source: §1 · pp. 1–2

  • Method

    The method parses descriptions into a language scene graph, strengthens initial proposals with object–object and object–scene context, then matches language nodes to a visual graph over refined proposals.

    Source: §3.1 · p. 3, §3.2 · p. 5, §6 · p. 8

  • Results

    The authors report state-of-the-art results on ScanRefer and Nr3D.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The paper reports accurate grounding through modeling relationships between natural-language descriptions and 3D scenes.

    Source: §6 · p. 8

Abstract

from arXiv · show

3D object grounding aims to locate the most relevant target object in a raw point cloud scene based on a free-form language description. Understanding complex and diverse descriptions, and lifting them directly to a point cloud is a new and challenging topic due to the irregular and sparse nature of point clouds. There are three main challenges in 3D object grounding: to find the main focus in the complex and diverse description; to understand the point cloud scene; and to locate the target object. In this paper, we address all three challenges. Firstly, we propose a language scene graph module to capture the rich structure and long-distance phrase correlations. Secondly, we introduce a multi-level 3D proposal relation graph module to extract the object-object and object-scene co-occurrence relationships, and strengthen the visual features of the initial proposals. Lastly, we develop a description guided 3D visual graph module to encode global contexts of phrases and proposals by a nodes matching strategy. Extensive experiments on challenging benchmark datasets (ScanRefer and Nr3D) show that our algorithm outperforms existing state-of-the-art. Our code is available at https://github.com/PNXD/FFL-3DOG.

1. Introduction

Free-form 3D object grounding must connect complex language descriptions to objects in sparse point-cloud scenes, a task that existing methods handle only partially. The paper proposes language and visual graph modules and reports state-of-the-art results on ScanRefer and Nr3D.

1. Introduction

  • Free-form descriptions can convey a target’s appearance, its spatial relations to nearby objects, and information about the global scene.The introduction illustrates this with a request to find a brown table using its location relative to a cabinet, curtain, and shoes.Source: §1 · p. 1
  • Grounding free-form language in point clouds is a comparatively new vision-and-language topic, with point-cloud representations becoming more available across application domains.Source: §1 · p. 1
  • Existing methods use holistic descriptions, coarse proposals, and global language–visual fusion, leaving long-range language links, proposal context, and phrase–object structure underused.These limitations affect both complex language processing and irregular point-cloud recognition.Source: §1 · p. 2
  • The proposed end-to-end network directly grounds free-form descriptions in raw point clouds using language scene, proposal relation, and description-guided visual graphs.The language graph captures phrase structure; the proposal graph models object–object and object–scene co-occurrence; and the visual graph matches phrase and proposal nodes.Source: §1 · p. 2
  • The paper reports state-of-the-art results on ScanRefer and Nr3D.Source: §1 · p. 2

2. Related work

Prior grounding work has largely focused on images or simpler language inputs, while connecting natural language to 3D point clouds remains relatively new. This paper distinguishes its approach by modeling free-form language structure, scene co-occurrence, and phrase–proposal context.

2. Related work

  • Most 2D grounding methods use two stages—generating image-box candidates and ranking them against a sentence—and often focus on single-sentence descriptions.The passage says these methods cannot be directly used for 3D point-cloud grounding.Source: §2 · p. 2
  • The work extends 3D object grounding from ScanRefer, capturing structure in free-form descriptions, extracting object-object and object-scene co-occurrences, and encoding global phrase and proposal contexts through node matching.Source: §2 · p. 3
  • The paper’s approach differs by capturing free-form language structure, object–object and object–scene co-occurrence, and global phrase–proposal context through node matching.Source: §2 · p. 3

3. Proposed Approach

The approach combines language-graph construction and proposal-context modeling with a description-guided visual graph over refined proposals. Language and visual nodes are matched to produce the final grounding.

3. Proposed Approach

  • The network has language scene, multi-context proposal relation, and description-guided 3D visual graph modules, with language-guided pruning and matching in the final stage.The proposal relation graph models object–object and object–scene co-occurrence; the visual graph is built over pruned and refined proposals.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Language Scene Graph Module

  • The language scene graph represents noun phrases as nodes and relations as edges, using parsing and post-processing to extract attributes, resolve pronouns, and normalize relation phrases.The description is also encoded to provide global context, while message propagation refines phrase features.Source: §3.1 · pp. 3–4

3. Proposed Approach

  • The proposal relation graph incorporates proposal appearance and spatial features, including box geometry and relative center locations, alongside global scene information.It models object–object and object–scene co-occurrence to strengthen initial proposal features.Source: §3.2 · pp. 4–5

3.3. Description guided 3D visual graph module

  • Language-guided pruning ranks phrase–proposal pairs, selects proposals, and regresses their boxes before a visual graph is formed over the refined proposals.The graph combines appearance, spatial, and description features and propagates context across proposal nodes and edges.Source: §3.3 · p. 5

4. Prediction model

The prediction model grounds noun phrases by matching nodes in the language scene graph and 3D visual graph, then selects the highest-scoring proposal for the subject. Its training objective combines six loss terms.

4. Prediction model

  • The model formulates grounding as node matching between language-scene-graph noun phrases and 3D-visual-graph proposals.A matching score is fused with the object-pruning score to produce the node-matching score.Source: §4 · p. 5
  • The proposal box with the highest score is selected as the target 3D bounding box, with selection limited to the subject.The model combines the matching score with the pruning score before selection.Source: §4 · p. 5
  • The final loss linearly combines vote, objectness, bounding-box, semantic-classification, description-classification, and reference losses.These terms supervise voting, objectness, box prediction, semantic and description classification, and similarity scores and offsets.Source: §4 · p. 6

5. Experiments

Experiments evaluate the method on ScanRefer and Nr3D, using ablations, benchmark comparisons, and qualitative visualizations to assess its components and grounding performance.

5.1. Datasets

  • ScanRefer contains 51,583 descriptions of 11,046 objects across 800 ScanNet scenes, while Nr3D contains 41,503 human utterances collected through an online reference game.The experiments use the official training and validation splits for both datasets.Source: §5.1 · p. 6

5.2. Implementation details

  • The model starts from 256 VoteNet proposals, selects 20 proposals for its visual graph, and is trained for 30 epochs with ADAM.The stated initial learning rate is 1e−3, with decay after epochs 5, 15, and 25.Source: §5.2 · p. 6

5.3. Ablation studies

  • Using ground-truth 3D boxes improves overall Acc@0.5 by 24.97%, indicating substantial headroom in current 3D object detection backbones.The study also reports that 2D projections are inaccurate in 3D and semantic class labels alone do not identify the referred object.Source: §5.3 · p. 6
  • Adding RGB to xyz features improves performance, and adding normals improves it further; removing LGVG causes a larger drop than removing LGS or MLPG.The authors relate the feature gains to language-described attributes and the larger LGVG ablation drop to ambiguity between proposals and phrases.Source: §5.3 · p. 7

5.4. Quantitative comparisons

  • On ScanRefer, the method reports ~1.5% and ~2.8% Acc@0.5 improvements for scenes with single-class and multiple-class objects; on Nr3D, it exceeds InstanceRefer by 2.9%, ReferIt3D by 6.1%, and ScanRefer by 7.5%.The ScanRefer comparison also reports larger gains in Multiple than Unique cases; the Nr3D comparison reports the highest gain in Hard cases.Source: §5.4 · p. 7, Table 2 · p. 7, Table 3 · p. 7

5.5. Visualization

  • In the first two Figure 3 examples, the proposed modules handle spatial relations and regress more accurate boxes than ScanRefer; later examples show ScanRefer failing on ambiguous objects.The figure covers ScanRefer in its first four columns and Nr3D in its last column.Source: §5.5 · p. 8, Figure 3 · p. 8
  • Figure 4 visualizes noun-phrase matches to same-category instances and strongly related objects, pruning and refining the initially redundant proposal set.The visualization shows the most relevant 3D bounding boxes for each noun phrase in the description-guided visual graph module.Source: §5.5 · p. 8, Figure 4 · p. 8

6. Conclusion

The proposed network combines language and visual graph reasoning to capture relationships within descriptions and 3D scenes. Experiments on two challenging benchmarks show quantitative and qualitative improvements over prior state of the art.

6. Conclusion

  • Experiments on two challenging benchmark datasets show that the method quantitatively and qualitatively outperforms existing state of the art in 3D object grounding.Source: §6 · p. 8
  • A language scene graph parses free-form descriptions and uses message propagation to compute context-aware phrase representations.Source: §6 · p. 8
  • A multi-level 3D relation graph models object-object and object-scene co-occurrences, while a visual graph encodes phrase and proposal context over refined proposals.Source: §6 · p. 8

Related papers

Loading 2103.16381v1…