PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi3DRefer: Grounding Text Description to Multiple 3D Objects

Yiming Zhang, ZeMing Gong, Angel X. Chang

arXiv:2309.05251v1cs.CV

TL;DR

Existing 3D visual grounding assumes a unique target, despite descriptions that may naturally refer to zero or multiple objects. Multi3DRefer generalizes ScanRefer with flexible-target data and evaluation, and introduces a CLIP-based online-rendering approach that outperforms prior work on ScanRefer.

  • Problem

    Existing 3D visual grounding tasks assume a unique target object, making descriptions referring to multiple objects problematic.

  • Method

    Multi3DRefer generalizes ScanRefer with zero-, single-, and multiple-target descriptions, new evaluation metrics, prior-work benchmarks, and a CLIP-based method using online-rendered object proposals with contrastive learning.

  • Results

    The proposed CLIP-based approach outperforms prior works on ScanRefer, while combining CLIP and 3D features yields the best ablation performance.

  • Takeaways & Limitations

    Multi3DRefer provides a more realistic flexible-number 3D grounding benchmark with practical relevance to robotics and embodied AI.

  • Takeaways & Limitations

    The current design relies on 3D detector features for global context and a 2D image encoder for per-object attributes, leaving positional encoding for future investigation.

Abstract

from arXiv · show

We introduce the task of localizing a flexible number of objects in real-world 3D scenes using natural language descriptions. Existing 3D visual grounding tasks focus on localizing a unique object given a text description. However, such a strict setting is unnatural as localizing potentially multiple objects is a common need in real-world scenarios and robotic tasks (e.g., visual navigation and object rearrangement). To address this setting we propose Multi3DRefer, generalizing the ScanRefer dataset and task. Our dataset contains 61926 descriptions of 11609 objects, where zero, single or multiple target objects are referenced by each description. We also introduce a new evaluation metric and benchmark methods from prior work to enable further investigation of multi-modal 3D scene understanding. Furthermore, we develop a better baseline leveraging 2D features from CLIP by rendering object proposals online with contrastive learning, which outperforms the state of the art on the ScanRefer benchmark.

1. Introduction

Multi3DRefer extends 3D visual grounding beyond unique-object assumptions by supporting descriptions that refer to zero, one, or multiple objects. It introduces an enhanced dataset, evaluation metrics, benchmarks, and a CLIP-based approach for this flexible setting.

  • Existing 3D visual grounding datasets assume each description refers to exactly one target object.
  • Multi3DRefer modifies ScanRefer language data and proposes evaluation metrics for flexible-number visual grounding.
  • 61926 descriptions across 800 ScanNet V2 scenes support grounding zero, single, or multiple target objects.
  • The benchmark includes three adapted prior 3D visual grounding approaches and an end-to-end CLIP method using online-rendered object proposals with contrastive learning.

2. Related work

Prior grounding work largely targets unique objects, with limited exceptions for unanswerable queries or short category-based multi-object queries. Multi3DRefer studies richer zero-, single-, and multiple-target descriptions in 3D scenes.

  • Most 2D grounding datasets associate each phrase with exactly one object, although VGPhraseCut supports potentially multiple instance segments.
  • Multi3DRefer contributes a 3D grounding setting in which one description may describe one or more objects.
  • Some recent work handles unanswerable queries, but other multi-object formulations typically use short, category-based queries.
  • ScanRefer combines object detection and grounding, whereas ReferIt3D discriminates one target from same-class objects using ground-truth boxes.
  • Prior 3D methods explore graph representations, spatial relations, neurosymbolic reasoning, multi-view imagery, 2D semantics, and unified grounding-captioning models.

3. Multi3DRefer dataset

Multi3DRefer extends ScanRefer with zero-, single-, and multiple-target descriptions, using automated augmentation, ChatGPT rephrasing, and manual verification to create diverse language data.

  • The dataset contains Zero Target, Single Target, and Multiple Targets description-scene pairs, representing zero, one, or multiple matching objects.
  • 9324 ScanRefer descriptions were found ambiguous and used as an initial source of multiple-target descriptions, alongside 7741 additional ambiguous descriptions from the ScanRefer authors.
  • The construction pipeline generates additional samples through automated description generation followed by verification and modification, reducing manual annotation requirements.
  • 3.2. Rephrasing using ChatGPT: ChatGPT rephrasing preserves original meaning while making descriptions more natural, correcting typos, and providing richer vocabulary with shorter descriptions.
  • All generated or rephrased scene-description pairs are manually checked for writing quality and accurate matching between descriptions and target objects.
  • 61926 descriptions are included, including 51583 directly obtained from ScanRefer, with 6688 zero-target and 13178 multiple-target descriptions.

4. Task

Multi3DRefer asks models to predict every axis-aligned bounding box matching a free-form description when the number of referred objects may be zero, one, or multiple.

  • The task takes a 3D point-cloud scene and free-form language description as input, then predicts axis-aligned bounding boxes for all matching objects.
  • The number of referred objects is flexible, so descriptions can match no objects, one object, or multiple objects.
  • Evaluation uses F1@0.25 and F1@0.5, measuring grounding at IoU thresholds of 0.25 and 0.5.
  • Performance is evaluated separately for five cases: zero targets with or without same-class distractors, single targets with or without distractors, and multiple targets.
  • Hungarian matching pairs predicted and ground-truth boxes one-to-one before IoU thresholding determines true positives.

5. Method

M3DRef-CLIP combines PointGroup proposals, rendered multi-view CLIP features, 3D features, and language through transformer fusion to score object-description matches.

  • M3DRef-CLIP is compared with 3DVG-Transformer, 3DJCG, and D3Net, using two-stage pipelines with detection followed by grounding.
  • 5.1. M3DRef-CLIP: PointGroup detects object proposals and supplies their 3D features, while an online renderer produces multi-view images for CLIP-based 2D feature extraction.
  • 5.1. M3DRef-CLIP: The model concatenates 2D and 3D object features, combines them with token-level language embeddings through transformer fusion, and outputs confidence scores for each proposal.
  • The total training loss combines detection, contrastive, and reference losses, with binary cross-entropy used for Multi3DRefer reference supervision.
  • A contrastive loss aligns sentence features with the mean object features of all target objects paired with a description.

6. Experiments

Experiments evaluate M3DRef-CLIP across ScanRefer, Nr3D, and Multi3DRefer under ground-truth and predicted-box settings, including ablations of features and contrastive learning. M3DRef-CLIP generally performs strongly, especially when combining CLIP and 3D features, while distractors remain challenging.

  • Evaluation setup: Experiments cover ScanRefer, Nr3D, and Multi3DRefer using both ground-truth and predicted bounding-box setups.The study compares M3DRef-CLIP with prior approaches and analyzes multiple target scenarios.
  • ScanRefer and Nr3D: PointGroup-based methods outperform VoteNet-based methods on ScanRefer’s unique subset, while CLIP text encoding is a key factor in M3DRef-CLIP’s ScanRefer performance.The authors also identify PointGroup as a strong 3D object detector for this setting.
  • ScanRefer and Nr3D: Nr3D results are comparable but less competitive because of a weaker ground-truth box encoder; additional 2D image information helps two-stage methods overall.SAT and MVT also use 2D images, but render them offline.
  • Multi3DRefer results: M3DRef-CLIP outperforms other methods on Multi3DRefer on most metrics.The comparison includes four two-stage methods evaluated with ground-truth and predicted boxes.
  • Multi3DRefer results: Same-class distractors are more challenging, while Multi3DRefer training achieves similar performance to ScanRefer training on the Single Target setting.Reworded training data also improves ScanRefer performance under ground-truth boxes.
  • Ablation studies: Combining CLIP and 3D features achieves the best ablation performance, whereas CLIP image features alone underperform 3D features.Contrastive loss benefits ScanRefer, Nr3D, and Multi3DRefer, especially Nr3D.

7. Conclusion

The paper presents Multi3DRefer as a more realistic flexible-cardinality 3D grounding task and develops online-rendered 2D cues with CLIP and contrastive learning. It identifies positional encoding for spatial relations as an important direction for future work and links the task to robotics and embodied AI.

  • Conclusion: Multi3DRefer grounds descriptions to zero, single, or multiple objects in real-world 3D scenes.This flexible target cardinality is presented as a more realistic grounding setting.
  • Conclusion: Online rendering of proposal objects supplies 2D cues, while CLIP and multimodal contrastive loss are useful components of the approach.The conclusion highlights these design choices as contributions of the method.
  • Conclusion: The authors position Multi3DRefer as relevant to bridging 3D vision and language for robotics and embodied AI tasks.This is stated as the dataset’s expected practical value and future direction.
  • Future Work: The current design relies on 3D detector features for global context and a 2D image encoder for per-object attributes.The authors suggest positional encoding may improve handling of spatial relations.

A. Web interface for verification

The verification interface presents generated descriptions alongside interactive 3D scenes, highlighting selected target objects for human checking. The dataset statistics and verification process include substantial human review and filtering of generated samples.

  • Web interface: Verifiers inspect generated descriptions with an interactive 3D mesh, where selected target objects are highlighted in green.They check whether the description matches the identified targets and correct mismatches.
  • Verification process: 64,513 description-scene pairs were checked by verifiers.The verification application supported large-scale review of generated data.
  • Qualitative examples: Figure 8 illustrates single-match, viewpoint-dependent multiple-match, and viewpoint-specified spatial descriptions.These examples show why spatial relations can produce different target sets depending on viewpoint.
  • Verification process: 2,587 samples were discarded, and 11,804 descriptions were modified during verification.Discarding limited zero-target descriptions per scene and overly similar descriptions; most edits were minor or added constraints.

B. Statistics and examples of Multi3DRefer

Multi3DRefer statistics describe how descriptions and target objects are distributed across scene and object types. Single Target descriptions follow real-world object distributions, whereas Zero Target descriptions are more evenly distributed.

  • Dataset statistics: Single Target descriptions reflect real-world object distributions, while Zero Target descriptions are more evenly distributed.The statistics are broken down by scene type and object type, including descriptions and average target objects per description.

C. Computational efficiency

M3DRef-CLIP is compared with D3Net for GPU memory use and runtime under matched training conditions. Its added computational and memory overhead, including rendering, is only 10–20%.

  • C. Computational efficiency: D3Net and M3DRef-CLIP are compared using torch.cuda.max memory reserved, GPU memory, and time per epoch.
  • C. Computational efficiency: Table 14 reports the GPU memory usage and running-time comparison between D3Net and M3DRef-CLIP.
  • C. Computational efficiency: 10–20% overhead is incurred by M3DRef-CLIP for memory and computation, including all rendering.The comparison uses the same input, batch size, 60 epochs, machine, and NVIDIA RTX A5000 GPU.

D. Analysis of matching strategies

The analysis varies matching strategy and IoU thresholds to assess Multi3DRefer performance. Hungarian matching generally performs better, particularly with noisier proposals and a lower training threshold, while τpred = 0.1 is optimal.

  • D. Analysis of matching strategies: The study evaluates F1 at IoU 0.5 while varying Hungarian versus All matching, τtrain values of 0.25 versus 0.50, and τpred from 0.0 to 0.4.
  • D. Analysis of matching strategies: τpred = 0.1 gives all models their best performance across the evaluated prediction thresholds.
  • D. Analysis of matching strategies: Hungarian matching is effective when proposals are noisy because it guarantees optimal one-to-one matching.For D3Net, the two strategies show no noticeable difference, which the authors associate with a less noisy detector.

E. Qualitative results on Multi3DRefer

Qualitative comparisons cover zero-, single-, and multiple-target descriptions on Multi3DRefer. M3DRef-CLIP produces false positives in zero-target cases, more accurate boxes for single targets, and better small-object identification for multiple targets.

  • E. Qualitative results on Multi3DRefer: M3DRef-CLIP tends to predict false positives for Zero Target examples.The qualitative figure uses blue boxes for ground truth, green boxes for true positives, and red boxes for false positives.
  • E. Qualitative results on Multi3DRefer: M3DRef-CLIP has more accurate bounding boxes than D3Net for Single Target examples.
  • E. Qualitative results on Multi3DRefer: For Multiple Targets, M3DRef-CLIP identifies small objects accurately, while D3Net produces false detections of large objects.
  • E. Qualitative results on Multi3DRefer: Figure 10 presents scene-description examples containing Zero Target, Single Target, and Multiple Targets, with blue boxes indicating ground truth.
Loading 2309.05251v1…