PaperlayerView exact revision on arXiv

Source-linked AI summary

ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language

Dave Zhenyu Chen, Angel X. Chang, Matthias Nießner

arXiv:1912.08830v3cs.CVcs.CLcs.LGeess.IV

TL;DR

3D object localization from natural-language descriptions remains limited by 2D methods that cannot capture objects’ full extent and spatial context. ScanRefer addresses this with an end-to-end 3D language-grounding method and a large dataset, outperforming a 2D-to-3D baseline on Acc@0.5IoU.

  • Problem

    Existing visual-grounding methods operate on 2D images, while 3D localization is needed to capture objects’ true extent and spatial context.

  • Method

    ScanRefer fuses point-cloud object proposals with sentence embeddings to match descriptions to objects and regress their 3D bounding boxes.

  • Results

    The end-to-end method significantly outperforms a 2D backprojection baseline, reported as 9.04 Acc@0.5IoU vs. 22.39 Acc@0.5IoU.

  • Takeaways & Limitations

    ScanRefer establishes a large-scale benchmark and method for studying natural-language object localization directly in 3D scenes.

Abstract

from arXiv · show

We introduce the task of 3D object localization in RGB-D scans using natural language descriptions. As input, we assume a point cloud of a scanned 3D scene along with a free-form description of a specified target object. To address this task, we propose ScanRefer, learning a fused descriptor from 3D object proposals and encoded sentence embeddings. This fused descriptor correlates language expressions with geometric features, enabling regression of the 3D bounding box of a target object. We also introduce the ScanRefer dataset, containing 51,583 descriptions of 11,046 objects from 800 ScanNet scenes. ScanRefer is the first large-scale effort to perform object localization via natural language expression directly in 3D.

1 Introduction

ScanRefer introduces natural-language object localization directly in 3D environments, predicting a target object's 3D bounding box from a description and point-cloud scene. It contributes the ScanRefer dataset and an end-to-end architecture that fuses language, 2D image, and 3D point-cloud features.

  • Motivation: 2D visual grounding methods localize objects in images but do not capture their true 3D extent or global physical context.These limitations matter for applications such as assistive robots and AR/VR agents.
  • Task: ScanRefer defines 3D object localization from natural-language descriptions by predicting a corresponding 3D bounding box for the best-described object in a scene.The input is a 3D point cloud together with a free-form description referring to the target object.
  • Dataset: 51,583 human-written free-form descriptions cover 11,046 objects in 3D scans through the ScanRefer dataset.The dataset facilitates research on the proposed 3D language-grounding task.
  • Method: The proposed neural architecture directly fuses 2D image and language features with 3D point-cloud features for localization.This fusion supports end-to-end prediction in 3D environments from language descriptions.
  • Result: 22.39 Acc@0.5IoU versus 9.04 Acc@0.5IoU demonstrates that the end-to-end method outperforms 2D visual grounding backprojected to 3D.The comparison is against the best 2D visual-grounding method that simply backprojects its predictions to 3D.

2 Related Work

Prior referring-expression work primarily grounds language in 2D images through bounding-box or mask prediction, often using proposal-and-ranking pipelines. ScanRefer extends this direction to free-form descriptions of objects in large-scale 3D RGB-D scans, addressing the lack of complete 3D context in existing datasets and methods.

  • Grounding Referring Expressions in Images: Visual grounding localizes a region described by a referring expression, with outputs including 2D bounding boxes and segmentation masks.The task is also known as referring expression comprehension or phrase localization.
  • Grounding Referring Expressions in Images: A common approach first proposes regions using an unsupervised or pretrained detector, then ranks them by similarity to the query.Single-stage methods also address referring-expression localization.
  • Limitations of Existing Work: Existing referring-expression methods operate on 2D image datasets, while an RGB-D dataset lacks complete 3D context beyond a single image.Another embodied-setting approach projects semantic annotations from 3D into 2D image bounding boxes.
  • ScanRefer: ScanRefer introduces a large-scale effort coupling free-form descriptions with objects in 3D scans.Its task takes a 3D scene point cloud and an object description as input and predicts the object bounding box.
  • 3D Referring Expressions: Earlier 3D-related work addressed primitive-shape arrangements or linked noun phrases in single-view RGB-D scene descriptions to 3D object bounding boxes.These settings differ from large 3D RGB-D reconstructions typical of semantic 3D scene understanding.

3 Task

The paper defines 3D object localization from natural-language descriptions in point-cloud scenes. Given a scene and free-form text, the task is to predict the matching object's 3D bounding box.

  • 3 Task: The task takes a 3D scene point cloud with per-point colors and normals plus free-form text describing a target object.The scene is represented as a point cloud with additional features for each point.
  • 3 Task: The goal is to predict the 3D bounding box of the object matching the input description.This defines object localization directly in the 3D scene.

4 Dataset

ScanRefer is built on ScanNet and provides broad object coverage through five descriptions per object, collected and verified via a two-stage annotation process. The resulting dataset contains 51,583 descriptions from 800 scenes and captures diverse indoor-object language.

  • ScanRefer is based on ScanNet, comprising 1,613 RGB-D scans from 806 unique indoor environments.
  • The dataset provides five descriptions for each object in each scene to pursue complete coverage of reconstructed objects.
  • Descriptions are collected through a two-stage AMT pipeline consisting of description collection and verification.
  • Verifiers filter descriptions that identify the wrong or multiple objects, remove 2,823 invalid descriptions, and correct writing issues in 2,129 descriptions.
  • 51,583 descriptions were collected for 800 ScanNet scenes, averaging 13.81 objects, 64.48 descriptions per scene, and 4.67 descriptions per object after filtering.
  • The descriptions cover over 250 types of common indoor objects and exhibit complex, diverse linguistic phenomena.

5 Method

ScanRefer uses a four-stage pipeline that detects and encodes 3D object proposals and language, fuses them, and localizes the proposal most likely described. It processes sampled RGB-D point-cloud features and pretrained GloVe word embeddings, using objectness masking, confidence scoring, and end-to-end training losses.

  • Architecture: The architecture comprises detection and encoding followed by fusion and localization, selecting the highest-confidence object proposal as the final output.Detection and encoding produce object proposals and a language embedding; fusion masks invalid proposals before localization.
  • Detection and Encoding: PointNet++ features are clustered into object proposals with objectness masks and axis-aligned bounding boxes, while a GRU encodes the description into a language embedding.Each proposal includes box center, lengths, and semantic predictions.
  • Input Representation: The input point cloud contains 40,000 sampled vertices with 3D coordinates and additional features such as colors, normals, and height from the ground.Coordinates provide geometric information, while the additional features augment the point representation.
  • Fusion and Localization: Proposal point features are concatenated with the language embedding, masked by objectness, fused by a multilayer perceptron, and converted into localization confidences with softmax.The point-cluster feature dimension is 128 and the language embedding dimension is 256 before fusion.
  • Training: Training combines end-to-end detection with localization cross-entropy, object detection losses, and an auxiliary language-to-object classification loss over 18 ScanNet benchmark classes excluding Floor and Wall.The localization target marks the proposal with the highest IoU to the ground-truth box as positive.

6 Experiments

Experiments evaluate ScanRefer on disjoint ScanNet splits using IoU-thresholded localization accuracy, baseline comparisons, subset analyses, and ablations. ScanRefer outperforms the baselines, while results show benefits from end-to-end training and language-based category supervision.

  • Experimental Setup: The data use disjoint train, validation, and test scenes with 36,665, 9,508, and 5,410 samples, respectively.Experiments use the validation split except where noted; the hidden test set is reserved for the ScanRefer benchmark.
  • Experimental Setup: Performance is measured by Acc@kIoU at IoU thresholds 0.25 and 0.5, reporting the percentage of predictions exceeding each threshold.The evaluation also compares unique and multiple subsets, where multiple contains same-category ambiguities.
  • Main Results: ScanRefer outperforms all baseline models by a significant margin on localization accuracy.Table 4 reports results for IoU thresholds 0.25 and 0.5 and separately evaluates unique and multiple subsets.
  • Subset Analysis: Category-only information performs relatively well for unique objects but struggles with multiple same-category objects, demonstrating the need for descriptive language to disambiguate targets.The comparison also indicates that 3D object detection remains a limitation in the unique case.
  • Ablation Study: End-to-end joint training of the detection backbone and localization module performs better than separate VoteNet[49] + GRU training, although the gap to VoteNetBest leaves room for improvement.Inference averages five differently seeded subsamplings of seed points and vote points.
  • Ablation Study: Adding a language-to-object classifier improves performance, indicating that predicting the target category from the description provides useful supervision.The ablation compares architectures with and without the +lobjcls classifier and also examines geometry and multiview image features.

7 Conclusion

The paper introduces 3D object localization from natural-language descriptions in point clouds and presents the ScanRefer dataset. It proposes an end-to-end method that generates point clusters and matches them with input-sentence embeddings.

  • Task: ScanRefer defines the task of localizing a target object in a 3D point cloud from a natural-language description.The description is free-form and specifies the target object.
  • Dataset: 51,583 unique descriptions cover 11,046 objects across 800 ScanNet scenes in the ScanRefer dataset.These figures describe the collected dataset used for the task.
  • Method: The proposed end-to-end method first generates point clusters of interest and then matches them to embeddings of the input sentence.This pipeline uses candidate point clusters and language embeddings to localize the referenced object.

A Dataset · A.1 Statistics · A.2 Collection Details

ScanRefer annotates 11,046 objects across 265 ScanNet categories and characterizes both category distributions and localization difficulty. Its descriptions were collected and verified through a large-scale, multi-stage AMT and student annotation process.

  • A.1 Statistics: 11,046 objects from 265 ScanNet categories are annotated in ScanRefer, with finer-grained categories aggregated into 17 coarse categories.The dataset covers varied indoor furniture with different sizes, colors, materials, and locations.
  • A.1 Statistics: Scenes containing beds have an average of 1.22 beds and a maximum of 3, illustrating per-category object-count variation.These statistics are computed for scenes in which the corresponding category appears.
  • A.1 Statistics: The “unique” subset contains one object of the target category, whereas the “multiple” subset requires distinguishing among several same-category objects.Unique cases can be localized from semantic class alone under perfect detection, while multiple cases require object-level disambiguation.
  • A.1 Statistics: The “multiple” subset is more difficult because it contains more same-category objects than the “unique” subset, increasing target-localization ambiguity.Table 7 reports average disambiguation counts for these subsets under ground-truth bounding boxes.
  • A.2 Collection Details: 1,929 AMT workers contributed to data collection and verification over one month, totaling approximately 4,984 man hours.The reported labor combines description collection and verification.
  • A.2 Collection Details: AMT workers in the United States, United Kingdom, Canada, and Australia completed batches of five description tasks within two hours.A web-based application was deployed on reconstructed RGB-D scans, with English-language quality encouraged through worker restrictions.
  • A.2 Collection Details: Descriptions were manually screened, then verified by trained students who selected matching objects and corrected spelling or wording errors.Verification used a WebGL-based application and required descriptions to be discriminative enough to identify the intended object.

B Additional Implementation Details · B.1 Fusion Module · C Additional quantitative analysis

The supplementary sections detail ScanRefer’s multimodal fusion module, annotation and verification interfaces, and class-wise object-detection analysis. The fusion process combines aggregated point clusters, language embeddings, and objectness masks into final fused features.

  • B Additional Implementation Details: The annotation interface asks annotators to describe batches of five target objects while adjusting viewpoints and selecting camera-view images.Amazon Mechanical Turk workers receive instructions before providing object descriptions.
  • B Additional Implementation Details: The verification interface asks verifiers to select objects matching provided descriptions, excluding descriptions that match multiple scene objects.Trained verifiers review instructions before beginning verification.
  • B.1 Fusion Module: Aggregated point clusters and expanded language features are concatenated to form raw fused features.The point clusters are represented as C = c_i ∈ R^M×128.
  • B.1 Fusion Module: Predicted objectness masks remove invalid raw fused features before further processing.The masking uses predicted objectness masks within the localization pipeline.
  • C Additional quantitative analysis: Object detection is analyzed with mean average precision at IoU 0.5 across 18 classes for VoteNet and multiple feature configurations.The configurations vary xyz, RGB, normals, multiview, and language object-class features; the passage specifically discusses comparisons involving point normals.
  • B.1 Fusion Module: A multi-layer perceptron maps the filtered feature maps into final fused multimodal point features C′ ∈ R^M×128.These final fused features are the output of the fusion module.

C.1 Object Detection Results · C.2 Training and Evaluation Variance

The object detector achieves 33% mAP at IoU 0.5, performing better on large separated objects than on flat, ambiguous-boundary, or small objects. Random point sampling introduces measurable training and evaluation variance, motivating averaged evaluation runs.

  • C.1 Object Detection Results: The detector is evaluated using mean average precision at an IoU threshold of 0.5 across object categories, excluding structural objects such as Floor.
  • C.1 Object Detection Results: 33% mAP is achieved by the best 3D object detector, indicating substantial room for improving 3D object detection.
  • C.1 Object Detection Results: The detector robustly identifies large objects away from walls but is less reliable for flat, unclear-boundary, and smaller objects.Examples include beds, chairs, sofas, toilets, bathtubs, pictures, windows, doors, cabinets, shelving, sinks, and others.
  • C.1 Object Detection Results: Adding point normals improves detection over corresponding models without normals, while high-level multi-view color features outperform raw RGB values.
  • C.2 Training and Evaluation Variance: Random sampling of 40,000 points in the VoteNet backbone produces a training-run stddev of 0.30 and an evaluation-run stddev of 0.37.
  • C.2 Training and Evaluation Variance: For more reliable results, VoteNet-based evaluation averages five runs using different random seeds.

C.3 Additional Ablation Study

The ablation study tests whether ScanRefer benefits from full natural-language descriptions rather than only semantic labels or object names. Richer descriptions improve localization, while semantic labels help when a category is unique but struggle when multiple objects share a broad category.

  • Input Length Ablation: The study compares full descriptions with semantic labels and object names as language inputs to measure how input information affects localization.The comparison evaluates whether the pipeline learns from complete descriptions or only object identity information.
  • Input Length Ablation: Semantic labels help in unique scenarios with only one object of a category but underperform in multiple-object scenarios requiring finer distinctions.Broad categories can group different targets together, whereas descriptions provide additional distinguishing information.

D Additional Qualitative Analysis

Additional qualitative examples show that ScanRefer generally localizes described objects more accurately than the compared VoteNet-based baselines, including scenes with multiple same-category instances. Failures arise when detector accuracy is limited or descriptions require difficult spatial and counting-based disambiguation.

  • Qualitative comparison: Pretrained VoteNet detection provides reasonable object bounding boxes but performs slightly worse than the proposed method.The comparison includes VoteNet [49]+GRU and VoteNetBest.
  • Multiple scenarios: The method correctly localizes some targets even when multiple same-category objects appear in the scene.Reported successful examples include Fig. 18 rows 1 and 5 and Fig. 19 rows 1–3 and 5–6.
  • Unique scenarios: In unique scenarios, the method localizes target objects in 3D indoor scenes using free-form descriptions.These scenarios contain only one object from a given category.
  • Limitations: Failures in multiple-object scenes result from limited detector accuracy and difficulty disambiguating instances or handling spatial relations.The method also fails on descriptions involving phrases such as “leftmost” or “rightmost” and counting-based references such as “the second one from the left.”
Loading 1912.08830v3…