PaperlayerView exact revision on arXiv

Source-linked AI summary

SAT: 2D Semantics Assisted Training for 3D Visual Grounding

Zhengyuan Yang, Songyang Zhang, Liwei Wang, Jiebo Luo

arXiv:2105.11450v2cs.CV

TL;DR

3D visual grounding is difficult because sparse, noisy point clouds provide weaker semantics than 2D images. SAT uses 2D object semantics during training to improve 3D scene-language representation learning while requiring no extra 2D inputs at inference. It improves grounding across Nr3D, Sr3D, and ScanRef and reports a new state of the art on multiple datasets.

  • Problem

    Sparse, noisy 3D point clouds make direct joint representation learning between 3D scenes and language challenging.

  • Method

    SAT uses object labels, image features, and 2D geometric features during training to assist 3D visual grounding without requiring 2D inputs during inference.

  • Results

    SAT achieves new state-of-the-art performance on multiple datasets and improves the identical-input non-SAT baseline from 37.7% to 49.2% on Nr3D.

  • Takeaways & Limitations

    Using 2D semantics during training helps SAT learn better 3D point-cloud object representations for visual grounding.

  • Takeaways & Limitations

    The study evaluates SAT within a two-stage grounding framework and leaves alternative frameworks for future work.

Abstract

from arXiv · show

3D visual grounding aims at grounding a natural language description about a 3D scene, usually represented in the form of 3D point clouds, to the targeted object region. Point clouds are sparse, noisy, and contain limited semantic information compared with 2D images. These inherent limitations make the 3D visual grounding problem more challenging. In this study, we propose 2D Semantics Assisted Training (SAT) that utilizes 2D image semantics in the training stage to ease point-cloud-language joint representation learning and assist 3D visual grounding. The main idea is to learn auxiliary alignments between rich, clean 2D object representations and the corresponding objects or mentioned entities in 3D scenes. SAT takes 2D object semantics, i.e., object label, image feature, and 2D geometric feature, as the extra input in training but does not require such inputs during inference. By effectively utilizing 2D semantics in training, our approach boosts the accuracy on the Nr3D dataset from 37.7% to 49.2%, which significantly surpasses the non-SAT baseline with the identical network architecture and inference input. Our approach outperforms the state of the art by large margins on multiple 3D visual grounding datasets, i.e., +10.4% absolute accuracy on Nr3D, +9.9% on Sr3D, and +5.6% on ScanRef.

1. Introduction

3D visual grounding maps language descriptions to referred regions in 3D scenes, but sparse and noisy point clouds make direct joint representation learning difficult. SAT addresses this challenge by using rich 2D semantics during training without requiring extra 2D inputs at inference.

  • 3D visual grounding maps a natural language description to the referred region of a 3D scene, typically represented by a 3D bounding box.
  • Directly maximizing positive object-query similarity is less effective because 3D point clouds are sparse and noisy.
  • SAT uses 2D object labels, image features, and geometric features to assist 3D scene-language representation learning during training.
  • SAT avoids requiring extra 2D inputs during inference, unlike approaches that use 2D semantics as visual inputs in both training and inference.
  • +10.4% absolute accuracy on Nr3D, +9.9% on Sr3D, and +5.6% on ScanRef are reported improvements from SAT.

2. Related Work

Prior 3D grounding work commonly uses two-stage object-proposal frameworks and has incorporated 2D semantics as extra inputs. SAT instead studies 2D semantics used only during training to avoid inference-time data and preprocessing requirements.

  • Earlier 3D grounding studies use two-stage frameworks that generate multiple 3D object proposals before grounding language descriptions.
  • Prior 3D detection and segmentation methods assist 3D tasks by projecting 2D detections into 3D space or concatenating image features with 3D points.
  • These approaches require extra 2D information during both training and inference, which can limit scenarios where such data or preprocessing is unavailable.
  • SAT uses 2D semantics only during training for 3D visual grounding and leaves alternative grounding frameworks for future study.

3. 3D Grounding Transformer

The 3D grounding transformer represents point-cloud proposals, language, and training-time 2D semantics as feature sequences, fuses them with transformer layers, and selects the proposal with the highest grounding score.

  • 3.1. 3D visual grounding inputs: The task input is an RGB-XYZ point-cloud scene and a natural-language query, with SAT adding 2D semantics only during training.
  • 3.2. Embedding for all modalities: Each 3D object proposal is encoded into a d-dimensional embedding from its point-cloud segment and normalized geometric information.
  • 3.2. Embedding for all modalities: For each proposal, 2D semantics combine region-of-interest visual features, a one-hot object-class vector, and 2D geometric features.
  • 3.2. Embedding for all modalities: A corresponding 2D semantic vector is sampled from multiple projected video frames during each training epoch.
  • 3.3. Fusion and grounding module: Transformer layers fuse query words, 3D proposals, and training-time 2D semantics into modality-specific output features.
  • 3.3. Fusion and grounding module: Two fully connected layers produce grounding scores for the fused 3D proposal features, and the highest-scoring proposal becomes the prediction.

4. 2D Semantics Assisted Training (SAT)

SAT adds auxiliary alignments between 2D semantics and 3D scenes or language queries during training, while preserving inference without 2D inputs. Its training framework combines grounding, object-correspondence, and classification objectives with attention masking that prevents 2D information leakage.

  • SAT learns auxiliary alignments between 2D semantics and 3D scenes or language queries to assist 3D visual grounding.
  • The visual grounding objective uses softmax loss over proposal grounding scores, labeling the highest-IoU proposal as positive.
  • During inference, SAT removes 2D inputs; masking prevents training-time dependence on unavailable image semantics, while unmasked alternatives can cause catastrophic accuracy drops.
  • The object correspondence objective is a triplet loss that aligns 3D and 2D object proposals using hard negatives within each scene.
  • The training loss combines visual grounding, object correspondence, and classification losses, with default weights of 10 for correspondence and 0.5 for classification.
  • SAT encodes fused 2D semantic features through a multimodal transformer whose attention mask blocks 2D inputs from directly informing query and 3D proposal features.

5. Experiments

Experiments show that SAT improves 3D visual grounding across Nr3D, Sr3D, and ScanRef while removing 2D inputs during inference. Ablations indicate that context-aware attention and combining visual, semantic, and geometric 2D features are important to this performance.

  • Evaluation setup: Ground-truth proposals are evaluated with grounding accuracy, while ScanRef detector-generated proposals additionally use Acc@0.25 and Acc@0.50.Accuracy measures selecting the referred object among M proposals; Acc@kIoU measures predicted-box overlap above the specified IoU threshold.
  • 3D visual grounding results: With identical architecture and inference inputs, SAT raises Nr3D accuracy from 37.7% to 49.2%.This is an 11.5% improvement over the non-SAT baseline.
  • 3D visual grounding results: SAT improves Sr3D accuracy from 47.4% to 57.9% and ScanRef accuracy from 48.2% to 53.8%.Extra training data further raises Sr3D to 60.7% and ScanRef to 57.0% with Nr3D data or 56.5% with Sr3D+ data.
  • 3D visual grounding results: On detector-generated ScanRef proposals, SAT achieves Acc@0.25 of 44.54% and Acc@0.50 of 30.14%.The models are evaluated after matching each predicted 3D proposal with the highest-IoU ground-truth 2D semantics object.
  • Ablation studies: Removing 2D-input masking causes accuracy to drop to 33.9% without 2D inference inputs, whereas the proposed mask reaches 49.2%.Encoding 2D semantics without scene context reaches 43.9%, below the proposed attention mask.
  • Ablation studies: Combining image, semantic, and geometric 2D features produces the best ablation accuracy of 49.2%, compared with 37.7% for non-SAT.Each tested 2D-semantics configuration improves over the non-SAT baseline.

6. How does SAT help?

SAT improves 3D visual grounding by learning better 3D object representations with 2D semantics, benefiting varied query conditions while retaining limitations on view understanding and ambiguous categories.

  • Representation quality: SAT learns better 3D object representations with 2D semantics, improving grounding across target classes, distractor counts, query lengths, and query types.The authors support this interpretation through performance breakdowns and linear probing results.
  • Representation quality: 60.1% linear probing accuracy versus 35.7% for the non-SAT baseline on Nr3D indicates stronger learned 3D object representations.Linear probing evaluates object classification from frozen proposal features.
  • Spatial relationship referring: 8.4% improvement, from 48.4% to 56.8%, occurs on Nr3D spatial queries when extra Sr3D+ training data is used, compared with 3.9% on remaining samples.The spatial-query subset contains 76.7% of Nr3D samples.
  • 2D input requirements: SAT performs comparably to methods using 2D inputs during inference, with 49.2% overall accuracy versus 50.3% for both 2D-input variants.This comparison uses the same network architecture and distinguishes training-only from training-and-inference 2D inputs.
  • Qualitative insights: SAT corrects non-SAT errors in object recognition, relationship modeling, and color or shape understanding, but still fails on view understanding and ambiguous object categories.Examples include confusing desks with beds, and office chairs with chairs.

7. Conclusion

The conclusion presents SAT as a training-only use of 2D semantics for 3D visual grounding. It reports improved accuracy, state-of-the-art performance across multiple datasets, and better 3D point-cloud object representations.

  • Conclusion: SAT uses 2D semantics during training to assist 3D visual grounding and ease joint representation learning between 3D scenes and language queries.The method does not require 2D inputs during inference, as established in the paper’s method description.
  • Conclusion: SAT beats the non-SAT baseline by 11.5% in absolute accuracy with identical network and inference inputs.
  • Conclusion: SAT achieves state-of-the-art results on multiple datasets and learns better 3D point-cloud object representations that help 3D visual grounding.

(Supplementary Material)

The supplementary material adds ablations, extends SAT to detector-generated 3D proposals, and examines how proposal quality affects grounding accuracy.

  • Supplementary Material: The supplementary material presents additional ablation studies and detailed result analyses.
  • Supplementary Material: It extends SAT to detector-generated 3D proposals and discusses how 3D proposal quality influences grounding accuracy.

A.1. Ablation studies

The ablation study separates SAT’s auxiliary objectives and shows that their combination produces the strongest Nr3D accuracy.

  • Training objectives: 38.0% accuracy is obtained by the baseline using only the main 3D grounding and classification losses.Although 2D semantics are input during training, they do not affect the main model without attention or 2D-related auxiliary losses.
  • Training objectives: 38.5% accuracy results from adding the 2D grounding loss, while 44.9% results from adding the object correspondence loss.
  • Training objectives: 49.2% accuracy is achieved when SAT jointly applies the 2D grounding and object correspondence auxiliary objectives.Classification loss also improves both the baseline and final SAT model.

A.2. Performance breakdown

SAT shows consistent improvements across Nr3D subsets with varying distractors, query lengths, target classes, and spatial language. The analysis also examines how extra training data affects queries containing spatial relationships.

  • Numbers of distractors: SAT’s advantage over the non-SAT baseline remains consistent across Nr3D subsets with different numbers of distractors.
  • SAT obtains consistent improvements across Nr3D subsets divided by target object class, query characteristics, and spatial relationships.
  • Numbers of query words: Longer queries are generally more challenging, and grounding accuracy decreases as query length increases.
  • Spatial relationships: 76.7% of Nr3D queries contain at least one spatial relationship keyword.
  • Spatial relationships: 8.4% improvement on “SAT-Nr3D” is reported for queries with spatial keywords after adding extra Sr3D+ training data.
  • SAT is compatible with detector-generated proposals in addition to the ground-truth proposal setting.

B.1. Method

The detector-generated-proposal extension pairs proposals with cached 2D semantics during training and excludes poorly aligned pairs from the correspondence loss. Online IoU matching preserves end-to-end optimization.

  • Cached 2D semantics avoid projecting detector-generated 3D proposals during every training iteration.
  • 3D proposals are paired online with 2D semantics by selecting the pair with maximum 3D IoU.
  • Pairs with IoU below 0.5 are excluded from the object correspondence loss.
  • Online IoU computation supports end-to-end optimization of the entire framework.

B.2. Experiment results

SAT improves 3D visual grounding with detector-generated and ground-truth proposals, outperforming the non-SAT baseline and prior state-of-the-art methods. Proposal quality remains an important factor, while the paper focuses on point-cloud-language representation learning.

  • The ScanRef table separates methods without extra inference inputs from methods using 2D semantics during both training and inference.
  • 37.64% and 25.16% versus 31.81% and 21.34% shows SAT outperforming the non-SAT baseline on ScanRef’s “multiple” subset.
  • 37.64% and 25.16% versus 28.83% and 22.92% shows SAT outperforming InstanceRefer on the ScanRef “multiple” subset.
  • 3D proposal quality: 54.33% oracle Acc@0.5 with VoteNet indicates that detector-generated proposal quality constrains final grounding accuracy.
  • SAT-GT achieves 66.01% Acc@0.5 on ScanRef, versus 40.06% for ScanRef-GT and 55.37% for InstanceRef-GT.
  • +10.4% absolute accuracy on Nr3D and +9.9% on Sr3D are reported for SAT over the state of the art.
Loading 2105.11450v2…