PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-View Transformer for 3D Visual Grounding

Shijia Huang, Yilun Chen, Jiaya Jia, Liwei Wang

arXiv:2204.02174v1cs.CVcs.CL

TL;DR

3D visual grounding must remain reliable when viewpoint changes alter scene position encodings and language-object alignment. MVT projects scenes into a multi-view space, jointly aggregates positional information, and learns a representation independent of the starting view. It substantially outperforms prior methods on major benchmarks, including methods using extra 2D assistance.

  • Problem

    View-dependent grounding can fail when 3D scene viewpoints change, because viewpoint changes alter positional information needed to align language with objects.

  • Method

    MVT rotates a 3D scene into equally spaced views, jointly models and aggregates their positional information, and shares point-cloud features across views to reduce computation.

  • Results

    MVT outperforms all state-of-the-art methods, including by 11.2% on Nr3D and 7.1% on Sr3D over the best competitors.

  • Takeaways & Limitations

    Multi-view positional modeling produces a more robust multimodal representation that eliminates dependence on specific starting views for 3D visual grounding.

Abstract

from arXiv · show

The 3D visual grounding task aims to ground a natural language description to the targeted object in a 3D scene, which is usually represented in 3D point clouds. Previous works studied visual grounding under specific views. The vision-language correspondence learned by this way can easily fail once the view changes. In this paper, we propose a Multi-View Transformer (MVT) for 3D visual grounding. We project the 3D scene to a multi-view space, in which the position information of the 3D scene under different views are modeled simultaneously and aggregated together. The multi-view space enables the network to learn a more robust multi-modal representation for 3D visual grounding and eliminates the dependence on specific views. Extensive experiments show that our approach significantly outperforms all state-of-the-art methods. Specifically, on Nr3D and Sr3D datasets, our method outperforms the best competitor by 11.2% and 7.1% and even surpasses recent work with extra 2D assistance by 5.9% and 6.6%. Our code is available at https://github.com/sega-hsj/MVT-3DVG.

1. Introduction

3D visual grounding must align language-described positions with objects despite sparse point-cloud inputs and view-dependent spatial relationships. MVT addresses this by jointly modeling multiple views to learn a view-robust representation, achieving large benchmark gains.

  • Motivation: 3D visual grounding maps a natural-language description to its target object in a 3D scene.The task supports applications including vision-language navigation, intelligent agents, and autonomous vehicles.
  • Motivation: Existing methods mainly generate candidate objects and then select the best-matched one, adapting a two-stage scheme from 2D grounding.These approaches do not fully account for the intrinsic properties of 3D data.
  • View dependence: View changes alter 3D coordinate encodings, making it harder to represent objects and align them with language such as “left” or “right.”The same pillow can occupy different coordinate positions when the scene view changes from front to back.
  • View dependence: Specific-view training can fail after viewpoint changes, while augmented rotations alone did not improve performance because visual features remain misaligned with language-embedded positions.View constraints are often implicit, so robust grounding cannot depend on speakers supplying additional view guidance.
  • Proposed approach: MVT rotates a scene into equally spaced views, jointly models and aggregates their positional information, and reduces dependence on the starting view.It also decouples point-cloud feature extraction from multi-view positional encoding to share point-cloud features across views.
  • Results: 11.2% and 7.1%: MVT exceeds the best competitor on Nr3D and Sr3D, respectively, while surpassing a method with extra 2D assistance by 5.9% and 6.6%.Experiments cover Nr3D, Sr3D, and ScanRefer.

2. Related Work

Related work has used two-stage grounding and multi-view projections for 3D vision. MVT differs by projecting scene positional information into multi-view space to improve vision-language alignment.

  • 3D Visual Grounding: 3D visual grounding benchmarks and baselines use natural-language queries to locate objects in ScanNet 3D scenes.Recent methods commonly follow two stages: generating 3D object proposals and selecting the best match.
  • Multi-View Learning: Prior multi-view methods project 3D point clouds onto multiple 2D planes to enhance visual features for detection or object classification.Examples include bird’s-eye and front views in MV3D and many rendered images in MVCNN.
  • Multi-View Learning: MVT instead projects 3D scene positional information into multi-view space to learn view-robust representations for vision-language alignment.This distinguishes its multi-view design from approaches focused on visual-feature enhancement.

3. Method

MVT projects a 3D scene into equally rotated views, fuses object and language features within each view, and aggregates them into a representation independent of the starting view. It also reduces multi-view encoding cost by sharing point-cloud features and enhances encoders with language-guided object classification.

  • Multi-View 3D Visual Grounding: MVT extends a scene into N equally spaced rotated views to model position information across viewpoints.The view angles are θ^j_v = 2π(j−1)/N, with N typically set to 1, 2, 4, or 8.
  • Multi-View 3D Visual Grounding: A shared network extracts view-specific features, and an order-independent aggregation function produces a representation invariant to the starting view.This design projects the scene into a multi-view space and aggregates information across its views before grounding prediction.
  • Object Feature Encoding: Object encoding decouples shared point-cloud feature extraction from view-specific positional encoding, reducing the cost of processing multiple views.Point-cloud features are computed once and combined with rotated coordinates and box-size information for each view.
  • Multi-Modal Feature Fusion: For each view, a shared transformer decoder models object interactions with self-attention and fuses object features with BERT language features through cross-attention.BERT supplies sentence- and word-level features, while the decoder produces multi-modal features for objects under each view.
  • Multi-View Aggregation: The method averages fused features across views before applying fully connected layers to obtain grounding scores, while also evaluating alternative aggregation stages and functions.The default aggregation is performed after multi-modal fusion; alternatives include earlier aggregation and max-based functions.
  • Improving Object Encoder: Language-guided object classification uses category-label text features as auxiliary supervision to enhance the object encoder alongside the grounding and text losses.The total loss combines reference grounding, text classification, and object classification losses, with α set to 0.5 by default.

4. Experiments

Experiments on Nr3D, Sr3D, and ScanRefer show that MVT substantially improves 3D visual grounding, with multi-view modeling providing robustness across views. Ablations further examine component contributions, view counts, aggregation choices, and visualization outcomes.

  • 3D Visual Grounding Results: Nr3D accuracy rises from 43.9% to 55.1%, a +11.2% improvement over the best competitor under the same training setting.MVT also exceeds SAT, which uses extra 2D semantics, by 5.9 percentage points.
  • 3D Visual Grounding Results: On Sr3D, MVT exceeds TranRefer3D by 7.1% and SAT with extra 2D assistance by 6.6%.The method achieves these gains without additional 2D data.
  • 3D Visual Grounding Results: On ScanRefer, increasing the view number from 1 to 4 improves Acc@0.25 from 38.33% to 40.80% and Acc@0.5 from 31.12% to 33.26%.Inference time changes only slightly, from 261ms with one view to 264ms with four views on a TITAN X (Pascal).
  • Effectiveness of each Component: The transformer decoder improves Nr3D accuracy from 36.9% to 40.4%, while the full configuration reaches 55.1%.The baseline does not model relationships between objects; the decoder adds object-relationship modeling and vision-language fusion.
  • Effectiveness of Multi-View Modeling: Training with four views but testing with one raises overall accuracy from 46.2% to 51.6% and increases View-dep. accuracy by 8.1%.Four views provide a robust representation, whereas eight views can slightly degrade performance because of redundant views and lower training efficiency.
  • Multi-view aggregation: Aggregating after multi-modal feature fusion achieves the best performance, while averaging positional encodings performs poorly because it destroys position information.Average aggregation is more effective than max aggregation, and combining average and max performs similarly to average alone.

5. Conclusion

The paper proposes Multi-View Transformer (MVT) for 3D visual grounding, modeling scene position information across views to learn robust multimodal representations. Experiments and ablations support the model’s effectiveness.

  • MVT projects the 3D scene into a multi-view space that models position information under different views simultaneously.This design supports a more robust multimodal representation and eliminates dependence on specific views.
  • MVT outperforms all state-of-the-art methods by a large margin in extensive experiments.
  • Ablation studies show the effectiveness of all components in MVT.
Loading 2204.02174v1…