PaperlayerView exact revision on arXiv

Source-linked AI summary

EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding

Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, Jian Zhang

arXiv:2209.14941v3cs.CVcs.RO

TL;DR

Existing 3D visual grounding methods can couple words into sentence-level features or focus on object names, leaving other semantic components less represented. EDA decouples text and densely aligns its components with point-cloud objects, and reports state-of-the-art results on established datasets alongside leadership on its name-free task. The paper also notes limitations in its component modules, especially text decoupling for lengthy descriptions.

  • Problem

    Existing methods may couple sentence-level features or focus on object names rather than retaining the description's word-level semantic components.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    EDA parses descriptions into semantic components and densely aligns them with visual features using position- and semantic-alignment losses.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3.2 · p. 5

  • Results

    EDA achieves state-of-the-art performance on ScanRefer and SR3D/NR3D and leads the proposed grounding-without-object-names task using the same model without retraining.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Grounding without object names provides a challenging evaluation of whether a model can locate objects from attributes and relations rather than their names.

    Source: §1 · p. 2

    Performance is limited by component modules, and text decoupling may fail on lengthy descriptions, degrading performance.

    Source: §5 · p. 8

Abstract

from arXiv · show

3D visual grounding aims to find the object within point clouds mentioned by free-form natural language descriptions with rich semantic cues. However, existing methods either extract the sentence-level features coupling all words or focus more on object names, which would lose the word-level information or neglect other attributes. To alleviate these issues, we present EDA that Explicitly Decouples the textual attributes in a sentence and conducts Dense Alignment between such fine-grained language and point cloud objects. Specifically, we first propose a text decoupling module to produce textual features for every semantic component. Then, we design two losses to supervise the dense matching between two modalities: position alignment loss and semantic alignment loss. On top of that, we further introduce a new visual grounding task, locating objects without object names, which can thoroughly evaluate the model's dense alignment capacity. Through experiments, we achieve state-of-the-art performance on two widely-adopted 3D visual grounding datasets, ScanRefer and SR3D/NR3D, and obtain absolute leadership on our newly-proposed task. The source code is available at https://github.com/yanmin-wu/EDA.

1. Introduction

3D visual grounding finds a described object in a point-cloud scene, but existing approaches can couple unrelated words or over-rely on object names. EDA decouples semantic components, aligns them densely with objects, and introduces a name-free grounding task to assess this capability.

1. Introduction

  • 3D visual grounding locates a queried object in a point-cloud scene, where sparse and incomplete point clouds and diverse descriptions make the task challenging.Descriptions may combine appearance, categories, and spatial relations.Source: §1 · p. 2
  • Existing methods can overemphasize object names, weakening attention to other properties, while coupled sentence features can mix target descriptions with words about auxiliary objects.Descriptions may include multiple objects and attributes even though the model must identify only the main object.Source: §1 · p. 2
  • EDA parses descriptions into semantic components and densely aligns relevant components with point-cloud objects, selecting the object most similar to those components.Components include the main object, pronoun, attributes, relations, and auxiliary-object words.Source: §1 · p. 2
  • Grounding without object names replaces the name with “object,” requiring localization from other attributes and relationships and testing for category-name shortcuts.The authors state that this setting reflects common name-free expressions and evaluates the model's alignment capacity.Source: §1 · p. 2
  • EDA achieves state-of-the-art performance on ScanRefer and SR3D/NR3D, and leads the name-free task using the same model without retraining.The paper proposes text decoupling and two dense-alignment losses, and introduces grounding without object names as a robustness evaluation.Source: §1 · p. 2

2. Related Work

Related work situates point-cloud visual grounding within 3D vision-and-language tasks, then reviews alternative grounding architectures and their constraints. EDA's dense alignment differs from approaches that match only object names or noun phrases.

2.1. 3D Vision and Language

  • 3D visual grounding pairs a language description with a 3D scene to locate a target object, whereas dense captioning describes each scene object in text.The paper focuses on point-cloud grounding as a foundational technology for embodied AI tasks.Source: §2.1 · pp. 2–3

2.2. 3D Visual Grounding

  • Mainstream two-stage grounding extracts language and candidate-object features independently before fusion and matching, while single-stage methods use text to modulate detection or feature extraction.Two-stage approaches risk missing objects overlooked during initial detection; the surveyed single-stage methods include text-related heatmaps and guided keypoint selection.Source: §2.2 · p. 3
  • Figure 2 depicts the query text, dependency-tree analysis, and its decoupling into five components.The five components are the main object, auxiliary object, attributes, pronoun, and relationship.Source: Figure 2 · p. 3
  • BUTD-DETR-style sparse alignment matches object-name words to visual features, whereas EDA aligns all object-related decoupled semantic components.The paper states that EDA's text decoupling identifies the target name through grammatical analysis rather than requiring ground-truth name labels.Source: §2.2 · p. 3

3. Proposed Method

EDA decouples descriptions into semantic components, extracts cross-modal object and text features, and trains their dense alignment with position- and semantic-alignment losses. At inference, it combines component-to-object similarities to select a grounding target.

3.1. Text Decoupling

  • Text decoupling parses each description into main-object, auxiliary-object, attribute, pronoun, and relationship components, with position labels marking each component’s words.The labels are masks: associated word positions are 1 and all others 0; decoupled features retain global sentence information through Transformer attention.Source: §3.1 · pp. 3–4

3.2. Multimodal Feature Extraction

  • The model uses Transformer-based cross-modal encoders and a decoder to produce proposal features, object position labels, projected object features, and optional box predictions.RoBERTa and PointNet++ encode text and point clouds; the GroupFree detector is optional and its box tokens assist box regression.Source: §3.2 · pp. 4–5

3.3.1 Dense Position Aligned Loss

  • Dense position alignment trains predicted object-to-text-position distributions against targets that combine the main object’s component labels, while distinguishing auxiliary and other objects.The KL-divergence loss aligns these distributions; unlike sparse alignment to the object-name position alone, it covers multiple text components.Source: §3.3.1 · p. 5

3.3.2 Dense Semantic Aligned Loss

  • Semantic alignment uses contrastive learning to match object features with multiple positive text components and averages object-side and text-side losses.The auxiliary-object text feature receives negative similarity weight 2, while the other terms receive weight 1; training also includes box regression loss.Source: §3.3.2 · p. 5

3.4. Explicit Inference

  • At inference, EDA sums similarities to main-object, attribute, pronoun, and relationship components, subtracts auxiliary-object similarity, and selects the highest-scoring candidate.When auxiliary objects were supervised during training, their target can be inferred using the attribute-component score alone.Source: §3.4 · p. 5

4. Experiments

Experiments compare EDA with other methods on regular 3D visual grounding, then assess grounding without object names and ablate losses and text components.

4. Experiments

  • Experiments compare methods on regular visual grounding and introduce a separate comparison and analysis for grounding without object names.Additional implementation details and experiments are in the supplementary material.Source: §4 · p. 6

4.1.3 Ablation studies

  • The semantic-aligned loss is marginally better alone, while combining it with the position-aligned loss achieves the best accuracy through complementary performance.The semantic loss contrasts matched and mismatched features; the position loss considers object-related components.Source: §4.1.3 · p. 6, Table 3 · p. 7
  • Dense-aligned sub-methods outperform main-object-only alignment, and integrating all text components reaches peak performance.Attributes support appearance cues, pronouns carry contextual information, relationships encode spatial cues, and auxiliary-object similarity is subtracted during inference.Source: §4.1.3 · pp. 6–7, Table 4 · p. 7

4.1.2 Comparison to the state of the art

  • 4.2% and 3.7% overall improvements to 54.59% and 42.26% establish state-of-the-art performance on ScanRefer.EDA also outperforms 2D-assisted methods using sparse 3D point-cloud features.Source: §4.1.2 · p. 6, Table 1 · p. 7
  • 68.1% and 52.1% are EDA’s best reported performance on SR3D/NR3D, with 3D-only data despite NR3D’s more detailed descriptions.Comparable NR3D methods rely on additional 2D images for training.Source: §4.1.2 · p. 6, Table 2 · p. 7

4.2.1 Experiment settings

  • The VG-w/o-ON setting replaces the target’s name with “object”; 9,253 samples were annotated and 255 ambiguous samples discarded.The set divides into attributes-only, relationships-only, both, and other subsets, approximately 15%, 20%, 63%, and 2%.Source: §4.2.1 · pp. 7–8

4.2.2 Result and analysis

  • 26.5% and 21.2% overall performance give EDA an absolute lead, over 10% above other methods on grounding without object names.Performance for all methods is significantly lower than on regular visual grounding.Source: Table 5 · p. 8, §4.2.2 · p. 8

5. Conclusion

EDA decouples text into semantic components and densely aligns them with visual features using position- and semantic-aligned losses. The authors report strong experimental performance, propose grounding without object names, and identify lengthy-text decoupling as a limitation.

5. Conclusion

  • EDA decouples text into semantic components and uses position- and semantic-aligned losses for dense visual-text alignment.The authors say this supports fine-grained feature fusion while avoiding imbalance and ambiguity in existing methods.Source: §5 · p. 8
  • The authors report that extensive experiments and ablations demonstrate EDA’s superiority.Source: §5 · p. 8
  • Grounding without object names is proposed as a challenging 3D visual-grounding subtask for comprehensive robustness evaluation.Source: §5 · p. 8
  • Performance is limited by modules including PointNet++, RoBERTa, and text parsing, with decoupling potentially failing on lengthy text.Source: §5 · p. 8

Supplementary Material for EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding

The supplementary material accompanies EDA and identifies its authors, affiliations, and supporting contents. It covers implementation and training details, additional experiments, and visualizations with qualitative analysis.

Supplementary Material

A. Implementation details

The implementation combines text parsing, pretrained visual and language encoders, and a transformer encoder-decoder with alignment and detection losses. Dataset-specific loss weights and training schedules are also reported.

A. Implementation details

  • Text length is capped at 256, and although sentences need not yield all five semantic components, the main-object component is required.Source: A. Implementation details · p. 13
  • PointNet++ tokenizes 1,024 seed points, RoBERTa tokenizes up to 256 text positions, and a 3-layer encoder with a 6-layer decoder produces candidate-object features.The reported feature dimension is 288; the setup uses 132 detection boxes and 256 candidate objects.Source: A. Implementation details · p. 13
  • The position-aligned loss weights the main-object component at 0.6 and other listed components at 0.1 or 0.2; semantic-aligned feature weights are 1.0, 0.2, 0.2, and 0.1.The auxiliary-object negative feature has weight 2, while the other negative features have weight 1.Source: A. Implementation details · p. 13
  • The total loss combines alignment, box, IoU, and point-sampling losses, with α=1 for SR3D/NR3D and α=0.5 for ScanRefer.The authors assign ScanRefer higher detection-loss weights because it requires bounding-box detection, unlike SR3D/NR3D, which provides candidate boxes.Source: A. Implementation details · p. 13
  • Training used batch size 12 on four 24-GB RTX 3090 GPUs, with dataset-specific learning rates and epoch counts.The reported best models appear around epoch 60 for ScanRefer, 45 epochs for SR3D, and 180 epochs for NR3D.Source: A. Implementation details · p. 13

B.1. Regular 3D Visual Grounding

The supplementary regular-grounding experiments examine ScanRefer with ground-truth boxes, dense-component and loss-weight ablations, and comparisons involving BUTD-DETR and text alignment. They also describe evaluation coverage and limitations of the baseline’s label-based procedure.

B.1. Regular 3D Visual Grounding

  • BUTD-DETR’s additional ground-truth target-class input makes inference comparisons unfair, and category synonyms can cause its position-label search to fail.The authors re-evaluate it using position labels from EDA’s text-decoupling module.Source: §B.1 · pp. 13–14
  • BUTD-DETR excludes 800 unnamed-object samples from NR3D validation, representing about 5% that do not participate in evaluation.Source: §B.1 · p. 14
  • With ground-truth boxes, ScanRefer performance improves significantly without retraining, exceeding 90% in the unique setting.Source: §B.1 · p. 14, Table 7 · p. 14
  • The supplement reports SR3D/NR3D results broken down by easy, hard, view-dependent, and view-independent subsets.The supplied passage identifies the breakdown and Acc@0.25IoU metric but gives no subset values.Source: §B.1 · p. 14, Table 8 · p. 14
  • The selected position-loss weighting uses λ1=0.6 and other component weights of 0.1 or 0.2; a higher λ1 can reduce performance.The reported grid search selects option (d) for implementation.Source: §B.1 · p. 14, Table 9 · p. 14
  • Randomly aligned words improve performance by 0.5%, but perform worse than aligning any one meaningful component and substantially worse than aligning all four.Source: §B.1 · p. 14

B.2. Language Modulated 3D Object Detection

The authors evaluate language-modulated 3D object detection on ScanNet using the visual-grounding model and compare it with BUTD-DETR. They also include an alignment comparison involving random words and decoupled text components.

B.2. Language Modulated 3D Object Detection

  • ScanNet detection uses the BUTD-DETR experimental setup, with the 18 classes concatenated into a sentence as text input.The model outputs bounding boxes and classes for all objects in the point cloud.Source: §B.2 · p. 14
  • The alignment comparison reports Acc@0.25IoU across a main-object baseline, four random words, individual decoupled components, and all four components.The table caption describes the comparison scope but does not provide the outcomes.Source: Table 10 · p. 15
  • 1.1% and 1.5% higher performance than BUTD-DETR, reaching 64.1% and 45.3%, respectively, on ScanRefer.The method is not specifically designed for object detection, and the evaluation uses the same model as the visual-grounding task.Source: Table 11 · p. 15, §B.2 · p. 15

C. Qualitative analysis

Qualitative examples show fine-grained matching for appearance, spatial relations, and ordinal numbers. The analysis also presents name-free grounding and identifies ambiguity, parsing errors, and matching failures as remaining challenges.

Regular 3D Visual Grounding

Grounding without Object Name

  • Without the target name, grounding requires inference from appearance and positional relationships with auxiliary objects; other contrastive methods perform weakly on this task.The authors attribute those methods’ weakness to reliance on object names to exclude interference candidates, which weakens learning of other attributes.Source: C. Qualitative analysis · p. 15, Figure 13 · p. 18

Failure Case Analysis

  • Remaining failures arise from intrinsically ambiguous descriptions, parsing errors on complex or diverse language, and cross-modal matching failures despite correct parsing.Figure 12 marks ground-truth boxes in yellow and predicted boxes in green, and illustrates ambiguity and parsing-error cases.Source: C. Qualitative analysis · p. 15, Figure 12 · p. 17
Loading 2209.14941v3…