Source-linked AI summary
TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, Si Liu
TL;DR
Fine-grained 3D visual grounding must identify a described object among distractors of the same category, while prior methods lacked explicit cross-modal entity and relation modeling. TransRefer3D uses Transformer-based contextual blocks with entity- and relation-aware attention. It reports superior performance on Nr3D and Sr3D, including an improvement of up to 10.4% on Nr3D with Sr3D+ augmentation.
Problem
Existing 3D visual-grounding methods lack explicit cross-modal entity and relation modeling, while the task requires distinguishing a referent from same-category distractors.
Source: §1 · p. 2Method
TransRefer3D stacks contextual blocks that combine self-attention with entity-aware matching and relation-aware cross-modal attention.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors conclude that Transformers are suitable for connecting language and 3D vision and that entity-and-relation-aware modeling is effective for fine-grained 3D grounding.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Recently proposed fine-grained 3D visual grounding is an essential and challenging task, whose goal is to identify the 3D object referred by a natural language sentence from other distractive objects of the same category. Existing works usually adopt dynamic graph networks to indirectly model the intra/inter-modal interactions, making the model difficult to distinguish the referred object from distractors due to the monolithic representations of visual and linguistic contents. In this work, we exploit Transformer for its natural suitability on permutation-invariant 3D point clouds data and propose a TransRefer3D network to extract entity-and-relation aware multimodal context among objects for more discriminative feature learning. Concretely, we devise an Entity-aware Attention (EA) module and a Relation-aware Attention (RA) module to conduct fine-grained cross-modal feature matching. Facilitated by co-attention operation, our EA module matches visual entity features with linguistic entity features while RA module matches pair-wise visual relation features with linguistic relation features, respectively. We further integrate EA and RA modules into an Entity-and-Relation aware Contextual Block (ERCB) and stack several ERCBs to form our TransRefer3D for hierarchical multimodal context modeling. Extensive experiments on both Nr3D and Sr3D datasets demonstrate that our proposed model significantly outperforms existing approaches by up to 10.6% and claims the new state-of-the-art. To the best of our knowledge, this is the first work investigating Transformer architecture for fine-grained 3D visual grounding task.
1 INTRODUCTION
Fine-grained 3D visual grounding must distinguish a referred object from same-category distractors, a challenge earlier methods did not explicitly address through entity and relation modeling. TransRefer3D uses Transformer-based attention to model these cues and achieves state-of-the-art performance on ReferIt3D.
1 INTRODUCTION
- Most prior visual-grounding work addressed 2D images, which may not capture the full information of 3D reality..Source: §1 · p. 1
- Fine-grained ReferIt3D grounding distinguishes objects within a fine-grained category, so object classification alone is insufficient.The benchmark contains multiple objects of the same fine-grained category as the referent.Source: §1 · p. 2
- Earlier methods lacked explicit cross-modal modeling of entities and their relationships, and some relied on simple visual-linguistic feature fusion.The authors describe these methods as relatively weak on the ReferIt3D benchmark.Source: §1 · p. 2
- Transformer self-attention preserves the permutation-invariant property of point-cloud features, motivating its use for 3D visual grounding.The paper notes that preserving this property is a basic design consideration for point-cloud encoders.Source: §1 · p. 2
- TransRefer3D combines self-attention with entity-aware and relation-aware attention in contextual blocks, stacking several blocks for hierarchical multimodal context modeling.Entity-aware attention matches object and linguistic entity features; relation-aware attention matches visual object-pair relations with linguistic relations.Source: §1 · p. 2
- TransRefer3D improves test accuracy by up to 10.4% on Nr3D with Sr3D+ augmentation over existing cutting-edge approaches.The paper reports state-of-the-art performance on ReferIt3D.Source: §1 · p. 3
2 RELATED WORKS
Related work spans 2D grounding, emerging 3D grounding benchmarks and methods, Transformer attention, and multimodal co-attention. The paper positions its approach as applying entity- and relation-aware Transformer attention to fine-grained 3D grounding.
2.1 2D Visual Grounding
- 2D visual grounding localizes descriptive words in images, while research on 3D visual grounding remains relatively preliminary.Prior 2D approaches include fragment matching, query-guided regression, and text-guided segmentation.Source: §2.1 · p. 3
2.2 3D Visual Grounding
- ScanRefer [5] targets 3D object localization, while ReferIt3D supplies segmented instances and same-category distractors for finer-grained grounding.InstanceRefer [34] uses instance attribute, relation, and localization perceptions; this paper instead builds on Transformer architecture with EA and RA modules.Source: §2.2 · p. 3
2.3 Attention Mechanism and Transformer
- Transformers combine multi-head self-attention with a feed-forward network, alongside layer normalization and residual connections.The attention heads project queries, keys, and values before their outputs are combined.Source: §2.3 · p. 3
2.4 Co-Attention for Multimodal Learning
- Co-attention models interactions between modalities; this work extends it to guide entity-and-relation context modeling for 3D visual grounding.The authors describe this as the first investigation of co-attention on 3D visual-linguistic tasks, to their knowledge.Source: §2.4 · p. 3
3 METHODOLOGY
TransRefer3D stacks entity- and relation-aware attention blocks to model multimodal context for fine-grained 3D visual grounding. Its modules match visual and linguistic features at the entity and relation levels.
3.1 Overview
- TransRefer3D stacks ERCBs after encoding objects and words, and trains referent selection alongside auxiliary object and utterance classification.Object features come from shared PointNet++; positional information is fused into visual and linguistic features.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 EA: Entity-aware Attention Module
- EA treats scene objects and word-level features as entities, using multi-head cross-modal attention in both visual-to-language and language-to-visual directions.The two modalities are matched symmetrically to model entities in each.Source: §3.2 · p. 4
3.3 RA: Relation-aware Attention Module
- Because same-category objects can receive similar entity scores, RA matches asymmetric pairwise visual relations to language and stacks binary relations for progressive context.The binary-relation representation assumes more complex relations can be described as sets of binary relations.Source: §3.3 · pp. 4–5, Figure 4 · p. 5
3.4 ERCB: Entity-and-Relation Aware Contextual Block
- ERCB applies within-modality self-attention, then EA and RA matching, before fusing enhanced features through feed-forward networks.Stacking ERCBs forms hierarchical multimodal context; the paper also reports that TransRefer3D and EACB are more powerful at multimodal context modeling than other methods.Source: Figure 5 · p. 5, §3.4 · p. 5
4 EXPERIMENTS
The experiments assess TransRefer3D against other methods and analyze the proposed method through quantitative and qualitative studies.
4.1 Implementation details and Datasets
The implementation follows ReferIt3D settings, evaluates on Nr3D, Sr3D, and Sr3D+, and includes joint-training evaluation on Nr3D.
4.1 Implementation details and Datasets
- Table 1 compares TransRefer3D with prior methods on Nr3D with or without Sr3D/Sr3D+ training.Source: Table 1 · p. 6
- Unless otherwise indicated, training uses Adam at learning rate 0.0005, feature dimension 128, batch size 32, four ERCBs, and four-head attention.The model is trained until convergence under the stated ReferIt3D comparison settings.Source: §4.1 · pp. 5–6
- Nr3D has 41.5K human descriptions, Sr3D has 83.5K synthetic descriptions, and Sr3D+ augments Sr3D with scenes where the referent may be the only fine-grained-category object.Nr3D and Sr3D contain same-category distractors; Sr3D templates describe five spatial-relation types.Source: §4.1 · p. 6
- Sr3D and Sr3D+ are jointly trained with Nr3D and evaluated on its test set; the paper also trains and tests on Sr3D alone.Source: §4.1 · p. 6
4.2 Quantitative Performance on Nr3D and Sr3D/Sr3D+
TransRefer3D outperforms previous methods on Nr3D and Sr3D, with results on Nr3D improving further under joint training with Sr3D or Sr3D+. The model performs especially well in hard Nr3D cases and achieves more than 9.4% accuracy improvement on Sr3D.
4.2.1 Evaluation on Nr3D.
- TransRefer3D performs especially well on hard Nr3D cases, which contain multiple distractors.The scenario split also distinguishes easy cases with one distractor and view-dependent from view-independent descriptions.Source: §4.2.1 · p. 6
- Joint training on Nr3D and Sr3D improves ReferIt3DNet by 1.6% and TransRefer3D by 5.1% over training only on Nr3D.The authors suggest that Transformers may require relatively more data.Source: §4.2.1 · p. 6
- The authors report a similar trend when TransRefer3D is jointly trained with Nr3D and Sr3D+.The model is evaluated on the Nr3D test set.Source: §4.2.1 · p. 6
4.2 Quantitative Performance on Nr3D and Sr3D/Sr3D+
- TransRefer3D outperforms previous models by a large margin on Nr3D and all prior works by more than 9.4% accuracy on Sr3D.The Sr3D result uses training and testing on Sr3D only.Source: §4.2.2 · p. 6, §4.2.1 · p. 6
4.3 Ablation Study
Ablations show that self-attention and the proposed entity- and relation-aware modules contribute to performance. The experiments also compare parallel and stacked EA/RA designs and test ERCB depth.
4.3 Ablation Study
- Removing self-attention produces the worst results, while removing the proposed EA and RA modules also hinders performance.The ablations include vision-guided EA, language-guided EA, and RA.Source: §4.3 · pp. 6–7, Table 3 · p. 7
- The default ERCB places EA and RA in parallel; the authors also test stacking EA before RA.They suggest parallel modules may better reflect the relative independence of entity and relation information.Source: §4.3 · p. 7
- Four ERCB layers achieve the best performance on Nr3D among the tested depths.The authors note that shallower models may miss complicated relations, while deeper Transformers can be hard to train due to unbalanced gradients and amplification.Source: §4.3 · p. 7, Figure 6 · p. 7
4.4 Qualitative Analysis
The qualitative examples compare TransRefer3D with ReferIt3DNet and visualize how EA and RA attend to entities and relations. The attention maps show more interpretable patterns in later layers.
4.4 Qualitative Analysis
- Compared with ReferIt3DNet, TransRefer3D performs better on the shown simple and hard cases, including a scene with five closely placed monitors.The authors attribute the examples to fine-grained multimodal context modeling.Source: §4.4.1 · p. 7
4.4.2 Attention visualization of EA and RA.
- In the last ERCB, EA maps the entity words to corresponding visual features for couches and tables more clearly than in the first layer.The displayed case uses the expression “Middle part of couch, seams end in between the 2 beige cocktail tables.”Source: §4.4.2 · pp. 7–8, Figure 8 · p. 8
- Later RA maps respond to “between” for a table located between two ottomans and to “middle” for object pairs surrounding the referred couch part.The authors interpret the latter pattern as capturing relations more complex than binary relations.Source: §4.4.2 · p. 8, Figure 8 · p. 8
5 CONCLUSION
The conclusion presents TransRefer3D as a stack of ERCBs integrating EA and RA for multimodal context modeling. It reports superior performance on Nr3D and Sr3D and argues that Transformer is suitable for connecting language and 3D vision.
5 CONCLUSION
- TransRefer3D stacks ERCBs that integrate EA and RA for entity-and-relation-aware multimodal context modeling.Source: §5 · p. 8
- The model achieves superior performance over previous approaches on both Nr3D and Sr3D.Source: §5 · p. 8
- The authors conclude that Transformer is suitable for connecting language and 3D vision and that entity-and-relation-aware modeling is effective.They suggest EA, RA, and ERCB may influence future 3D multimodal machine learning work.Source: §5 · p. 8