PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning

Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guanbin Li, Zhen Li, Shuguang Cui

arXiv:2203.00843v3cs.CV

TL;DR

Sparse 3D point clouds provide weaker texture and color information than images, while methods using extra 2D inputs can add inference cost. X-Trans2Cap transfers 2D knowledge to a 3D-only student during training, and reports CIDEr gains over prior methods on ScanRefer and Nr3D.

  • Problem

    Point-cloud-only 3D dense captioning has weaker texture and color information than image representations, while adding 2D input can impose computation and availability constraints at inference.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    X-Trans2Cap trains a multimodal teacher to guide a 3D-only student through Transformer-based cross-modal knowledge transfer, fusion, and feature alignment.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    +21.0 CIDEr on ScanRefer and +16.7 CIDEr on Nr3D over previous methods, as reported for X-Trans2Cap.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method reports more faithful 3D dense captions while using only 3D inputs at inference.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

3D dense captioning aims to describe individual objects by natural language in 3D scenes, where 3D scenes are usually represented as RGB-D scans or point clouds. However, only exploiting single modal information, e.g., point cloud, previous approaches fail to produce faithful descriptions. Though aggregating 2D features into point clouds may be beneficial, it introduces an extra computational burden, especially in inference phases. In this study, we investigate a cross-modal knowledge transfer using Transformer for 3D dense captioning, X-Trans2Cap, to effectively boost the performance of single-modal 3D caption through knowledge distillation using a teacher-student framework. In practice, during the training phase, the teacher network exploits auxiliary 2D modality and guides the student network that only takes point clouds as input through the feature consistency constraints. Owing to the well-designed cross-modal feature fusion module and the feature alignment in the training phase, X-Trans2Cap acquires rich appearance information embedded in 2D images with ease. Thus, a more faithful caption can be generated only using point clouds during the inference. Qualitative and quantitative results confirm that X-Trans2Cap outperforms previous state-of-the-art by a large margin, i.e., about +21 and about +16 absolute CIDEr score on ScanRefer and Nr3D datasets, respectively.

1. Introduction

Dense captioning describes detected objects individually, but 3D point clouds lack the appearance detail of images, and prior methods can require 2D input at inference. X-Trans2Cap transfers 2D knowledge during training so a 3D-only student can caption at inference, with reported gains on ScanRefer and Nr3D.

1. Introduction

  • Dense captioning extends single-sentence image captioning by providing customized descriptions for individual objects according to their nature and context.The paper frames this as describing each object in a scene rather than the image as a whole.Source: §1 · p. 1
  • Prior 3D dense captioning representations learned solely from sparse point clouds lack strong texture and color information, while extra 2D input can be computationally intensive or unavailable at inference.The paper notes that this limits use in LiDAR scenarios with only point-cloud input.Source: §1 · p. 2
  • X-Trans2Cap uses a teacher-student Transformer framework: the teacher receives multimodal inputs, while the student uses 3D inputs and learns through cross-modal fusion and feature alignment.The student can therefore perform inference using only 3D input.Source: Figure 1 · p. 1, §1 · p. 2
  • +21.0 CIDEr on ScanRefer and +16.7 CIDEr on Nr3D over previous methods, according to the reported contribution summary.The introduction also reports ScanRefer CIDEr rising from 75.75 to 87.09.Source: §1 · p. 2

2. Related Work

The related work traces dense captioning from image-based object descriptions to 3D scene understanding, then positions X-Trans2Cap as a cross-modal teacher-student approach that avoids requiring 2D input at inference.

2.1. Image Captioning and Dense Captioning

  • Image dense captioning generates captions for detected objects and has incorporated global image context and object relations, but limited single-image views degrade performance when transferred directly to 3D.The section also surveys attention, graph neural network, and Transformer approaches for image captioning.Source: §2.1 · p. 2

2.2. 3D Vision and Language

  • 3D vision-language work includes visual grounding and 3D dense captioning, while point-cloud-only instance features lack the stronger texture and color information available in 2D images.Figure 2 depicts object-level 3D proposals and their 2D counterparts as inputs to the teacher-student framework.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3. Cross-modal Knowledge Transferring

  • Prior 2D-assisted 3D methods often require 2D input during both training and inference, adding evaluation cost; X-Trans2Cap instead uses cross-modal fusion in a teacher-student framework.The paper situates this approach among knowledge distillation methods that transfer knowledge across modalities.Source: §2.3 · p. 3

3. Method

X-Trans2Cap uses a teacher–student Transformer framework to transfer information from multimodal features to a 3D-only student. Its method specifies object representations, a Transformer captioning baseline, cross-modal fusion, training objectives, and training and inference paths.

3. Method

  • X-Trans2Cap uses pure 3D input for the student and multimodal input for the teacher, then transfers cross-modal knowledge through its training and inference framework.The method comprises object representations, a TransCap baseline, a cross-modal fusion module, training objectives, and separate training and inference data flows.Source: §3 · p. 3

3.1. Object Representation

  • Each object is represented using 3D features, predicted semantics, bounding-box information, and positional encoding; the teacher also receives 2D image features projected from object boxes.For training, an image is randomly selected from ScanNet videos and Faster-RCNN features are extracted from the corresponding 2D box.Source: §3.1 · p. 4

3.2. Baseline Model: TransCap

  • TransCap encodes object features with Transformer self-attention and generates the target object's caption using a decoder conditioned on prior words and encoder features.The encoder self-attention includes two persistent memory vectors that learn prior knowledge.Source: §3.2 · p. 4

3.3. Cross-Modal Fusion Module

  • At matching encoder levels, CMF adds student features to randomly masked teacher features and passes the fused features to the teacher's next encoder layer.Because the connection runs from student to teacher, the teacher can be discarded at inference without adding computation to the student.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

3.4. Objective Function

  • Training combines feature-alignment and captioning losses for teacher and student; an enhanced X-Trans2Cap (C) also uses CIDEr-D reward.The reward is baselined using the mean of rewards, and the three loss terms are combined linearly with weights α = 1, β = 1, and γ = 0.1.Source: §3.4 · p. 5

3.5. Training and Inference Schemes

  • During training, both networks and CMF operate together; when only 3D input is available at inference, the student alone is used.When auxiliary 2D information is available at inference, the teacher framework is used instead.Source: §3.5 · p. 5

4. Experiment

The experiments evaluate X-Trans2Cap on ScanRefer and Nr3D using ground-truth instances and detected objects, and compare its captioning and knowledge-transfer designs. The reported results favor X-Trans2Cap over prior methods, while ablations show gains associated with feature alignment and cross-modal fusion.

4.1. Datasets

  • ScanRefer contains 800 indoor scenes and 51,583 language queries; Nr3D contains 41,503 queries and lacks ScanRefer’s fixed or redundant sentence patterns.The paper does not compare on Sr3D because that dataset is generated entirely from machine templates.Source: §4.1 · p. 5

4.2. Tasks and Metrics

  • Oracle DC supplies each instance’s point cloud, while Scan DC first detects objects from full 3D scans; Oracle DC uses CIDEr, BLEU-4, METEOR, and ROUGE.Scan DC combines caption metrics with box IoU thresholds and also reports detection mAP.Source: §4.2 · p. 6

4.3. 3D Dense Captioning Results

  • 11.04 and 9.42 CIDEr improvement over TransCap on ScanRefer and Nr3D, respectively; X-Trans2Cap is about 20 CIDEr higher than Scan2Cap on both.Table 1 covers ground-truth instances and reports conventional captioning metrics with and without extra 2D input at inference.Source: §4.3 · p. 6, Table 1 · p. 6
  • X-Trans2Cap without extra 2D inference input outperforms Scan2Cap in Scan DC, and its detection performance also improves.The comparison uses 3D scans on ScanRefer; the reported detection improvement occurs without extra 2D input to the detector during training or testing.Source: §4.3 · pp. 6–7, Table 2 · p. 7
  • Pure distillation and other cross-modal transfer variants offer limited gains, while offline pretraining the teacher is associated with a 7.38 CIDEr drop.The authors report that X-Trans2Cap performs better in Oracle DC on ScanRefer and attribute its performance to the teacher-student framework and CMF module.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4. Analysis and Ablation Studies

  • Removing feature alignment lowers CIDEr from 87.09 to 79.58, while removing CMF causes a 5.25 CIDEr drop.Without alignment, the ablated model remains 3.83 CIDEr above the TransCap baseline.Source: §4.4 · p. 8, Table 4 · p. 8
  • Discarding CMF’s random mask hampers caption results; attention slightly improves BLEU-4 and ROUGE but increases complexity and lowers CIDEr.Source: §4.4 · p. 8, Table 4 · p. 8

5. Conclusion

X-Trans2Cap uses cross-modal knowledge transfer to enhance 3D dense captioning. The paper reports better performance than previous methods on multiple datasets and proposes applying the approach to broader 3D vision-and-language scenarios.

5. Conclusion

  • X-Trans2Cap is an enhanced 3D dense captioning method built around cross-modal knowledge transfer.The authors describe its network architecture and knowledge distillation method as carefully designed.Source: §5 · p. 8
  • The authors report that X-Trans2Cap outperforms previous methods by a large margin on multiple datasets and produces more faithful captions.Source: §5 · p. 8
  • The authors propose applying 2D priors and cross-modal knowledge transfer to 3D vision-and-language scenarios with severe missing texture details.They frame this as a potential broader application of their work.Source: §5 · p. 8

Supplementary MaterialS

The supplementary material covers implementation, efficiency, subjective evaluation, additional Nr3D scan dense-captioning experiments, and instance-representation attributes.

Supplementary MaterialS

  • The supplementary material describes implementation details, model efficiency, and subjective evaluation.These topics are covered in Sections B, C, and D, respectively.Source: A. Overview · p. 12
  • Additional scan dense-captioning experiments on Nr3D are provided in Section E.Source: A. Overview · p. 12
  • Section F discusses the effectiveness of each attribute in the instance representation.Source: A. Overview · p. 12

B. Implementation Details

Implementation differs by task: Oracle DC uses PointNet++ object features and ground-truth categories, while Scan DC uses VoteNet proposal features and detector predictions.

B. Implementation Details

C. Running Time Evaluation

In Oracle DC inference, X-Trans2Cap (3D) runs more than 20 times faster than its baseline and the version using extra 2D modality.

C. Running Time Evaluation

  • More than 20× faster: X-Trans2Cap (3D) outpaces both its baseline and X-Trans2Cap using extra 2D modality in Oracle DC inference.The evaluation considers per-scan inference time and parameter counts.Source: C. Running Time Evaluation · p. 12

D. Subjective Evaluation

A three-volunteer subjective evaluation assessed whether ScanRefer captions reflected object colors and local spatial relations; X-Trans2Cap was judged more faithful on both aspects.

D. Subjective Evaluation

  • Three volunteers evaluated 100 randomly selected ScanRefer descriptions generated by Scan2Cap and X-Trans2Cap under Oracle DC.They manually checked object color attributes and spatial relations in the local environment.Source: D. Subjective Evaluation · p. 12
  • The evaluation focused on whether descriptions correctly represented object color attributes.Source: D. Subjective Evaluation · p. 12
  • Volunteers found X-Trans2Cap captions more faithful in representing attributes and spatial relationships.Source: D. Subjective Evaluation · p. 12

E. Scan Dense Captioning on Nr3D

On Nr3D, X-Trans2Cap achieved the highest captioning scores across metrics, with reported CIDEr@0.25 gains over TransCap and Scan2Cap.

E. Scan Dense Captioning on Nr3D

F. Analysis and Ablation Studies

The ablations identify object class and positional encoding as important student representation inputs, while teacher-side results favor 2D bounding-box information over extracted 2D features.

F. Analysis and Ablation Studies

Related papers

Loading 2203.00843v3…