PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End 3D Dense Captioning with Vote2Cap-DETR

Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Tao Chen, Gang YU

arXiv:2301.02508v1cs.CV

TL;DR

3D dense captioning must localize objects and describe each one, while existing detect-then-describe methods rely on hand-crafted components. Vote2Cap-DETR uses a transformer encoder-decoder with vote queries to predict locations and captions in parallel. It surpasses prior works on two widely used datasets.

  • Problem

    3D dense captioning must localize scene objects and describe each one, a challenge with sparse point clouds and cluttered object distributions.

    Source: §1 · p. 1

  • Method

    Vote2Cap-DETR uses a transformer encoder-decoder and vote-query-driven decoder to predict object locations and captions in parallel as a set-to-set task.

    Source: §1 · p. 2

  • Results

    Vote2Cap-DETR surpasses prior works by a large margin on two widely used 3D dense captioning datasets.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    The authors report that vote queries boost convergence and detection performance, alongside a lightweight query-driven caption head for informative captions.

    Source: §5 · p. 8

    The paper notes that 3DETR boxes around sampled query centers can lie far from real objects on sparse surfaces, slowing convergence and causing missed detections.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

3D dense captioning aims to generate multiple captions localized with their associated object regions. Existing methods follow a sophisticated ``detect-then-describe'' pipeline equipped with numerous hand-crafted components. However, these hand-crafted components would yield suboptimal performance given cluttered object spatial and class distributions among different scenes. In this paper, we propose a simple-yet-effective transformer framework Vote2Cap-DETR based on recent popular \textbf{DE}tection \textbf{TR}ansformer (DETR). Compared with prior arts, our framework has several appealing advantages: 1) Without resorting to numerous hand-crafted components, our method is based on a full transformer encoder-decoder architecture with a learnable vote query driven object decoder, and a caption decoder that produces the dense captions in a set-prediction manner. 2) In contrast to the two-stage scheme, our method can perform detection and captioning in one-stage. 3) Without bells and whistles, extensive experiments on two commonly used datasets, ScanRefer and Nr3D, demonstrate that our Vote2Cap-DETR surpasses current state-of-the-arts by 11.13\% and 7.11\% in CIDEr@0.5IoU, respectively. Codes will be released soon.

1. Introduction

The introduction frames 3D dense captioning as jointly localizing objects and describing each one, a challenge in sparse point clouds and cluttered scenes. It presents Vote2Cap-DETR as a one-stage transformer that predicts object locations and captions in parallel.

1. Introduction

  • 3D dense captioning requires localizing all scene objects and generating a descriptive sentence for each, despite sparse point clouds and cluttered object distributions.Source: §1 · p. 1
  • Prior methods use serial detect-then-describe pipelines and hand-crafted components, making captioning dependent on detection and adding hyperparameters for cluttered scenes.Examples include radii, 3D operators, proposal-neighbor definitions, and non-maximum suppression.Source: §1 · pp. 1–2
  • Vote2Cap-DETR uses a full transformer encoder-decoder to predict object locations and captions in parallel as a one-stage set-to-set task.Target instances and language annotations are matched one-to-one with queries.Source: §1 · p. 2
  • Its vote decoder adds spatial bias for localization, while its query-driven caption head models relations and attributes using local and global context.The vote query combines seed-point embeddings with the vote transformation of a box relative to the seeds.Source: §1 · p. 2
  • Vote2Cap achieves state-of-the-art performance on Nr3D (45.53% C@0.5) and ScanRefer (73.77% C@0.5).Source: §1 · p. 2

2. Related Work

Related work covers 3D dense captioning systems built around detector proposals, transformer-based detection, and image-captioning approaches. Vote2Cap-DETR is distinguished by parallel box localization and caption generation as a set-prediction task.

3D Dense Captioning

  • Earlier 3D dense captioning methods use detector outputs, adding graph reasoning, spatial modeling, task coupling, 2D knowledge transfer, or contextual information.Scan2Cap [11] and MORE [18] use hand-crafted graph rules; SpaCap3D [36] models spatial relations; 3DJCG [4] and D3Net [7] study joint promotion with visual grounding; χ-Trans2Cap [38] transfers 2D knowledge; [42] considers contextual information.Source: §2 · p. 2

DETR: from 2D to 3D

  • DETR [5] frames object detection as set prediction without non-maximum suppression, while 3DETR [24] samples seed points to generate 3D proposals.Vote2Cap-DETR extends this transformer line to parallel box localization and caption generation, with vote queries intended to improve performance and convergence.Source: §2 · p. 2

Image Captioning

  • Image-captioning approaches use region, grid, or combined visual features, but do not directly address the need to localize and describe individual 3D objects.Vote2Cap-DETR's caption head uses the object query and its local 3D context.Source: §2 · pp. 2–3

3. Method

Vote2Cap-DETR processes a 3D scene with a transformer encoder-decoder and predicts object locations and captions through parallel task heads. Its method uses vote queries, contextual captioning, and set-to-set training.

3. Method

  • The model predicts a set of object boxes and captions from a 3D point cloud using a transformer decoder and separate detection and caption heads.The decoder attends to object-object and object-scene interactions.Source: §3 · p. 3

3.1. 3DETR Encoder

  • The 3DETR scene encoder tokenizes the point cloud and processes point tokens with a masked transformer encoder to produce encoded scene tokens.The encoded representation contains 1,024 tokens with 256-dimensional features alongside 3D positions.Source: §3.1 · p. 3

3.2. Vote Query

  • Vote queries combine seed-point embeddings with predicted vote offsets and gathered point-cloud features, adding spatial bias and content information to object queries.The authors report that vote queries improve detection and accelerate convergence.Source: §3.2 · p. 3

3.3. Parallel Decoding

  • Parallel detection and caption heads use a Dual-Clued Captioner that pairs vote-query features with surrounding context to generate informative captions.The vote-query feature identifies the described region, while nearby context supplies additional information.Source: §3.3 · pp. 3–4

3.4. Set prediction loss for 3D Dense Captioning

  • Set-to-set training assigns sampled instance captions to proposals and combines vote, detection, and caption supervision, then fine-tunes caption generation with CIDEr-reward SCST.Caption training first uses word-level cross-entropy; the full system also averages caption losses over annotated instances.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate Vote2Cap-DETR on ScanRefer and Nr3D, compare it with existing methods, and examine model components and qualitative outputs. They report benchmark comparisons alongside ablations of vote queries, contextual caption features, set-to-set training, and NMS.

4.1. Datasets, Metrics, and Implementation Details

  • Comparisons cover ScanRefer and Nr3D using m@kIoU under MLE and SCST; baselines differ in detectors, grounding supervision, and SCST reward design.Evaluation first filters proposals with NMS and matches each instance to its highest-IoU remaining proposal.Source: §4.1 · p. 5, §4.2 · p. 6

4.3. Ablation Study

  • Adding position features raises detection by 0.97% mAP50 but slows early convergence; adding content features raises it another 2.98% and speeds training.The position-only variant's vote-generation module takes time to learn accurate spatial offsets.Source: §4.3 · p. 6, Table 3 · p. 7

4.2. Comparison with Existing Methods

  • On ScanRefer, Vote2Cap-DETR scores 59.32% C@0.5 versus 49.48% for 3DJCG under MLE and 70.63% versus 62.64% for D3Net under SCST.It also surpasses prior methods on Nr3D.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7

4.3. Ablation Study

  • Contextual information improves captioning with object features as prefixes, and local vote-query context is more beneficial than whole-scene context.The authors interpret this as evidence that close surroundings matter when describing an object.Source: §4.3 · p. 7, Table 4 · p. 7
  • Set-to-set training matches sentence training under MLE, improves C@0.5 by 2.38% under SCST, and converges faster with a larger caption-head batch.The convergence comparison is shown for MLE and SCST training.Source: Table 5 · p. 7, §4.3 · p. 7, Figure 7 · p. 8

4.4. Qualitative Results

  • Compared with SpaCap3D and 3DJCG, Vote2Cap-DETR produces tight boxes and accurate descriptions of object attributes, classes, and spatial relationships.The qualitative comparison marks correct attribute words in green and incorrect descriptions in red.Source: Figure 6 · p. 8, §4.4 · p. 8

4.3. Ablation Study

  • Removing NMS raises C@0.5 because m@kIoU does not penalize redundant predictions; Vote2Cap-DETR's detection performance remains stable without NMS.By contrast, SpaCap3D and 3DJCG lose 14.47% and 17.55% mAP50, respectively.Source: Table 6 · p. 8, §4.3 · p. 8

5. Conclusion.

Vote2Cap-DETR is a one-stage transformer approach that decodes vote queries into boxes and captions in parallel. The paper reports it surpasses prior works with heavy reliance on hand-crafted components and achieves state-of-the-art results on the Scan2Cap online test benchmark.

5. Conclusion.

  • Vote2Cap-DETR decodes vote queries into box predictions and captions in parallel, using spatial bias, content-aware features, and a lightweight query-driven caption head.The authors report that vote queries boost convergence and detection performance, while the caption head generates informative captions.Source: §5 · p. 8
  • Vote2Cap-DETR surpasses prior works that rely heavily on hand-crafted components.Source: §5 · p. 9

Appendix

  • Vote2Cap-DETR achieves a new state-of-the-art on the Scan2Cap online test benchmark.The benchmark result is shown in Figure 8.Source: Appendix · p. 11

A. VoteNet baseline with set-to-set training

The VoteNet baseline benefits from the proposed caption head and set-to-set training, but remains behind Vote2Cap-DETR. It surpasses 3DJCG on two reported CIDEr metrics.

A. VoteNet baseline with set-to-set training

B. Experiments

The experiments section provides evaluations on the Scan2Cap online test benchmark and additional experimental details.

B. Experiments

B.1. Scan2Cap Test Benchmark

Vote2Cap-DETR achieves a new state-of-the-art on all metrics of the Scan2Cap online test benchmark.

B.1. Scan2Cap Test Benchmark

  • Vote2Cap-DETR achieves a new state-of-the-art on the Scan2Cap online test benchmark.Source: §B.1 · p. 11
  • The reported state-of-the-art covers all metrics on the benchmark.Figure 8 presents the benchmark result.Source: §B.1 · p. 11

B.2. Per-Class mAP Results

Table 8 reports per-class detection AP for VoteNet, 3DETR, and Vote2Cap-DETR on ScanNet scenes at an IoU threshold of 0.5; overall performance is listed in the main paper.

B.2. Per-Class mAP Results

B.3. Implementation Details

Vote2Cap-DETR encodes the point cloud, generates vote queries, and decodes them into boxes and captions. Its Scan2Cap test benchmark results are state of the art across all reported metrics.

B.3. Implementation Details

  • The pipeline encodes the point cloud, generates vote queries from encoded features, then decodes them into bounding boxes and captions.Source: §B.3 · p. 11
  • Vote2Cap-DETR achieves a new state of the art on all metrics of the Scan2Cap online test benchmark.Source: Figure 8 · p. 12
  • The encoder converts 40,000 input points into 2,048 tokens, then encodes and downsamples them to 1,024 scene tokens.It uses a three-layer transformer encoder, a set-abstraction layer, and radius-based attention masks.Source: §B.3 · p. 12
  • A vote-query generator produces 256 object queries from the 1,024 encoded scene points.An FFN estimates offsets and projects features; set abstraction gathers scene features for the query positions.Source: §B.3 · p. 12
  • An eight-layer transformer decoder feeds parallel detection and caption heads; the caption head uses query and surrounding-context features.The detection head predicts centers, sizes, and semantic classes without rotation angles; the caption decoder uses dataset-specific vocabularies.Source: §B.3 · pp. 12–13

C. Qualitative Results

Qualitative visualizations show Vote2Cap-DETR producing tight boxes and accurate descriptions on Nr3D, with vote queries concentrated on objects and accurate box predictions in scenes.

C. Qualitative Results

Loading 2301.02508v1…