Source-linked AI summary
Scan2Cap: Context-aware Dense Captioning in RGB-D Scans
Dave Zhenyu Chen, Ali Gholami, Matthias Nießner, Angel X. Chang
TL;DR
Dense captioning has been limited by 2D views that miss objects’ physical extent, size, and environmental locations. Scan2Cap jointly localizes and describes 3D objects using attention and graph-based relation features, outperforming 2D-3D back-projected captioning baselines by 27.61%.
Problem
2D dense captioning is limited by viewpoint-specific images that fail to capture 3D objects’ physical extent and locations in the environment.
Method
Scan2Cap is an end-to-end architecture that jointly localizes and describes 3D objects using message passing for object relations and attention-based captioning.
Results
27.61%: Scan2Cap outperforms 2D-3D back-projected results of 2D captioning baselines on 3D dense captioning.
Takeaways & Limitations
Using 3D object size, location, and local relations enables more accurate descriptions while addressing localization and description together.
Takeaways & Limitations
Qualitative examples still show mistakes with local surroundings, indicating room for improvement.
Abstract
from arXiv · showhide
We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors. As input, we assume a point cloud of a 3D scene; the expected output is the bounding boxes along with the descriptions for the underlying objects. To address the 3D object detection and description problems, we propose Scan2Cap, an end-to-end trained method, to detect objects in the input scene and describe them in natural language. We use an attention mechanism that generates descriptive tokens while referring to the related components in the local context. To reflect object relations (i.e. relative spatial relations) in the generated captions, we use a message passing graph module to facilitate learning object relation features. Our method can effectively localize and describe 3D objects in scenes from the ScanRefer dataset, outperforming 2D baseline methods by a significant margin (27.61% CiDEr@0.5IoUimprovement).
1. Introduction
The paper introduces dense captioning in 3D scenes to jointly localize and describe objects, using 3D geometry, spatial context, and object relations to improve descriptions over 2D approaches.
- 2D dense captioning is limited by viewpoint, field of view, and its inability to capture objects’ physical extent and environmental locations.
- Dense captioning in 3D scenes jointly localizes and describes each object using natural language.
- Scan2Cap uses a relational graph with message passing to learn 3D object and object-relation features.
- Context-aware attention generates descriptive tokens while attending to object and relation features.
- 27.61% improvement in the reported comparison shows Scan2Cap outperforms 2D-3D back-projected captioning baselines.
2. Related work
Prior work developed image captioning, dense captioning, and object-relation modeling, while this paper extends these ideas to capture appearance and spatial information in 3D environments.
- 3D object detection methods use volumetric grids or point-cloud pipelines such as Hough voting with PointNet++ backbones.
- Attention-based image captioning focuses on grid regions or detected objects, while newer methods model object relations with graphs or transformers.
- Dense captioning generates captions for all detected image objects, with later methods adding global context or explicit region relations.
- Prior single-image methods cannot capture the large context available in 3D environments because of limited view.
- The paper focuses on decomposing 3D scenes and capturing objects’ appearance and spatial information.
3. Task
The task takes a feature-augmented point cloud of a 3D scene and outputs bounding boxes and natural-language descriptions for its underlying object instances.
- The input is a scene point cloud containing object geometries, RGB values, and normal vectors.
- The output consists of object bounding boxes and corresponding natural-language descriptions.
4. Method
Scan2Cap processes point clouds through detection, relational graph, and context-aware captioning modules to generate 3D object proposals and descriptions. The architecture incorporates object appearance, spatial relations, and local context during caption generation.
- Detection Backbone: The detection backbone adapts PointNet++ and VoteNet to aggregate point clusters representing candidate objects.Back-projected image features and height information augment the point-cloud representation before proposal generation.
- Detection Backbone: The proposal module predicts objectness masks and axis-aligned bounding boxes with semantic predictions for up to M object proposals.Each box encodes its center, lengths, and an 18-dimensional semantic prediction vector.
- Relational Graph: The relational graph treats object proposals as nodes and nearby-object relationships as edges, using the nearest K surrounding objects.Message passing enhances object features and extracts object relation features for subsequent captioning.
- Relational Graph: Message passing updates node features with learned functions of paired node features and their differences, then produces enhanced object and relation features.The final graph step supplies enhanced object features, while an additional layer outputs relation features and predicts angular deviations.
- Context-aware Attention Captioning: The context-aware attention captioning module combines enhanced object features with relation features to generate descriptions one token at a time.Attention incorporates neighboring objects and their associated relations into context features, while Fusion and Language GRUs produce token predictions.
- Training Objective: Training jointly combines detection, angular-deviation, and description losses with tuned weights, while descriptions are selected using proposal overlap during optimization.The reported implementation trains end-to-end with ADAM and applies proposal-based selection for the description loss.
5. Experiments
Experiments on ScanRefer evaluate captioning and localization jointly, comparing 3D methods with back-projected 2D baselines and analyzing context, relations, and training design. Scan2Cap substantially improves quantitative and qualitative performance, although detection and local-context errors remain.
- 5.1. Quantitative Analysis: Scan2Cap outperforms all listed baselines in 3D dense captioning, using metrics that combine caption scores with predicted-box IoU thresholds of 0.25 and 0.5.Table 1 reports conventional captioning metrics averaged with the percentage of boxes exceeding each IoU threshold.
- 5.2. Qualitative Analysis: With ground-truth detections, Oracle2Cap3D outperforms 2D counterparts, while the 3D-versus-2D advantage is especially pronounced for spatial-relation descriptions.The comparison evaluates object categories, appearance attributes, and spatial relations.
- 5.2. Qualitative Analysis: Scan2Cap predicts better bounding boxes and richer relational descriptions than the qualitative baselines, whereas 2D-3D Proj is limited by 2D input.Oracle2Cap3D can provide more descriptive detail with ground-truth object information but still makes local-surrounding mistakes.
- 5.3. Analysis and Ablations: Context-aware attention improves performance over the basic GRU captioner on all metrics, and the full context-enhancement configuration outperforms the other ablations.The full configuration combines the relational graph and context-aware attention captioning.
- 5.3. Analysis and Ablations: End-to-end fine-tuning of VoteNet improves description performance over the fixed-backbone ablation on all m@0.5IoU metrics despite similar detection results.The authors attribute this to gradient updates from relative-orientation and description-generation losses compensating for detection errors.
6. Conclusion
Scan2Cap introduces an end-to-end architecture that jointly localizes 3D objects in RGB-D scans and describes them in natural language. Its attention-based captioning and message passing network use local object context, outperforming 2D-based dense captioning methods.
- Scan2Cap jointly localizes 3D objects in point clouds and generates natural-language descriptions.
- An attention-based captioning pipeline generates descriptive tokens while referring to related components in the local context.
- A message passing network captures object relations to support 3D object description.
- The architecture outperforms 2D-based dense captioning methods on 3D dense description generation.
Supplementary Material
The supplement provides additional experimental details, including 2D input selection and qualitative comparisons across rendered, matched, and bird’s-eye-view frames. It documents why rendered frames are preferred for the 2D captioning experiments.
- The supplement covers 2D captioning choices, 3D-to-2D projection, additional ablations, and qualitative 3D examples.
- Rendered annotated-viewpoint frames, matched video frames, and bird’s-eye-view frames are compared for caption generation.
- Bird’s-eye-view frames provide a broad overview of large objects but can miss smaller objects beneath desks.
A.1. Experimental setup
The 2D experiments evaluate input choices, captioning methods, and visual features on ScanRefer. Rendered frames are selected because matching video viewpoints is difficult and rendered inputs yield better captioning performance.
- 2D experiments are implemented with PyTorch and Detectron2 to select the input, captioning method, and visual features.
- Choice of 2D input: Rendered frames produce better captions than matched video frames despite rendering artifacts.
- Choice of 2D input: Matched video frames are difficult to select because recordings may differ in viewpoint and have limited field of view or motion blur.
- Captioning method: Captioning methods include show and tell, bottom-up and top-down attention, and a meshed-memory transformer.
- Visual features: Visual features include global image, target-object, and context-object features, using oracle, detected, or back-projected object boxes.
A.2. Results
The supplementary results evaluate 2D detection, instance segmentation, and captioning across rendered viewpoints and input configurations. They report metric definitions, viewpoint and feature comparisons, and the selection of show and tell as the 2D baseline.
- 2D experiments evaluate instance segmentation and captioning methods.
- Rendered frames consistently outperform matched frames across tested visual features and captioning methods.
- 2D detection and instance segmentation are evaluated with mAP, mAP50, and mAP75 on the ScanRefer validation split.
- Caption quality is measured with CiDEr, BLEU-4, METEOR, and ROUGE, without incorporating IoU into these 2D metrics.
- Show and tell outperforms top-down attention and the meshed-memory transformer on CiDEr and METEOR, motivating its use as the main 2D baseline.
- For 3D-to-2D captioning, detected 3D boxes are projected into rendered single-view images using estimated camera viewpoints and ScanNet camera intrinsics.
C.1. Additional quantitative analysis
Feature ablations show that adding colors and normals to geometry and multi-view features produces the best captioning performance.
- Geometry, multi-view features, and normal vectors achieve the best performance among the evaluated feature ablations.The configuration is labeled xyz+multiview+normal.
C.2. Additional qualitative analysis
Qualitative analyses examine feature choices and captioning components, showing that relational graph reasoning improves descriptions of object appearance and spatial relationships.
- All three compared methods produce good bounding boxes with IoU>0.5, but their captioning quality differs.
- VoteNet+GRU makes mistakes describing target objects, while VoteNet+CAC refers to nearby objects but misses relative spatial relationships.
- VoteNet+RG+CAC properly handles both target appearance and relative spatial relationships in the local environment.
- Table 9 reports standard captioning metrics separately for predicted boxes with IoU greater than 0.25 and 0.5, where higher values are better.