Source-linked AI summary
An End-to-End Transformer Model for 3D Object Detection
Ishan Misra, Rohit Girdhar, Armand Joulin
TL;DR
3D point-cloud detection has relied on specialized architectures and hand-designed inductive biases, motivating a simpler set-to-set alternative. 3DETR uses an end-to-end Transformer with non-parametric queries and Fourier positional embeddings, and achieves strong results across indoor detection benchmarks while remaining flexible for other 3D tasks.
Problem
3DETR addresses whether Transformers can perform 3D detection on unordered point clouds without hand-designed 3D inductive biases.
Method
3DETR replaces PointNet++ with a Transformer encoder and uses a parallel Transformer decoder with non-parametric queries and Fourier positional embeddings.
Results
3DETR achieves 65.0% AP on ScanNetV2 and 59.0% AP on SUN RGB-D, outperforming an improved VoteNet baseline by 9.5% AP50 on ScanNetV2.
Takeaways & Limitations
3DETR is a flexible framework that supports variable numbers of predictions at test time, incorporates 3D components, and benefits other 3D tasks such as shape classification.
Abstract
from arXiv · showhide
We propose 3DETR, an end-to-end Transformer based object detection model for 3D point clouds. Compared to existing detection methods that employ a number of 3D-specific inductive biases, 3DETR requires minimal modifications to the vanilla Transformer block. Specifically, we find that a standard Transformer with non-parametric queries and Fourier positional embeddings is competitive with specialized architectures that employ libraries of 3D-specific operators with hand-tuned hyperparameters. Nevertheless, 3DETR is conceptually simple and easy to implement, enabling further improvements by incorporating 3D domain knowledge. Through extensive experiments, we show 3DETR outperforms the well-established and highly optimized VoteNet baselines on the challenging ScanNetV2 dataset by 9.5%. Furthermore, we show 3DETR is applicable to 3D tasks beyond detection, and can serve as a building block for future research.
1. Introduction
3DETR asks whether Transformers can detect 3D objects from unordered point clouds while avoiding hand-designed 3D inductive biases. It uses an end-to-end encoder-decoder Transformer with non-parametric queries and Fourier positional embeddings, achieving strong benchmark performance.
- Point clouds are unordered, sparse, and irregular, making 3D detection different from conventional grid-based vision.
- Existing set-to-set 3D detectors rely on carefully designed inductive biases, radii, specialized operators, and loss functions.
- 3DETR tests whether Transformers can process unordered 3D point clouds without relying on hand-designed inductive biases.
- 3DETR replaces PointNet++ with a standard Transformer encoder and adapts the decoder using non-parametric queries and Fourier positional embeddings.
- 65.0% AP on ScanNetV2 and 59.0% AP on SUN RGB-D were achieved, with a 9.5% AP50 improvement over an improved VoteNet baseline on ScanNetV2.
2. Related Work
Prior 3D detection methods use grid, graph, convolutional, voting, or multi-stage architectures to process point clouds. 3DETR instead builds on the Transformer’s permutation-invariant set processing and long-range self-attention.
- Grid-based methods convert 3D data into 2D projections or voxelized 3D grids before applying convolutional networks.
- Graph-based models construct relationships from local neighborhoods, attribute similarity, or point-edge interactions.
- Continuous point convolutions generate weights from coordinates or use soft assignments and specific point orderings.
- Direct point-cloud detectors include two-stage pipelines and single-stage models such as VoteNet, which uses Hough voting, feature grouping, and hand-designed 3D operations.
- 3DETR builds on Transformers because self-attention is applicable to unordered sets and captures large contexts.
3. Approach
3DETR encodes sampled point features with Transformer blocks and predicts an unordered set of boxes through parallel query-based decoding. It trains by matching predictions to ground truth with a generic bipartite objective.
- 3. Approach: 3DETR takes a 3D point cloud and predicts object locations as 3D bounding boxes using Transformer encoder-decoder blocks.
- 3.2. 3DETR: Encoder-decoder Transformer: The encoder produces point features, while the parallel decoder uses query embeddings to produce multiple boxes without a prescribed ordering.
- 3.2. 3DETR: Encoder-decoder Transformer: Queries are sampled from input points with farthest-point sampling and converted into embeddings through Fourier positional encodings and an MLP.
- 3.5. 3DETR-m: 3DETR-m demonstrates that 3D inductive biases can be added by masking encoder self-attention while keeping the decoder and loss fixed.
- 3.3. Bounding box parametrization and prediction: Each query produces a box defined by center, dimensions, orientation, and semantic class, with an additional background class for unmatched predictions.
- 3.4. Set Matching and Loss Function: Predicted and ground-truth boxes are paired through generic bipartite matching rather than VoteNet’s hand-defined radii.
- 3.4. Set Matching and Loss Function: The Hungarian algorithm matches all predictions to ground-truth boxes, assigning unmatched predictions to the background class to discourage over-prediction.
4. Experiments
Experiments evaluate 3DETR on indoor 3D detection, module interchangeability, design choices, and transfer beyond detection. Across these settings, Transformer-based components achieve competitive detection and shape-classification performance with fewer 3D-specific design decisions.
- 3DETR on 3D Detection: 3DETR achieves +2.3% AP25 over VoteNet on ScanNetV2 but −1.5% AP25 on SUN RGB-D.
- 3DETR on 3D Detection: 3DETR-m gains +4.6% AP25 and +9.5% AP50 over VoteNet on ScanNetV2 while remaining comparable to VoteNet on SUN RGB-D.
- 3DETR on 3D Detection: 3DETR significantly outperforms BoxNet by +13% AP25 on ScanNetV2 and +3.9% AP25 on SUN RGB-D.
- Qualitative Results: 3DETR detects objects from partial single-view depth scans and predicts amodal boxes or objects missing from SUN RGB-D annotations.
- Modules of VoteNet and BoxNet vs. 3DETR: Replacing the Transformer encoder with PointNet++ leaves performance unchanged or slightly worsens it, indicating compatibility with prior 3D-specific encoders.
- Modules of VoteNet and BoxNet vs. 3DETR: Using Box or Vote losses with the Transformer encoder-decoder causes similar degradation, suggesting these losses depend on PointNet++-specific design choices.
- Shape Classification: 3DETR and 3DETR-m encoders perform competitively for shape classification, suggesting applicability beyond detection.
- Design decisions in 3DETR: Replacing parametric queries with non-parametric queries improves ScanNetV2 AP25 by +40%, while Fourier positional embeddings add +5% AP25.
5. Ablations
The ablations examine how 3DETR’s set loss, layer allocation, and test-time decoder settings affect detection performance and computation. They show that decoder depth matters more than encoder depth, while decoder layers and non-parametric queries support inference-time adaptation without retraining.
- Set loss and NMS: 3DETR’s set loss limits the performance drop without NMS to 3% AP, compared with 50% for VoteNet.The loss enforces one-to-one matching between ground-truth and predicted boxes, reducing duplicate detections.
- Encoder and decoder layers: Adding decoder layers improves AP50 more than adding encoder layers: +7% versus +2% for five additional layers.The decoder’s greater impact is attributed to iterative refinement of bounding-box predictions.
- Adapting decoder depth: 3DETR can reduce decoder depth at test time without retraining, performing as well as or better than separately trained reduced-depth models.Intermediate decoder layers are trained to predict boxes using the same prediction MLPs.
- Adapting number of queries: Increasing the number of queries produces more boxes and better performance but increases running time, enabling a performance–runtime trade-off at inference.Non-parametric queries allow the number of predictions to change at test time.
6. Conclusion
The conclusion presents 3DETR as a flexible Transformer framework for point-cloud detection that uses few 3D-specific design decisions. Its design supports variable inference computation, incorporates prior 3D components, and benefits other 3D tasks.
- 3DETR requires few 3D-specific design decisions or hyperparameters while supporting end-to-end Transformer-based point-cloud detection.
- Non-parametric queries and Fourier encodings are critical for good 3D detection performance and also benefit shape classification.
- 3DETR’s set loss generalizes to prior 3D architectures, and its framework can incorporate existing detection components.
- 3DETR combines variable numbers of predictions at test time with variable decoder-layer counts.
A.1. Architecture
The architecture samples point features, processes them with Transformer encoder layers, and decodes box predictions through query–point attention. Training uses set matching between predicted and ground-truth boxes, with background predictions receiving semantic loss.
- Architecture: The architecture starts from 20,000 or 40,000 input points, samples 2,048 points, and projects them into 256-dimensional features.The respective input sizes are used for SUN RGB-D and ScanNetV2.
- Encoder: Each encoder layer maps N′ × d point features to N′ × d features using self-attention followed by an MLP.
- Decoder: The decoder maps B query features and N′ encoder point features to B box features through decoder self-attention and cross-attention.
- Set matching: For B predicted and G ground-truth boxes, 3DETR computes a B × G cost matrix and uses the Hungarian algorithm for optimal assignment.The remaining B−G predictions are matched to background.
- Training loss: Matched predictions use box and semantic losses, while unmatched background predictions receive only weighted semantic cross-entropy loss.The background class has weight 0.2 and the K object classes have weight 0.8.
B. Experiments
The appendix describes tuned baseline training and reports that these changes improve VoteNet and BoxNet, supporting fairer comparisons with 3DETR. The baselines use extended training and a scheduled Adam learning rate.
- Improved baselines: The VoteNet and BoxNet baselines were improved through grid search and optimization-hyperparameter tuning.
- Training setup: The improved baselines were trained for 360 epochs with Adam and a learning rate of 1 × 10−3, decayed after 160, 240, and 320 epochs.
- Training setup: Cosine schedules, training beyond 360 epochs, and AdamW did not significantly change baseline performance.
- Improved baselines: The baseline improvements were summarized in Table 8 and intended to make the main-paper comparisons fair.
B.2. Per-class Results
The appendix reports per-class AP25 results for ScanNetV2 and SUN RGB-D and gives a detailed comparison with state-of-the-art 3D detection methods. Most compared methods build on VoteNet, whereas 3DETR avoids voting and uses fewer 3D-specific design decisions.
- Per-class mAP results are provided for ScanNetV2 and SUN RGB-D, complementing the overall results reported in the main paper.
- Most state-of-the-art detection methods build upon VoteNet through 3D primitives, hierarchical graphs, or geometric aggregation.
- 3DETR does not use voting and makes fewer 3D-specific decisions than the compared detection methods.
B.4. 3DETR-m with Vote loss
Using the 3DETR-m encoder, a tuned VoteNet loss achieves strong detection results, but the 3DETR set loss remains better. Additional experiments show query-count robustness depends on training choices, while encoder attention focuses on object parts.
- 3DETR-m with Vote loss: 60.7% and 56.1% mAP are achieved on ScanNetV2 and SUN RGB-D, respectively, with a tuned VoteNet loss and the 3DETR-m encoder.
- 3DETR-m with Vote loss: The VoteNet loss gains 6% and 3% with 3DETR-m compared with the vanilla 3DETR encoder, indicating dependence on encoder inductive biases and design.
- 3DETR-m with Vote loss: The 3DETR set loss still performs better than the VoteNet loss for 3DETR-m.
- Query adaptation: A model trained with 64 queries is more robust to test-time query changes, but its optimal detection performance is worse than that of a model trained with 256 queries.
- Encoder attention: Encoder attention groups geometric parts across multiple objects or focuses on a single part of an instance, such as a chair backrest.