PaperlayerView exact revision on arXiv

Source-linked AI summary

An End-to-End Transformer Model for 3D Object Detection

Ishan Misra, Rohit Girdhar, Armand Joulin

arXiv:2109.08141v1cs.CVcs.AIcs.LG

TL;DR

3D point-cloud detection has relied on specialized architectures and hand-designed inductive biases, motivating a simpler set-to-set alternative. 3DETR uses an end-to-end Transformer with non-parametric queries and Fourier positional embeddings, and achieves strong results across indoor detection benchmarks while remaining flexible for other 3D tasks.

  • Problem

    3DETR addresses whether Transformers can perform 3D detection on unordered point clouds without hand-designed 3D inductive biases.

    Source: §1 · p. 1

  • Method

    3DETR replaces PointNet++ with a Transformer encoder and uses a parallel Transformer decoder with non-parametric queries and Fourier positional embeddings.

    Source: §1 · p. 1, §4.2.2 · p. 7

  • Results

    3DETR achieves 65.0% AP on ScanNetV2 and 59.0% AP on SUN RGB-D, outperforming an improved VoteNet baseline by 9.5% AP50 on ScanNetV2.

    Source: §1 · p. 2

  • Takeaways & Limitations

    3DETR is a flexible framework that supports variable numbers of predictions at test time, incorporates 3D components, and benefits other 3D tasks such as shape classification.

    Source: §6 · p. 8

Abstract

from arXiv · show

We propose 3DETR, an end-to-end Transformer based object detection model for 3D point clouds. Compared to existing detection methods that employ a number of 3D-specific inductive biases, 3DETR requires minimal modifications to the vanilla Transformer block. Specifically, we find that a standard Transformer with non-parametric queries and Fourier positional embeddings is competitive with specialized architectures that employ libraries of 3D-specific operators with hand-tuned hyperparameters. Nevertheless, 3DETR is conceptually simple and easy to implement, enabling further improvements by incorporating 3D domain knowledge. Through extensive experiments, we show 3DETR outperforms the well-established and highly optimized VoteNet baselines on the challenging ScanNetV2 dataset by 9.5%. Furthermore, we show 3DETR is applicable to 3D tasks beyond detection, and can serve as a building block for future research.

1. Introduction

3DETR asks whether Transformers can detect 3D objects from unordered point clouds while avoiding hand-designed 3D inductive biases. It uses an end-to-end encoder-decoder Transformer with non-parametric queries and Fourier positional embeddings, achieving strong benchmark performance.

  • Point clouds are unordered, sparse, and irregular, making 3D detection different from conventional grid-based vision.Source: §1 · p. 1
  • Existing set-to-set 3D detectors rely on carefully designed inductive biases, radii, specialized operators, and loss functions.Source: §1 · p. 1
  • 3DETR tests whether Transformers can process unordered 3D point clouds without relying on hand-designed inductive biases.Source: §1 · p. 1
  • 3DETR replaces PointNet++ with a standard Transformer encoder and adapts the decoder using non-parametric queries and Fourier positional embeddings.Source: §1 · pp. 1–2
  • 65.0% AP on ScanNetV2 and 59.0% AP on SUN RGB-D were achieved, with a 9.5% AP50 improvement over an improved VoteNet baseline on ScanNetV2.Source: §1 · p. 2

2. Related Work

Prior 3D detection methods use grid, graph, convolutional, voting, or multi-stage architectures to process point clouds. 3DETR instead builds on the Transformer’s permutation-invariant set processing and long-range self-attention.

  • Grid-based methods convert 3D data into 2D projections or voxelized 3D grids before applying convolutional networks.Source: §2 · p. 2
  • Graph-based models construct relationships from local neighborhoods, attribute similarity, or point-edge interactions.Source: §2 · p. 2
  • Continuous point convolutions generate weights from coordinates or use soft assignments and specific point orderings.Source: §2 · p. 2
  • Direct point-cloud detectors include two-stage pipelines and single-stage models such as VoteNet, which uses Hough voting, feature grouping, and hand-designed 3D operations.Source: §2 · p. 2
  • 3DETR builds on Transformers because self-attention is applicable to unordered sets and captures large contexts.Source: §2 · p. 2

3. Approach

3DETR encodes sampled point features with Transformer blocks and predicts an unordered set of boxes through parallel query-based decoding. It trains by matching predictions to ground truth with a generic bipartite objective.

3. Approach

  • 3DETR takes a 3D point cloud and predicts object locations as 3D bounding boxes using Transformer encoder-decoder blocks.Source: §3.2 · p. 3

3.2. 3DETR: Encoder-decoder Transformer

  • The encoder produces point features, while the parallel decoder uses query embeddings to produce multiple boxes without a prescribed ordering.Source: §3.2 · p. 3
  • Queries are sampled from input points with farthest-point sampling and converted into embeddings through Fourier positional encodings and an MLP.Source: §3.2 · pp. 3–4

3.5. 3DETR-m

  • 3DETR-m demonstrates that 3D inductive biases can be added by masking encoder self-attention while keeping the decoder and loss fixed.Source: §3.2 · p. 4

3.3. Bounding box parametrization and prediction

  • Each query produces a box defined by center, dimensions, orientation, and semantic class, with an additional background class for unmatched predictions.Source: §3.3 · p. 4

3.4. Set Matching and Loss Function

  • Predicted and ground-truth boxes are paired through generic bipartite matching rather than VoteNet’s hand-defined radii.Source: §3.4 · p. 4
  • The Hungarian algorithm matches all predictions to ground-truth boxes, assigning unmatched predictions to the background class to discourage over-prediction.Source: §3.4 · p. 4

4. Experiments

Experiments evaluate 3DETR on indoor 3D detection, module interchangeability, design choices, and transfer beyond detection. Across these settings, Transformer-based components achieve competitive detection and shape-classification performance with fewer 3D-specific design decisions.

3DETR on 3D Detection

  • 3DETR achieves +2.3% AP25 over VoteNet on ScanNetV2 but −1.5% AP25 on SUN RGB-D.Source: §4.1 · p. 5
  • 3DETR-m gains +4.6% AP25 and +9.5% AP50 over VoteNet on ScanNetV2 while remaining comparable to VoteNet on SUN RGB-D.Source: §4.1 · p. 5
  • 3DETR significantly outperforms BoxNet by +13% AP25 on ScanNetV2 and +3.9% AP25 on SUN RGB-D.Source: §4.1 · p. 5

Qualitative Results

  • 3DETR detects objects from partial single-view depth scans and predicts amodal boxes or objects missing from SUN RGB-D annotations.Source: §4.1 · p. 5, Figure 3 · p. 6

Modules of VoteNet and BoxNet vs. 3DETR

  • Replacing the Transformer encoder with PointNet++ leaves performance unchanged or slightly worsens it, indicating compatibility with prior 3D-specific encoders.Source: §4.2.1 · p. 6, Table 2 · p. 6
  • Using Box or Vote losses with the Transformer encoder-decoder causes similar degradation, suggesting these losses depend on PointNet++-specific design choices.Source: §4.2.1 · p. 6, Table 3 · p. 6

Shape Classification

  • 3DETR and 3DETR-m encoders perform competitively for shape classification, suggesting applicability beyond detection.Source: §4.2.1 · p. 7, Table 4 · p. 7

Design decisions in 3DETR

  • Replacing parametric queries with non-parametric queries improves ScanNetV2 AP25 by +40%, while Fourier positional embeddings add +5% AP25.Source: §4.2.2 · p. 7

5. Ablations

The ablations examine how 3DETR’s set loss, layer allocation, and test-time decoder settings affect detection performance and computation. They show that decoder depth matters more than encoder depth, while decoder layers and non-parametric queries support inference-time adaptation without retraining.

Set loss and NMS

  • 3DETR’s set loss limits the performance drop without NMS to 3% AP, compared with 50% for VoteNet.The loss enforces one-to-one matching between ground-truth and predicted boxes, reducing duplicate detections.Source: §5 · pp. 7–8

Encoder and decoder layers

  • Adding decoder layers improves AP50 more than adding encoder layers: +7% versus +2% for five additional layers.The decoder’s greater impact is attributed to iterative refinement of bounding-box predictions.Source: §5 · p. 8, Figure 4 · p. 8

Adapting decoder depth

  • 3DETR can reduce decoder depth at test time without retraining, performing as well as or better than separately trained reduced-depth models.Intermediate decoder layers are trained to predict boxes using the same prediction MLPs.Source: §5.1 · p. 8, Figure 5 · p. 8

Adapting number of queries

  • Increasing the number of queries produces more boxes and better performance but increases running time, enabling a performance–runtime trade-off at inference.Non-parametric queries allow the number of predictions to change at test time.Source: §5.1 · p. 8, Figure 5 · p. 8

6. Conclusion

The conclusion presents 3DETR as a flexible Transformer framework for point-cloud detection that uses few 3D-specific design decisions. Its design supports variable inference computation, incorporates prior 3D components, and benefits other 3D tasks.

  • 3DETR requires few 3D-specific design decisions or hyperparameters while supporting end-to-end Transformer-based point-cloud detection.Source: §6 · p. 8
  • Non-parametric queries and Fourier encodings are critical for good 3D detection performance and also benefit shape classification.Source: §6 · p. 8
  • 3DETR’s set loss generalizes to prior 3D architectures, and its framework can incorporate existing detection components.Source: §6 · p. 8
  • 3DETR combines variable numbers of predictions at test time with variable decoder-layer counts.Source: §6 · p. 8

A.1. Architecture

The architecture samples point features, processes them with Transformer encoder layers, and decodes box predictions through query–point attention. Training uses set matching between predicted and ground-truth boxes, with background predictions receiving semantic loss.

Architecture

  • The architecture starts from 20,000 or 40,000 input points, samples 2,048 points, and projects them into 256-dimensional features.The respective input sizes are used for SUN RGB-D and ScanNetV2.Source: §A.1 · p. 9

Encoder

  • Each encoder layer maps N′ × d point features to N′ × d features using self-attention followed by an MLP.Source: Figure 6 · p. 9

Decoder

  • The decoder maps B query features and N′ encoder point features to B box features through decoder self-attention and cross-attention.Source: Figure 6 · p. 9

Set matching

  • For B predicted and G ground-truth boxes, 3DETR computes a B × G cost matrix and uses the Hungarian algorithm for optimal assignment.The remaining B−G predictions are matched to background.Source: §A.2 · p. 10

Training loss

  • Matched predictions use box and semantic losses, while unmatched background predictions receive only weighted semantic cross-entropy loss.The background class has weight 0.2 and the K object classes have weight 0.8.Source: §A.2 · p. 10

B. Experiments

The appendix describes tuned baseline training and reports that these changes improve VoteNet and BoxNet, supporting fairer comparisons with 3DETR. The baselines use extended training and a scheduled Adam learning rate.

Improved baselines

  • The VoteNet and BoxNet baselines were improved through grid search and optimization-hyperparameter tuning.Source: §B.1 · p. 10

Training setup

  • The improved baselines were trained for 360 epochs with Adam and a learning rate of 1 × 10−3, decayed after 160, 240, and 320 epochs.Source: §B.1 · p. 10
  • Cosine schedules, training beyond 360 epochs, and AdamW did not significantly change baseline performance.Source: §B.1 · p. 10

Improved baselines

B.2. Per-class Results

The appendix reports per-class AP25 results for ScanNetV2 and SUN RGB-D and gives a detailed comparison with state-of-the-art 3D detection methods. Most compared methods build on VoteNet, whereas 3DETR avoids voting and uses fewer 3D-specific design decisions.

  • Per-class mAP results are provided for ScanNetV2 and SUN RGB-D, complementing the overall results reported in the main paper.Source: §B.2 · p. 10
  • Most state-of-the-art detection methods build upon VoteNet through 3D primitives, hierarchical graphs, or geometric aggregation.Source: §B.3 · p. 10
  • 3DETR does not use voting and makes fewer 3D-specific decisions than the compared detection methods.Source: §B.3 · p. 10

B.4. 3DETR-m with Vote loss

Using the 3DETR-m encoder, a tuned VoteNet loss achieves strong detection results, but the 3DETR set loss remains better. Additional experiments show query-count robustness depends on training choices, while encoder attention focuses on object parts.

3DETR-m with Vote loss

  • 60.7% and 56.1% mAP are achieved on ScanNetV2 and SUN RGB-D, respectively, with a tuned VoteNet loss and the 3DETR-m encoder.Source: §B.4 · p. 10
  • The VoteNet loss gains 6% and 3% with 3DETR-m compared with the vanilla 3DETR encoder, indicating dependence on encoder inductive biases and design.Source: §B.4 · p. 10
  • The 3DETR set loss still performs better than the VoteNet loss for 3DETR-m.Source: §B.4 · p. 10

Query adaptation

  • A model trained with 64 queries is more robust to test-time query changes, but its optimal detection performance is worse than that of a model trained with 256 queries.Source: §B.5 · p. 11, Figure 7 · p. 11

Encoder attention

  • Encoder attention groups geometric parts across multiple objects or focuses on a single part of an instance, such as a chair backrest.Source: Figure 8 · p. 11, §B.6 · p. 11

Related papers

Loading 2109.08141v1…