PaperlayerView exact revision on arXiv

Source-linked AI summary

Center-based 3D Object Detection and Tracking

Tianwei Yin, Xingyi Zhou, Philipp Krähenbühl

arXiv:2006.11275v2cs.CV

TL;DR

3D box detection is challenged by arbitrary object orientations and sparse point-clouds. CenterPoint detects object centers, regresses their properties, refines them with a second stage, and tracks them by velocity-based closest matching. It achieves state-of-the-art detection and tracking on Waymo and nuScenes benchmarks.

  • Problem

    3D detection is difficult because point-clouds are sparse and axis-aligned box representations poorly fit objects with varied orientations, sizes, and shapes.

  • Method

    CenterPoint detects centers with a keypoint heatmap, regresses size, orientation, and velocity, refines estimates using additional point features, and matches centers for tracking.

  • Results

    CenterPoint achieves state-of-the-art performance on Waymo and nuScenes for 3D detection and tracking, including 65.5 NDS and 63.8 AMOTA on nuScenes.

  • Takeaways & Limitations

    A center-based representation supports a simple, near-real-time system for simultaneous 3D object detection and tracking.

  • Takeaways & Limitations

    Two-stage refinement does not improve over single-stage CenterPoint on nuScenes, possibly because its 32-lane Lidar provides only about 30k points per frame.

Abstract

from arXiv · show

Three-dimensional objects are commonly represented as 3D boxes in a point-cloud. This representation mimics the well-studied image-based 2D bounding-box detection but comes with additional challenges. Objects in a 3D world do not follow any particular orientation, and box-based detectors have difficulties enumerating all orientations or fitting an axis-aligned bounding box to rotated objects. In this paper, we instead propose to represent, detect, and track 3D objects as points. Our framework, CenterPoint, first detects centers of objects using a keypoint detector and regresses to other attributes, including 3D size, 3D orientation, and velocity. In a second stage, it refines these estimates using additional point features on the object. In CenterPoint, 3D object tracking simplifies to greedy closest-point matching. The resulting detection and tracking algorithm is simple, efficient, and effective. CenterPoint achieved state-of-the-art performance on the nuScenes benchmark for both 3D detection and tracking, with 65.5 NDS and 63.8 AMOTA for a single model. On the Waymo Open Dataset, CenterPoint outperforms all previous single model method by a large margin and ranks first among all Lidar-only submissions. The code and pretrained models are available at https://github.com/tianweiy/CenterPoint.

1. Introduction

CenterPoint replaces orientation-sensitive 3D boxes with object centers, then predicts object properties and refines them in a second stage. This representation also simplifies tracking through velocity-based point matching and achieves strong results on Waymo and nuScenes.

  • 3D point-cloud detection is difficult because measurements are sparse, objects vary widely in shape and size, and boxes are not aligned to a global frame.
  • CenterPoint represents objects as points, avoiding intrinsic orientation and reducing the detector’s search space for rotated objects.
  • The detector finds object centers with a keypoint head and regresses 3D size, orientation, and velocity from center features.
  • A lightweight second stage extracts features from estimated object-face centers to refine object locations and other estimates.
  • 63.8 AMOTA is achieved on nuScenes, exceeding the prior state of the art by 8.8 AMOTA, while the system runs at 11 FPS on Waymo and 16 FPS on nuScenes.

2. Related work

Prior 3D detectors largely adapt 2D box-based or voting approaches to point-clouds. CenterPoint instead uses center-based detection and velocity-linked center tracking.

  • 2D detectors use candidate boxes or directly estimate object centers, and many 3D detectors inherit these design patterns.
  • VoxelNet, SECOND, and PointPillars encode sparse point-clouds into voxel or pillar features before producing detections.
  • VoteNet detects objects through vote clustering, whereas CenterPoint directly regresses 3D boxes from center-point features without voting.
  • CenterPoint uses velocity estimates to match object centers across frames, avoiding a separate dedicated tracker.

3. Preliminaries

The preliminaries describe keypoint-based detection and the limitations of anchor-based 3D boxes. CenterPoint addresses these limitations with center heatmaps and dense regressions from map-view features.

  • CenterNet-style detection identifies objects as local maxima in class heatmaps and retrieves regression values at those peak locations.
  • A 3D object box is parameterized by its center, size, and yaw rotation from an orderless point-cloud.
  • Modern 3D encoders quantize point-clouds into regular bins, pool point features, and produce map-view feature maps.
  • Anchor-based 3D detectors struggle with varying object sizes and orientations, and their IoU-based target assignment requires class- and dataset-specific thresholds.

4. CenterPoint

CenterPoint detects 3D object centers and regresses dense box attributes, then refines proposals with point features and IoU-guided confidence. Its tracking stage projects centers using velocity and matches them greedily to prior tracks.

  • First-stage detection: The first stage predicts class heatmaps, object size, sub-voxel location refinement, rotation, and velocity from map-view backbone features.Center heatmaps are trained from projected 3D box centers, while regression heads recover full 3D state information.
  • Tracking: At inference, CenterPoint projects current centers backward using negative velocity and matches them to tracked objects by closest distance.Unmatched tracks are retained for up to three frames before deletion.
  • Motivation for refinement: The center-feature design can lack sufficient localization information when the sensor observes only an object’s side, motivating the refinement stage.This limitation concerns inferring all object properties from a single center feature.
  • Two-stage refinement: CenterPoint extracts features at the predicted box center and four outward-facing face centers for second-stage refinement.The second stage uses a shared MLP to predict box regression refinements and an IoU-guided confidence score.
  • Two-stage refinement: The refinement confidence target is derived from proposal 3D IoU, and the final confidence combines first- and second-stage scores geometrically.The second-stage confidence is trained with binary cross entropy, while box refinement uses an L1 loss.

5. Experiments

CenterPoint is evaluated on Waymo and nuScenes for 3D detection and tracking, including comparisons with prior methods and anchor-based baselines. Center-based detection improves performance across backbones and object geometries, while velocity-based matching provides accurate, low-cost tracking.

  • Experimental setup: CenterPoint evaluates 3D detection and tracking on Waymo and nuScenes using VoxelNet and PointPillars encoders.The experiments use official benchmark datasets and metrics, including mAPH, mAP, NDS, MOTA, MOTP, and AMOTA.
  • Main results: 71.8 level 2 mAPH for vehicles and 66.4 level 2 mAPH for pedestrians are achieved on the Waymo test set, surpassing previous methods by 7.1% and 10.6% mAPH, respectively.These results are reported for vehicle and pedestrian detection.
  • Main results: 5.2% mAP and 2.2% NDS improvements over CBGS are achieved on the nuScenes test set.The comparison is against the previous challenge winner using multi-scale inputs and a multi-model ensemble.
  • Tracking results: 19.4 and 18.9 MOTA improvements are reported for Waymo vehicle and pedestrian tracking, while nuScenes tracking exceeds the prior challenge winner by 8.8 AMOTA.Tracking uses velocity-based closest-distance matching and adds only 1ms over detection.
  • Center-based versus anchor-based: 4.3 mAPH and 4.5 mAPH gains result from switching from anchors to centers with VoxelNet and PointPillars on Waymo.On nuScenes, CenterPoint improves anchor-based counterparts by 3.8-4.1 mAP and 1.1-1.8 NDS across backbones.
  • Center-based versus anchor-based: Center-based detectors perform better than anchor-based baselines for rotated objects and objects deviating from average size, indicating rotation and size invariance.The comparison uses subsets organized by heading angle and object size on Waymo validation.
  • Ablation studies: Two-stage refinement adds 6ms-7ms overhead and improves both 3D encoders, but does not improve the single-stage model on nuScenes.The authors attribute the nuScenes result to its lower-density 32-lane Lidar, producing about 30k points per frame.
  • Tracking ablation: 3.7 AMOTA improvement and 73ms versus 1ms runtime are reported for velocity-based closest-point matching over Kalman filter-based matching.The approach models learned point velocity and matches center points without hidden-state computation.

6. Conclusion

CenterPoint is a center-based framework for simultaneous 3D object detection and tracking from Lidar point-clouds, using dense bird’s-eye-view predictions, refinement, and closest-distance matching.

  • CenterPoint uses a standard 3D point-cloud encoder and convolutional head to produce a bird’s-eye-view heatmap and dense regression outputs.
  • Detection extracts local peaks and refines their estimates, while tracking links objects through closest-distance matching.
  • CenterPoint is described as simple, near real-time, and state-of-the-art on the Waymo and nuScenes benchmarks.

A. Tracking algorithm

The center-based tracking algorithm maintains tracked objects and associates current detections using confidence ordering, distance thresholds, and track-age management.

  • Tracked objects store center, ground-plane velocity, category, box attributes, tracking ID, and inactive age.
  • Current-frame detections are processed in descending confidence and emitted as the updated track set.
  • Matching uses a distance threshold τ and limits how long inactive tracks remain available.
  • The algorithm initializes tracks and matches as empty sets before associating detections or creating new tracks.
  • Matched tracks update their center locations during the association process.

B. Implementation Details

The implementation builds on established Lidar backbones, standardizes dataset-specific training and augmentation, and applies additional test-time refinements for nuScenes.

  • The implementation uses open-sourced CBGS code with PointPillars and VoxelNet implementations for nuScenes.
  • NuScenes experiments merge Lidar points from non-annotated frames into following annotated frames to densify inputs and support velocity estimation.
  • Augmentation includes independent X/Y flipping, global scaling, dataset-specific rotation ranges, and ground-truth sampling on nuScenes.
  • NuScenes models use AdamW, a one-cycle learning-rate policy, maximum learning rate 1e-3, weight decay 0.01, and 20 epochs on 4 V100 GPUs.
  • Waymo models use maximum learning rate 3e-3, train for 30 epochs, and use 6-epoch fine-tuning for refinement ablations.
  • NuScenes test-time changes improve CenterPoint-Voxel from 64.8 NDS to 65.4 NDS, while flip testing yields 66.5 AMOTA on validation.

C. nuScenes Performance across classes

The section directs readers to Table 13 for per-class comparisons between CenterPoint and state-of-the-art methods.

  • Table 13 reports per-class comparisons with state-of-the-art methods.

D. nuScenes Detection Challenge

CenterPoint’s nuScenes challenge submission combined several complementary enhancements and achieved the strongest reported challenge results. The progression isolates gains from multimodal point annotation, test-time augmentation, and model ensembling.

  • CenterPoint was used by three of the top four entries in the NeurIPS 2020 nuScenes detection challenge.
  • CenterPoint improved the 2019 nuScenes challenge winner CBGS by 14.3 mAP and 8.1 NDS.
  • PointPainting increased validation NDS from 65.4 to 68.0 by augmenting lidar points with image-based instance-segmentation results.
  • Double-flip testing and yaw-rotation augmentation raised NDS from 68.0 to 70.3.
  • An ensemble of five models with varying input grid sizes achieved 68.2 mAP and 71.7 NDS on nuScenes validation.
Loading 2006.11275v2…