PaperlayerView exact revision on arXiv

Source-linked AI summary

Offboard 3D Object Detection from Point Cloud Sequences

Charles R. Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, Dragomir Anguelov

arXiv:2103.05073v1cs.CV

TL;DR

The paper addresses the limited high-quality perception available for offboard 3D uses such as automatic labeling. It proposes 3D Auto Labeling, which combines sequence-based multi-frame detection with object-centric refinement, and reports gains over onboard detectors, human-comparable labels, and semi-supervised-learning utility on Waymo.

  • Problem

    Real-time onboard 3D detectors use limited temporal input and speed-constrained designs, leaving high-quality offboard perception and automatic 3D labeling under-explored.

  • Method

    3D Auto Labeling uses a multi-frame detector, tracking, and object-centric models to process each object’s long-term 4D track data into refined boxes.

  • Results

    On Waymo, 3D Auto Labeling achieves significant gains over onboard detectors, reaches comparable or slightly better performance than experienced human labels, and improves student models in semi-supervised settings.

  • Takeaways & Limitations

    The results support using high-quality sequence-based auto labels for offboard perception and for training models when human labels are limited or domains are unseen.

  • Takeaways & Limitations

    The qualitative analysis reports failures on occluded or sparse objects and rare pedestrian poses, with camera information suggested as a potential remedy.

Abstract

from arXiv · show

While current 3D object recognition research mostly focuses on the real-time, onboard scenario, there are many offboard use cases of perception that are largely under-explored, such as using machines to automatically generate high-quality 3D labels. Existing 3D object detectors fail to satisfy the high-quality requirement for offboard uses due to the limited input and speed constraints. In this paper, we propose a novel offboard 3D object detection pipeline using point cloud sequence data. Observing that different frames capture complementary views of objects, we design the offboard detector to make use of the temporal points through both multi-frame object detection and novel object-centric refinement models. Evaluated on the Waymo Open Dataset, our pipeline named 3D Auto Labeling shows significant gains compared to the state-of-the-art onboard detectors and our offboard baselines. Its performance is even on par with human labels verified through a human label study. Further experiments demonstrate the application of auto labels for semi-supervised learning and provide extensive analysis to validate various design choices.

1. Introduction

The paper formulates offboard 3D detection for high-quality perception and proposes 3D Auto Labeling, which exploits whole point-cloud sequences through multi-frame detection and object-centric refinement. On Waymo, it reports gains over onboard detectors, human-comparable labels, and usefulness for semi-supervised learning.

  • Motivation: Offboard perception targets high-quality applications such as automatic 3D labeling, simulation, and datasets for supervising or evaluating downstream modules.Most existing 3D perception research instead focuses on real-time onboard use cases with current-frame or limited history input.
  • Approach: 3D Auto Labeling formulates offboard 3D detection as a modular pipeline using whole sensor sequences, multi-frame detection, and object-centric auto-labeling models.The design expands the available input and relaxes causality and inference-speed constraints.
  • Approach: Different viewpoints provide complementary geometric information, but frame-based multi-frame detectors gain diminishing returns as more frames are added.This motivates processing longer temporal context at the object level rather than merging entire frames.
  • Approach: The pipeline localizes objects with a multi-frame detector, links them across frames, extracts 4D object tracks, and refines them into temporally consistent high-quality boxes.Object-centric models process each object’s point clouds and detector boxes across the sequence.
  • Results: On the Waymo Open Dataset, 3D Auto Labeling dramatically improves perception quality over real-time onboard detectors, with larger gains at higher standards.The human label study found auto labels on par with or slightly better than experienced human labels on selected test segments.
  • Results: Auto labels also support semi-supervised learning, producing significantly improved student models under low-label and unseen-domain settings.The paper additionally reports ablations and analyses validating its design choices.

2. Related Work

Prior work largely addresses single-frame or short-term temporal 3D detection and has explored sequence learning, multimodal inputs, and auto-labeling. This paper focuses on long temporal intervals with point clouds and object-centric processing to improve 3D labeling precision.

  • 3D object detection: Most 3D object detectors use single-frame inputs, with voxel-, point-, perspective-view-, or hybrid representations.Recent methods aggregate temporal Lidar information for densification, shape completion, or spatiotemporal feature encoding.
  • 3D object detection: Multimodal camera-and-point-cloud methods improve 3D detection, especially for small and far-away objects, whereas this work uses point clouds alone over long temporal intervals.The paper distinguishes its point-cloud-only, long-context focus from multimodal approaches.
  • Learning from point cloud sequences: Point-cloud sequence learning has been applied to scene flow, action recognition, and semantic segmentation, but this paper applies long sequences to 3D object detection.Prior work shows that longer sequence inputs can improve several point-cloud tasks.
  • Auto labeling: Existing 3D auto-labeling methods can provide loose localization at IoU 0.5 but leave a considerable gap for applications requiring higher precision.Other methods assist human annotation or reduce required supervision rather than directly solving high-precision offboard labeling.

3. Offboard 3D Object Detection

Offboard 3D detection uses temporal data beyond fixed frame-centric designs, adapting the amount of context to each object through tracking and object-centric refinement.

  • The task uses sequences of point clouds with sensor poses to localize and classify objects in 3D for every frame.
  • Access to history and future sensor data expands the detector design space beyond single-frame input.
  • Multi-frame detectors improve quality but are difficult to scale beyond a few frames, cannot compensate object motion, and show diminishing gains as frames accumulate.
  • 3D Auto Labeling adaptively selects temporal context per object by detecting initial localizations, tracking objects, and refining each track.

4. 3D Auto Labeling Pipeline

The 3D Auto Labeling pipeline extracts object tracks from long point-cloud sequences, classifies motion, and applies separate static and dynamic box-refinement models.

  • MVF++: MVF++ aggregates multiple ego-motion-compensated LiDAR scans and adds temporal offsets before detection.
  • MVF++: Test-time augmentation rotates the point cloud at 10 angles and ensembles predictions, with parallelization supported in offboard use.
  • Multi-object tracking: The tracker links detections across frames using detector-box associations and Kalman-filter state updates.
  • Object track data extraction: Object track data combines object-specific point clouds and tracked boxes across visible frames after transforming them into world coordinates.
  • The pipeline uses modular detector, tracking, object-track extraction, motion classification, and static or dynamic auto-labeling modules.
  • Static object auto labeling: Static objects receive one box from merged world-coordinate points, which can then be transformed into each frame.
  • Static object auto labeling: The static model aligns points to a box coordinate, segments foreground points, iteratively regresses the box, and performs test-time augmentation.
  • Dynamic object auto labeling: Dynamic objects require per-frame boxes; the proposed model processes point and detector-box sequences in sliding windows to predict refined center-frame boxes.

5. Experiments

Experiments evaluate 3D Auto Labeling on Waymo across detection, human-label comparison, semi-supervised learning, and design ablations. The pipeline improves high-threshold detection, approaches human labeling quality, supports data-efficient training, and benefits from object-centric temporal context.

  • State-of-the-art comparison: 3D Auto Labeling further improves vehicle detection by 14.52 3D AP@0.8 points over single-frame MVF++ and 8.39 points over multi-frame MVF++.The comparison is especially strong at the higher IoU threshold of 0.8.
  • State-of-the-art comparison: 7.67 points at pedestrian IoU=0.6 separates 3D Auto Labeling from single-frame MVF++, while the gain over multi-frame MVF++ is 3.13 points.The pipeline consistently leads, particularly at the higher localization standard.
  • Human-label comparison: Auto labels are around 1 point below the average human labeler on 3D AP@0.7 (L1) and exceed average human labels by 1.28 BEV AP@0.8 points.The larger gap at 3D AP@0.8 is mostly attributed to inaccurate heights.
  • Semi-supervised learning: Using 10% human labels and 90% auto labels yields student-model AP gaps smaller than 1 point relative to training with 100% human labels.The setup uses 79 human-labeled sequences to train 3D Auto Labeling and 719 additional sequences for auto-label generation.
  • Semi-supervised learning: Auto-labeling an unseen domain improves student AP by approximately 5 points, whereas student self-labeling improves results by approximately 1 point.The teacher is trained on the main Waymo training set and labels adaptation-domain data.
  • Ablations and analysis: Adding frames has diminishing returns: AP@0.8 gains 0.81 from 1 to 2 frames but only 0.14 point from 4 to 5 frames.The ablations also report gains from increased model capacity, five-frame input, test-time augmentation, coordinate transformation, segmentation, and iterative refinement.
  • Ablations and analysis: Larger temporal contexts improve object box accuracy, with the entire object track producing the best performance.The analysis compares single-frame, multi-frame, and object-track contexts for static and dynamic objects.
  • Qualitative analysis: Qualitative failures remain for occluded objects, objects with very few points, pedestrians in rare poses, and geometrically similar false positives.The passage suggests added camera information through multimodal learning as a possible way to address hard cases.

6. Conclusion

The conclusion presents 3D Auto Labeling as a high-quality offboard detector that exploits long-term point-cloud sequences through multi-frame detection and object-centric refinement. Results on Waymo show gains over onboard detectors, human-comparable labels, and usefulness for low-label and unseen-domain student training.

  • Conclusion: 3D Auto Labeling is a state-of-the-art offboard 3D object detection solution using point cloud sequences as input.Its key components are an object-centric formulation, an offboard multi-frame detector, and object auto-labeling models.
  • Conclusion: The pipeline shows significant gains over prior onboard 3D detectors, especially under high-standard metrics.The test-set evaluation notes that the method peeks into the future for object-centric refinement, which is feasible offboard.
  • Conclusion: Auto labels reach comparable performance to experienced human labels in the reported human label study.The study compares auto labels with experienced human labels on Waymo sequences.
  • Conclusion: Semi-supervised experiments show auto labels are useful for student training in low-label regimes and unseen domains.The conclusion identifies both settings as applications of the generated labels.

B.2. Domain Adaptation Results

Domain-adaptation experiments compare 3D Auto Labeling with PointPillars and multi-frame MVF++ on a Waymo adaptation test set, and also examine downstream tracking. The pipeline achieves substantially higher detection AP and improved tracking when its refined boxes are used.

  • Domain adaptation: 3D Auto Labeling achieves 32.56 higher 3D AP than PointPillars and 8.03 higher 3D AP than multi-frame MVF++ on the domain-adaptation test set.The models are trained on the main Waymo training set, while evaluation uses the adaptation test set.
  • Domain adaptation: Temporal information aggregation likely compensates for lower point densities in the rainy-weather domain-adaptation data.The passage describes this as the probable reason for the large gains.
  • Tracking: Improved box estimation from 3D Auto Labeling enhances tracking performance over boxes from single-frame or multi-frame detectors.All compared methods use the same tracker, linking the comparison to localization quality.

C. Implementation Details of the MVF++ Detectors

MVF++ fuses point-wise, Bird’s Eye View, and Perspective View features, then produces detection results with auxiliary segmentation supervision and offboard-oriented architectural changes. Training uses geometric augmentation, specified detection ranges, and large-scale TPU optimization.

  • Network Architecture: MVF++ projects point encodings into 128-dimensional features and fuses dimension-reduced point features with 3D segmentation features, producing N × 144 outputs.The architecture extracts contextual information from Bird’s Eye View and Perspective View features before concatenation.
  • Network Architecture: View-dependent point features are voxelized into 2D maps, processed by separate view-dependent ConvNets, and passed to a backbone for detection.The backbone replaces plain convolution layers with ResNet layers while preserving output dimensions and feature-map resolution.
  • Loss Function: MVF++ minimizes classification, centerness, box-regression, and auxiliary segmentation losses, with weights w1 = 1.0, w2 = 2.0, and w3 = 1.0.At inference, classification and centerness scores are multiplied for box ranking, helping NMS downplay boxes far from object centers.
  • Data Augmentation: Training applies random x-axis flips, rotations uniformly sampled from [−π/4, +π/4], scaling from [0.9, 1.1], and translation noise drawn from N(0, 0.6).The LiDAR point cloud and ground-truth boxes receive the global transformations simultaneously.
  • Hyperparameters: The detector uses class-specific voxel sizes and ranges, yielding 468 × 468 vehicle and 624 × 624 pedestrian Bird’s Eye View feature maps.Test-time augmentation triggers weighted box fusion at class-specific IoU thresholds with box score 0.5.
  • Training: Training uses Adam with cosine learning-rate decay, 64 TPUs, global batch size 128, and 43,000 optimization steps.The learning rate starts at 1.33 × 10−3 and reaches 3.0 × 10−3 after 1000 warm-up steps.

D. Implementation Details of the Tracker

The tracker transforms detections into world coordinates, associates Bird’s Eye View boxes with the Hungarian algorithm, and handles heading angles explicitly during state updates.

  • Tracking: All boxes are transformed to world coordinates to reduce sensor ego-motion effects during tracking.Detections scoring below 0.1 are filtered before tracking to reduce false positives.
  • Tracking: Bird’s Eye View boxes are associated with the Hungarian algorithm using an IoU threshold of 0.1.The association operates on BEV boxes rather than full 3D box geometry.
  • Heading Handling: Heading updates first align detection angles with the track state and then average angles in cyclic space.This avoids discontinuities such as averaging 6 rad and 0.5 rad as 3.25.

E. Implementation Details of the Motion State Estimator

The pipeline estimates motion state and refines tracked objects with object-centric networks operating on point and box sequences. Static and dynamic models use PointNet-based segmentation and box regression, with temporal context and track-specific training procedures.

  • Motion State Estimator: Tracks require at least 7 valid measurements for motion-state classification and auto labeling; shorter tracks bypass both models and merge their boxes directly.Motion features are computed from world-coordinate box-center variance and begin-to-end track distance.
  • Motion State Estimator: For vehicles, the linear motion-state classifier achieves more than 99% accuracy, while all pedestrian tracks are treated as dynamic.Remaining vehicle errors usually occur in short, noisy, heavily occluded, or distant tracks.
  • Static Object Auto Labeling: The static auto-labeling model uses PointNet-based foreground segmentation and box regression, using only XYZ point coordinates.PointNet++ did not improve results over the PointNet-based models in this problem.
  • Static Object Auto Labeling: Static-model training uses extracted object tracks with randomly selected initial boxes, variable frame subsampling, 4,096-point sampling, and spatial augmentations.The training set contains around 50K vehicle tracks, with around 10K validation tracks for static objects.
  • Dynamic Object Auto Labeling: The dynamic model adds temporal encoding to object points and represents each box in the sequence as a parameterized point containing center, size, heading, and time.The box sequence uses nearly the entire sequence, with s = 50 producing a length of 101.
  • Dynamic Object Auto Labeling: A PointNet encodes the box trajectory, and point, trajectory, and joint embeddings feed box-regression heads trained with segmentation and three box losses.Empirical study found PointNet effective and simple compared with ConvNet, fully connected, and graph-based alternatives.

G. Details of the Human Label Study

The human label study measures label consistency on Waymo validation sequences and finds substantial human variability, while the authors report that auto-label quality is already on par with human labels.

  • Study Design: The study used 5 Waymo validation sequences, 15 relabeling tasks, 12 experienced labelers, and approximately 2.3K object-track labels.The 15 tasks comprised three sets of relabelings for each run segment.
  • Evaluation: Human labels are compared pairwise and against verified Waymo ground truth using 3D and Bird’s Eye View box IoUs.Objects may be unmatched because occlusions can produce false positives or false negatives for individual labelers.
  • Human Consistency: Human box consistency reaches around 81% 3D IoU and around 88% BEV IoU, rather than the expected 95% IoU.Nearby objects achieve higher mean IoU because they provide more visible points and more complete viewpoints.
  • Auto Labels versus Humans: Auto-label and human-label IoUs are not directly comparable because they cover different object sets, but the authors report that auto labels are already on par with human labels.The comparison uses average auto-label IoU with Waymo ground truth as a rough quality indication.

H. More Details about the Semi-supervised Learning Experiment

The semi-supervised experiments use 3D Auto Labeling to generate annotations from limited labeled sequences, then train an onboard student on combined data. Cross-domain evaluation reports strong performance without using target-domain data.

  • Experimental setup: The onboard single-frame MVF++ detector serves as the student in the semi-supervised experiments.Networks use an effective batch size of 256 scenes per iteration, with warmup followed by cosine learning-rate decay.
  • Intra-domain learning: 10% of the sequences, comprising around 15K frames from 79 sequences, train 3DAL, which reaches 78.11% AP on the validation set.3DAL then annotates the remaining training data for student training.
  • Intra-domain learning: 3DAL annotates around 142K frames from 719 remaining sequences, and the student trains on the union of 798 sequences for 43K iterations.
  • Cross-domain learning: 3DAL reaches 78.0% AP on the domain adaptation test set without using any data from that domain.The reported test set contains 100 sequences, while the validation set contains only 20 sequences.

I. More Analysis Experiments for Object Auto Labeling

The analysis examines initialization, temporal context, augmentation, tracking, motion-state estimation, and runtime choices in object auto labeling. Results favor selecting strong viewpoints, non-causal context, comprehensive augmentation, and accurate tracking, while motion-state refinement has limited impact.

  • Initial box selection: Frame-centric box estimation produces the most unfavorable auto-labeling results because it cannot leverage the best object viewpoint.Randomly selecting one box from a sequence corresponds to the frame-centric setting.
  • Initial box selection: A highest-confidence initial box improves auto-labeling accuracy by 4.46 Acc@0.8 over a random box.The highest-score strategy is analogous to choosing the object’s best viewpoint.
  • Temporal contexts: Causal static-object models are limited by reduced context, especially at track beginnings, and by a more restricted pool of initial boxes.The cited analysis attributes their lower accuracy to these two factors.
  • Temporal contexts: Non-causal models outperform causal models for static and dynamic object auto labeling, although causal models still improve over raw detector accuracy.For static objects, selecting the best key frame weighs more than adding points from a few history frames.
  • Data augmentation: All proposed augmentations consistently produce the most accurate models, while random Z-axis rotation is most critical for dynamic objects.For static objects, augmentation types contribute similarly to accuracy.
  • Tracking accuracy: Tracking accuracy is evaluated by comparing a Kalman filter tracker with an oracle tracker that associates detector boxes using ground-truth object IDs.Both trackers share the same detection and object auto-labeling models.
  • Motion-state estimation: Replacing predicted motion-state classification with ground-truth-box classification yields gains that are not significant.The analysis attributes this to the linear classifier already achieving over 99% accuracy.
  • Inference speed: For a 20-second sequence, detection takes around 15 minutes, while tracking and refinement take about 3 seconds and 25 seconds respectively.Tracking plus refinement totals 28 seconds, or 0.14 seconds per frame and 3% extra time over detection; offboard steps can run in parallel.
Loading 2103.05073v1…