PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular Quasi-Dense 3D Object Tracking

Hou-Ning Hu, Yung-Hsu Yang, Tobias Fischer, Trevor Darrell, Fisher Yu, Min Sun

arXiv:2103.07351v1cs.CV

TL;DR

Monocular 3D tracking seeks to estimate and associate objects in 3D from image sequences without 3D sensors. The paper combines quasi-dense similarity learning with 3D dynamics and reports near 500% improvement over camera-based state of the art on nuScenes, alongside a camera-only Waymo baseline.

  • Problem

    Autonomous-driving systems commonly rely on multiple views and sensors, motivating monocular 3D detection and tracking for low-cost 3D reasoning from 2D images.

  • Method

    The framework combines quasi-dense similarity learning, multimodal 3D association, depth-ordering matching, motion-based trajectory prediction, and an LSTM-based motion model.

  • Results

    Near 500% improvement over camera-based state of the art on nuScenes 3D tracking, while bridging the gap to LiDAR-based methods and establishing a camera-only Waymo baseline.

  • Takeaways & Limitations

    The pipeline offers robust data association for urban-driving scenarios and provides a camera-only baseline across Waymo 3D detection and tracking challenges.

  • Takeaways & Limitations

    On Waymo, limited camera coverage and LiDAR-based annotations prevent meaningful evaluation scores for the method on the 3D detection and tracking benchmarks.

Abstract

from arXiv · show

A reliable and accurate 3D tracking framework is essential for predicting future locations of surrounding objects and planning the observer's actions in numerous applications such as autonomous driving. We propose a framework that can effectively associate moving objects over time and estimate their full 3D bounding box information from a sequence of 2D images captured on a moving platform. The object association leverages quasi-dense similarity learning to identify objects in various poses and viewpoints with appearance cues only. After initial 2D association, we further utilize 3D bounding boxes depth-ordering heuristics for robust instance association and motion-based 3D trajectory prediction for re-identification of occluded vehicles. In the end, an LSTM-based object velocity learning module aggregates the long-term trajectory information for more accurate motion extrapolation. Experiments on our proposed simulation data and real-world benchmarks, including KITTI, nuScenes, and Waymo datasets, show that our tracking framework offers robust object association and tracking on urban-driving scenarios. On the Waymo Open benchmark, we establish the first camera-only baseline in the 3D tracking and 3D detection challenges. Our quasi-dense 3D tracking pipeline achieves impressive improvements on the nuScenes 3D tracking benchmark with near five times tracking accuracy of the best vision-only submission among all published methods. Our code, data and trained models are available at https://github.com/SysCV/qd-3dt.

1 INTRODUCTION

The paper targets robust monocular 3D detection and tracking for low-cost autonomous driving, estimating and associating 3D object states from image sequences captured by a moving platform. Its pipeline combines quasi-dense appearance learning, 3D cues, motion-aware association, and trajectory modeling, with strong benchmark results.

  • Monocular 3D detection and tracking aim to support 3D reasoning for low-cost mobility without 3D sensors.
  • The framework estimates 3D bounding-box centers, depths, dimensions, and orientations from monocular image proposals.
  • Quasi-dense similarity learning matches instances using densely populated object proposals rather than only scarce annotated regions.
  • Motion-aware association, depth-ordering matching, and 3D trajectory prediction address occlusion and object reappearance.
  • The revised pipeline adds 3D confidence, motion-based similarity, and VeloLSTM modeling of velocity, heading angle, and dimension.
  • The method is evaluated on simulated data, KITTI, nuScenes, and Waymo Open urban-driving datasets.

2 RELATED WORKS

Related work spans monocular 3D detection, image-domain and 3D tracking, joint detection-tracking, and autonomous-driving datasets. The paper builds on these lines while emphasizing online tracking from monocular dash-cam video.

  • Image-based 3D Object Detection: Image-based 3D detection methods regress or transform image representations to estimate 3D bounding boxes for navigation and autonomous driving.
  • 2D and 3D Object Tracking: Image-domain tracking uses visual features and location cues, but dynamic 3D environments motivate incorporating 3D information for trajectory stability.
  • Joint Detection and Tracking: Joint detection and tracking methods use detection information for tracking and tracking information as priors for future detection.
  • Joint Detection and Tracking: Prior work includes correlation maps, bounding-box deformation, track-level embeddings, and densely connected contrastive pairs for instance similarity.
  • Autonomous Driving Datasets: Driving benchmarks provide annotations for visual odometry, reconstruction, detection, tracking, and segmentation, but annotation effort limits dataset scale.
  • Autonomous Driving Datasets: The paper focuses on online tracking, where only the current frame and a constant number of past frames are accessible.

3 JOINT 3D DETECTION AND TRACKING

The framework jointly detects and tracks objects in 3D world coordinates from monocular video, combining per-instance 3D estimation, appearance embeddings, depth ordering, motion-aware association, and recurrent motion refinement.

  • Framework overview: The system generates 2D proposals, estimates each instance’s 3D position, orientation, dimensions, and appearance embedding, then links objects across frames.Its state includes world location, orientation, dimensions, appearance features, and velocity.
  • Motion Model Refinement: The complete pipeline uses recurrent networks for inter-frame matching and an LSTM motion estimator to update object velocities, states, and smooth world-coordinate trajectories.The motion model refines 3D location through newly matched trajectories while accounting for camera movement.
  • Quasi-Dense Similarity Learning: Quasi-dense similarity learning compares proposal pairs across neighboring frames to learn embeddings that separate different instances and associate the same instance.The method uses positive and negative proposals, including dense proposals beyond sparse ground-truth boxes.
  • Data Association and Tracking: Depth-ordering matching uses 3D object states, including centroid distance, orientation, and dimensions, to assign detections to nearby tracklets.An exponential distance weighting reduces the affinity of distant candidates.
  • Data Association and Tracking: Motion-aware association combines accumulated tracklet motion with pseudo motion from single-frame depth estimates to match objects after occlusion.The motion similarity also incorporates centroid location because similar vectors can occur at different positions.

4 3D VEHICLE TRACKING SIMULATION DATASET

The authors create a large-scale synthetic driving dataset to address the cost and scarcity of real-world 3D trajectory annotations. Its pipeline provides accurate instance-linked 3D annotations, and its statistics show greater scale diversity and more realistic instance distributions than comparison datasets.

  • The dataset uses driving simulation to obtain accurate 3D bounding-box annotations without human labeling effort.The pipeline is designed to address the expense of annotating large-scale 3D bounding-box image datasets.
  • Global instance IDs and corresponding 3D annotations are captured directly through the Grand Theft Auto V in-game API.This avoids the heuristic matching inconsistencies described for VIPER and supports real-time data collection.
  • The proposed dataset offers greater diversity in instance scales and an instance distribution closer to real scenes than the compared datasets.The comparison includes KITTI, VKITTI, VIPER, and Cityscapes.

5 EXPERIMENTS

The experiments evaluate the monocular 3D detection and tracking pipeline across synthetic and real-world urban-driving benchmarks, with dataset-specific training, aggregation, and metric choices. The authors report competitive camera-only performance on nuScenes but identify substantial Waymo limitations caused by incomplete camera coverage and IoU-based evaluation.

  • Datasets: The evaluation covers Waymo Open, nuScenes Tracking, KITTI MOT, and the proposed large-scale simulation dataset.The datasets span real-world driving scenes and diverse simulated road conditions.
  • Cross-camera aggregation: nuScenes and Waymo results are aligned across cameras online by removing duplicate objects within 2 meters, or 1 meter for smaller objects.When duplicates are found, the box with the higher 3D score is retained.
  • Dataset limitations: Camera-only nuScenes results are competitive with LiDAR baselines because its six cameras can be stitched into a 360-degree scene.The authors use this setting to assess monocular performance on a multi-camera benchmark.
  • Dataset limitations: Waymo's five-camera setup covers only 5/8 of the horizon and lacks rear-camera information, limiting meaningful 3D detection and tracking evaluation.Waymo objects are annotated in LiDAR sweeps, creating an additional mismatch for the camera-only method.
  • Training setup: The model uses dataset-specific network and training configurations, including different image resolutions, architectures, and projected annotations for nuScenes and Waymo.Training uses 24 epochs with SGD, warm-up learning-rate scheduling, and horizontal-flip augmentation.
  • Evaluation metrics: The experiments evaluate detection with dataset-specific 3D metrics and tracking with KITTI, nuScenes, and Waymo protocols, including centroid-distance matching for nuScenes.The reported measures include KITTI 3D mAP, nuScenes NDS and AMOTA, and Waymo mAPH and IoU-based tracking metrics.

5.3 Ablation Experiments

Ablations show that deep-feature and 3D-state affinities improve association, while depth-ordering and motion-aware matching strengthen tracking performance. Accurate 3D center projection also improves motion-based association.

  • Deep-feature affinity distinguishes near-overlapping objects, while 3D-state affinity also contributes effectively to AMOTA@1.
  • 40.1% relative AMOTA@1 improvement follows from adding depth-order matching to the boxes-IoU affinity.
  • Motion-aware association reaches 0.2306 AMOTA@1, compared with 0.2234 using cosine angle difference and 0.2300 using centroid distance.
  • Estimating the projected 3D bounding-box center improves tracking capacity for candidate association and reduces centroid misalignment.

5.4 Motion Modeling Comparison.

The motion-model comparison evaluates detection-only, momentum, Kalman-filter, and VeloLSTM approaches for refining object locations over trajectories. VeloLSTM provides the strongest reported performance by learning from prior velocity predictions and current observations.

  • The comparison includes pure detection, fixed-momentum updates, a 3D Kalman filter, and learned VeloLSTM motion estimation.
  • VeloLSTM achieves the highest AMOTA@1, AMOTP, and NDS among the compared motion models.
  • VeloLSTM uses past n = 5 velocity predictions and the current frame observation to compensate for observation error.
  • The Kalman filter provides a small NDS improvement through trajectory smoothing within prediction and observation.

5.5 Real-world Evaluation.

Evaluation on nuScenes, Waymo Open, and KITTI tests the pipeline across real-world urban-driving benchmarks. The method reports strong camera-only results, while the Waymo IoU threshold exposes a metric limitation for monocular tracking.

  • nuScenes: 0.217 AMOTA is achieved on nuScenes, near five times the tracking accuracy of the best published vision-only submission.
  • Waymo Open: The pipeline establishes the first camera-only entry for both Waymo Open 3D tracking and 3D detection challenges.
  • Waymo Open: The Waymo IoU threshold of 0.7 prevents meaningful evaluation of camera-based tracking performance.
  • KITTI: On KITTI, VeloLSTM continues unmatched tracklets and motion-based association recovers vehicles after occlusion.
  • Evaluation metrics: Camera-based GT yields more meaningful results than LiDAR-based GT, whose annotations cover a broader 360-degree observable area.
  • Cross-dataset evaluation: Centroid-based evaluation shows similar MOTA and MOTP across nuScenes and Waymo Open, indicating robustness across datasets.

5.7 Amount of Data Matters.

Experiments show that performance improves consistently as the amount of training data increases, especially for depth estimation and distant or small objects. The tracker also uses greedy matching because trained embeddings and 3D states provide robust pairings with lower complexity.

  • Performance improves consistently when depth estimation is trained with 1%, 10%, and 100% of the GTA data.
  • Large-scale 3D annotation is especially helpful for distant and small objects.
  • Greedy matching yields similar results to Hungarian matching with lower computational complexity on well-trained quasi-dense embedding pairs.

6 CONCLUSION

The paper presents an online monocular 3D detection and tracking framework that combines quasi-dense similarity learning with 3D instance dynamics. Its experiments report robust urban-driving association and strong benchmark performance, including camera-only results on Waymo and near-fivefold gains over the best camera-based nuScenes submission.

  • The framework combines monocular 3D inference, cross-frame contrastive features, multimodal association, and an LSTM-based motion model for online world-coordinate tracking.It also uses 3D confidence, depth-ordering matching, motion-based re-identification, and VeloLSTM trajectory updates.
  • 3D bounding-box depth ordering and motion-based trajectory prediction support robust association and re-identification of occluded vehicles.
  • The pipeline offers robust data association on urban-driving situations by exploiting dynamic 3D trajectories.
  • On Waymo Open, the framework establishes a strong monocular-camera baseline with positive scores in both 3D detection and tracking challenges.
  • The nuScenes pipeline outperforms the camera-based state of the art by near 500% while narrowing the gap to LiDAR-based methods.

APPENDIX A DATASET STATISTICS

The appendix describes a large-scale synthetic driving dataset designed to address limited training data and provide diverse object and scene conditions. Its statistics cover object properties, weather, time of day, and varied environments.

  • Dataset Comparison: The dataset is presented as a large-scale benchmark with detection, tracking, and driving annotations, addressing the limited scale of related datasets.The comparison includes real-world and synthetic datasets and notes the dataset's dynamic-scene focus.
  • Object Statistics: The dataset statistics include object truncation and occlusion categories.
  • Weather and Time of Day: The virtual environment covers varied weather cycles and times of day to expose the model to changing perceptual conditions.The authors state that weather diversity supports environmental understanding and time-of-day variation helps handle perceptual changes.
  • Examples of Our Dataset: Example scenes span different times, weather conditions, and locations.

APPENDIX B TRAINING DETAILS

Training procedures are dataset-specific and are adjusted to data volume and GPU-memory constraints. The appendix gives separate optimization schedules for nuScenes, Waymo, and GTA.

  • The model uses different training procedures for each dataset according to available data and GPU-memory limits.
  • nuScenes: For nuScenes, training uses all 10 object classes, while tracking evaluation covers 7 classes and detection evaluation covers 10.Training runs for 24 epochs on 8 GPUs with total batch size 32.
  • Waymo: Waymo training uses 8 GPUs, batch size 16, 24 epochs, and learning-rate reductions after epochs 8, 12, 16, 20, and 22.
  • GTA: GTA training uses 8 GPUs, batch size 24, and 24 epochs with a warm-up learning rate increasing from 1 × 10^-3 to 1 × 10^-2.

APPENDIX C EXPERIMENTS

The appendix reports runtime, dataset composition, association settings, and benchmark results for the proposed tracking framework. Results include improved KITTI performance, strong qualitative tracking on nuScenes and Waymo, and a top-ranking camera-based nuScenes result.

  • Inference Time: The complete framework averages 162.0 ms per frame on a single RTX 2080Ti GPU over 11095 frames.The reported sum is 123.3 + 2.3 + 12.3 + 1.9 + 22.2 ms.
  • Dataset Comparison: The synthetic dataset comparison emphasizes detection, tracking, driving annotations, dynamic scenes, and fine-grained object attributes.The dataset is described as the largest 3D tracking benchmark for dynamic scene understanding.
  • Data Association Weights: Association experiments balance appearance with 3D extents by setting wdeep to 0.5 for 3D-related methods.Deep-appearance-only methods instead receive 100% weighting on wdeep.
  • Tracking Performance on KITTI: KITTI MOTA improves from 84.52 to 86.41, while the method reaches third place and ranks among the top published methods.
  • Tracking Performance on nuScenes: On nuScenes, the method achieves about five times the 3D tracking performance of the best camera-based submission and ranks first among reported methods.Qualitative results also illustrate long-term trajectories on nuScenes and a strong monocular baseline on Waymo.
Loading 2103.07351v1…