PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Monocular 3D Vehicle Detection and Tracking

Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang, Ji Lin, Min Sun, Philipp Krähenbühl, Trevor Darrell, Fisher Yu

arXiv:1811.10742v3cs.CV

TL;DR

Monocular 3D vehicle detection and tracking must recover 3D layout and trajectories from image sequences despite depth ambiguity and ego-motion. The paper proposes an online joint framework using 3D association, occlusion-aware matching, and LSTM motion estimation, and reports competitive image-only tracking, including better Argoverse performance within 30 meters than LiDAR-centric baselines.

  • Problem

    Monocular 3D detection and tracking lack direct depth measurements, while accurate 3D layout and tracking are mutually dependent under ego-motion.

  • Method

    An online framework jointly detects and tracks vehicles from monocular image sequences using world-coordinate association, depth ordering, occlusion-aware matching, and LSTM trajectory updates.

  • Results

    The method outperforms published methods on KITTI and beats LiDAR-centric baselines on Argoverse at 50m and 30m ranges, while image-based accuracy declines faster with range.

  • Takeaways & Limitations

    Complete 3D information and temporal observations improve association and position estimation compared with traditional 2D tracking and single-frame estimation.

  • Takeaways & Limitations

    Image-based tracking is strongest for nearby vehicles but loses accuracy faster with increasing range than LiDAR-based tracking.

Abstract

from arXiv · show

Vehicle 3D extents and trajectories are critical cues for predicting the future location of vehicles and planning future agent ego-motion based on those predictions. In this paper, we propose a novel online framework for 3D vehicle detection and tracking from monocular videos. The framework can not only associate detections of vehicles in motion over time, but also estimate their complete 3D bounding box information from a sequence of 2D images captured on a moving platform. Our method leverages 3D box depth-ordering matching for robust instance association and utilizes 3D trajectory prediction for re-identification of occluded vehicles. We also design a motion learning module based on an LSTM for more accurate long-term motion extrapolation. Our experiments on simulation, KITTI, and Argoverse datasets show that our 3D tracking pipeline offers robust data association and tracking. On Argoverse, our image-based method is significantly better for tracking 3D vehicles within 30 meters than the LiDAR-centric baseline methods.

1. Introduction

The paper frames monocular 3D detection and tracking as an intertwined, ill-posed problem and proposes an online framework that combines temporal association, occlusion handling, 3D reasoning, and learned motion estimation. It also addresses limited trajectory supervision by collecting large-scale synthetic 3D trajectories.

  • Motivation: Monocular 3D layout estimation is ill-posed because a single view lacks sufficient depth information, making tracking difficult under large ego-motion.The paper emphasizes that tracking and detection are intertwined: temporal tracking can improve detection, while accurate 3D detection can factor out ego-motion.
  • Approach: The proposed online architecture jointly detects and tracks vehicles in 3D from sequences of monocular images.After 2D detection, it uses world and reprojected camera coordinates to associate instances across frames.
  • Approach: Occlusion-aware association and depth-ordering matching address vehicle occlusion and reappearance during tracking.The depth ordering of tracked vehicles provides a perceptual cue for reducing mismatches, while occlusion-aware association supports reappearing objects.
  • Approach: An LSTM estimates vehicle motion along trajectories by integrating associated single-frame observations over time.The motion is modeled in world coordinates, allowing 3D pose updates along each track.
  • Data: Existing datasets lack sufficient static-scene coverage, trajectory labels, or scale, so the paper collects large-scale 3D trajectories from a realistic synthetic driving environment.The collected data include dynamic metadata for observed scenes and objects.
  • Contribution: The framework estimates complete 3D track information, including position, orientation, dimensions, and projected 3D box centers, from monocular video.The paper reports that 3D information improves association over traditional 2D tracking and that sequence-based position estimation is more accurate than single-frame estimation.

2. Related Works

Prior tracking methods largely rely on 2D visual features and are less suited to dynamic 3D scenes, while related 3D approaches often use stereo depth or target only 2D tracking. The paper situates its online monocular approach alongside the need for substantial 3D supervision and synthetic driving datasets.

  • 2D Tracking: Traditional online trackers use foreground-background classifiers trained from patches near the target and farther background patches.These methods typically build on correlation filters or pre-trained recognition networks.
  • 2D Tracking: Trackers based only on 2D visual features restrict search near the original position, which is problematic for dynamic 3D environments.The paper motivates using 3D information to narrow the search space and stabilize target trajectories.
  • 3D Cues: Some prior methods use 3D cues for 2D vehicle tracking or add a 3D Kalman filter, but they do not consider complete 3D box dimensions and orientation.These approaches differ from the paper’s focus on full 3D vehicle tracking.
  • 3D Cues: Stereo-camera 3D tracking methods benefit from directly perceived depth, but large-baseline stereo is not always practical for applications such as ADAS.The paper therefore focuses on monocular imagery.
  • Datasets: Large-scale 3D tracking supervision is difficult to obtain, motivating the use of virtual driving datasets.The paper extends prior game-based data resources with 2D and 3D object tracking and fine-grained attributes.
  • Problem Setting: The proposed setting emphasizes online tracking, where the tracker accesses only past and current frames.This matches the practical requirement of autonomous driving systems.

3. Joint 3D Detection and Tracking

The framework jointly estimates complete 3D vehicle states and tracks instances online from monocular video. It combines 3D layout, depth-aware association, occlusion handling, and recurrent motion estimation so detection and tracking inform one another.

  • 3.1. Problem Formulation: The system infers precise 3D location, orientation, dimensions, and projected 3D box centers from monocular video while tracking vehicles across frames.It uses a single-frame detector for pose inference and recurrent networks for inter-frame association and matching.
  • 3.1. Problem Formulation: Each object state combines world position, velocity, orientation, dimensions, and appearance, while reconstructed 3D boxes support projection and association.Projected 2D boxes are generated using camera intrinsics and GPS- or IMU-derived extrinsics.
  • 3.2. Monocular 3D Detection: The detector extracts 2D proposals and projected 3D centers, then ROI features are used to estimate object center, orientation, dimension, and depth.Faster R-CNN generates proposals, and ROIalign supplies regional representations for the 3D estimation modules.
  • 3.5. Motion Model: The LSTM motion estimator fuses previous object motion with current observations to update 3D states and produce dense trajectories in world coordinates.The motion model is designed to operate independently of camera movement or interactions with other objects.
  • 3.4. Data Association and Tracking: Data association matches tracks and detections using projected overlap and deep appearance similarity, with trajectories projected using object velocity and camera ego-motion.The affinity combines appearance, 2D overlap, and 3D overlap factors.
  • 3.4. Data Association and Tracking: Depth-ordering matching prioritizes nearby tracklets by comparing detection overlap with non-occluded regions after sorting instances by depth.Distant candidates can be filtered before weighted bipartite matching, reducing mismatches.

4. 3D Vehicle Tracking Simulation Dataset

The paper uses driving simulation to address the shortage of large-scale real datasets with 3D bounding-box trajectories. Its collection pipeline captures accurate annotations and dynamic scene metadata from a photo-realistic environment.

  • 4. 3D Vehicle Tracking Simulation Dataset: Large-scale real-world 3D trajectory datasets are scarce because annotation is laborious and expensive, while existing datasets have limited scenes, trajectories, or scale.The authors therefore use realistic video games to obtain training data.
  • 4. 3D Vehicle Tracking Simulation Dataset: The pipeline collects large-scale 3D trajectories with dynamic metadata for observed scenes and objects from a realistic synthetic driving environment.The simulation is based on Grand Theft Auto V, and in-game APIs provide global instance identities and corresponding 3D annotations.

5. Experiments

The experiments evaluate the pipeline on synthetic GTA, KITTI, and Argoverse tracking data using 3D estimation, tracking, and association metrics. Ablations show benefits from depth ordering, occlusion-aware association, LSTM motion modeling, and projected 3D centers, while real-world results report strong KITTI and nearby Argoverse performance.

  • Evaluation setup: The pipeline is evaluated on a large-scale dataset, KITTI Tracking, and Argoverse Tracking, with separate training and evaluation splits described for each benchmark.Argoverse uses the validation benchmark because the evaluation server was unavailable at submission time.
  • Evaluation setup: The evaluation combines 3D IoU mAP with Dimension Score, Center Score, orientation score, and CLEAR tracking metrics including MOTA, MOTP, mismatch, false positive, and false negative rates.3D IoU mAP reflects the conjunction of estimated dimensions, rotation, and depth.
  • Ablation studies: Depth-order matching and occlusion-aware association filter out relatively 6−8% of possible mismatching trajectories in the GTA ablation.The ablation also reports that deep features reduce the false-negative rate when distinguishing near-overlapping objects.
  • Ablation studies: LSTM motion modeling achieves the highest tracking accuracy among the compared methods across all IoU thresholds, exceeding single-frame estimation and the smaller improvement from KF3D.The comparison attributes the difference to learned vehicle-motion modeling in 3D coordinates rather than Kalman-filter smoothing between prediction and observation.
  • Ablation studies: Estimating the projected 3D bounding-box center reduces ID switches and track fragmentation while increasing MOTA compared with using the 2D box center on KITTI.The projected center gives the motion module more accurate inputs for associating candidates.
  • Real-world evaluation: The monocular 3D tracking method outperforms published methods on KITTI and beats LiDAR-centric baselines on Argoverse at 50m and 30m ranges.On Argoverse, image-based accuracy drops faster with increasing range, while nearby vehicles can be estimated and associated accurately.

6. Conclusion

The framework combines monocular 3D estimation with temporal vehicle tracking, using learned dynamics and 3D cues to improve association and location updates.

  • The pipeline combines single-frame monocular 3D inference with a motion LSTM for inter-frame association and object-state updates.
  • Occlusion-aware association addresses inter-object occlusion, while depth ordering filters distant tracklet candidates during matching.
  • The LSTM motion estimator updates object velocity independently of camera movement.
  • Qualitative and quantitative results indicate that the model benefits from 3D estimation and collected dynamic 3D trajectories.

A. Projection of 3D Center

Accurate 3D center localization requires estimating box depth and projection because the 2D bounding-box center can be displaced from the projected 3D center, especially under occlusion or truncation.

  • 3D vehicle position is estimated from 2D-box depth together with observer pose and camera calibration.
  • The 2D bounding-box center can differ substantially from the projection of the 3D box center.
  • Center displacement can place the visible 2D center projection above or below the ground plane in 3D coordinates.
  • Occlusion and truncation produce center shifts that misalign ground-truth and predicted tracks and harm 3D IOU AP.

B. Data Association Details

Data association combines predicted tracklet continuation with depth-ordering filtering and overlap matching to handle occlusion, disappearance, and reappearance.

  • Unmatched tracklets are predicted until leaving the 0.15m-to-100m tracking range or dying after 20 time-steps.
  • Depth filtering keeps tracklets within a loose reachable-distance bound based on the diagonal length of two cars.
  • The overlap function retains non-occluded pixels from the nearest-depth tracklet, treating equal depth order within 1 meter.
  • Each detection matches the highest-IOU tracklet while modeling nearer vehicles occluding farther ones.

C. Motion Model

The motion model compares Kalman filtering with LSTM-based prediction and updating, using recent 3D velocities and observations to smooth trajectory estimates.

  • KF2D often misses tracking during substantial dynamic-scene changes caused by sudden ego-motion.
  • The proposed deep motion model uses separate LSTMs for predicting motion and refining 3D locations.
  • The predicting LSTM learns 3D velocity from the previous five updated velocities and the preceding location.
  • The updating LSTM combines predicted and observed locations into a 128-dim feature to update 3D location and velocity.
  • Training uses L1 and linear-motion losses to reduce location error and encourage smooth transitions.

D. Dataset Statistics

The dataset is characterized by diverse vehicle instances, partial visibility, and varied driving conditions. Its statistics are presented alongside comparisons with related datasets.

  • Dataset Composition: The GTA dataset includes 15 fine-grained vehicle subcategories and distributions of truncated and occluded instances.These statistics support analysis of partially visible objects in 3D coordinates.
  • Dataset Composition: The dataset statistics summarize scene and object properties through separate scene- and object-level analyses.
  • Dataset Diversity: The dataset contains diverse instance scales and an instance distribution closer to real scenes than the compared datasets.Examples span different times, weather conditions, and locations.

E. Training Details

Training uses dataset-specific image resolutions, detection setups, and pretrained models. The dataset comparison covers real-world and synthetic benchmarks with detection and tracking annotations.

  • Training Procedure: Training uses 1920 × 1080 images for GTA, 1920 × 1216 for Argoverse, and 1248 × 384 for KITTI.The model is trained on 4 GPUs with an effective minibatch size of 20.
  • Training Procedure: The RPN uses 4 anchor scales and 3 ratios to detect small distant objects.
  • Dataset-Specific Training: GTA training estimates 2D detections and 3D centers with Faster R-CNN, while KITTI and Argoverse use separate detection pipelines.The KITTI pipeline uses RRC results, and Argoverse uses Faster R-CNN results from MMDetection.
  • Dataset Comparison: The dataset comparison includes real-world and synthetic benchmarks with detection and tracking annotations, plus object and driving-related metadata.

F. Experiments

Experiments evaluate tracking robustness, runtime, association policies, and qualitative performance across KITTI, GTA, and Argoverse. The results emphasize the value of 3D information and the proposed association mechanisms.

  • Ego-Motion Evaluation: Estimated visual odometry maintains similar MOTA to ground-truth camera poses on KITTI.The comparison evaluates the impact of ego-motion estimation on tracking.
  • Efficiency: The framework’s total inference time is 203 ms on a single P100 GPU.KITTI runtime reports only the nondetection portion, which is 92 ms.
  • Tracking Policy: Tracklets are retained until leaving the 10m to 100m tracking range or dying after 20 time-steps, with longer retention preferred for re-identification.
  • Ablation Study: The association ablation compares location, maximum tracklet age, 2D and 3D cues, and appearance weighting.The table caption describes comparisons using different location and max-age settings.
  • Benchmark Results: The method outperforms all published methods reported on the KITTI tracking benchmark at the time of submission.The KITTI comparison reports only published methods, with a marker indicating use of 3D information.
  • Ablation Study: An 8% relative improvement in track recall and an 18% relative improvement in MOTA occur when moving from 2D to 3D tracking.Depth-order matching and occlusion-aware association filter out 6−8% of possible mismatching trajectories.
Loading 1811.10742v3…