Source-linked AI summary
Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
Wenjie Luo, Bin Yang, Raquel Urtasun
TL;DR
Autonomous-driving systems commonly separate detection, tracking, forecasting, and planning, allowing early errors to propagate through the cascade. FaF uses a single spatio-temporal convolutional network over a BEV representation to jointly perform 3D detection, tracking, and short-term forecasting. On a very large-scale dataset, it significantly outperforms the state of the art while completing all tasks in as little as 30 ms.
Problem
Cascaded autonomous-driving modules are usually learned independently, and uncertainty is rarely propagated, allowing early errors to cause catastrophic downstream failures.
Method
FaF is a single one-stage network that applies 3D convolutions across space and time to a multi-frame 4D occupancy representation and jointly predicts 3D detections, tracks, and short-term future trajectories.
Results
30 ms: FaF significantly outperforms the state of the art on a very large-scale North American dataset while performing all three tasks.
Takeaways & Limitations
Holistic reasoning and shared computation provide accurate real-time estimates for detection, tracking, and motion forecasting in autonomous driving.
Abstract
from arXiv · showhide
In this paper we propose a novel deep neural network that is able to jointly reason about 3D detection, tracking and motion forecasting given data captured by a 3D sensor. By jointly reasoning about these tasks, our holistic approach is more robust to occlusion as well as sparse data at range. Our approach performs 3D convolutions across space and time over a bird's eye view representation of the 3D world, which is very efficient in terms of both memory and computation. Our experiments on a new very large scale dataset captured in several north american cities, show that we can outperform the state-of-the-art by a large margin. Importantly, by sharing computation we can perform all tasks in as little as 30 ms.
1. Introduction
FaF replaces the usual cascaded autonomous-driving pipeline with a joint, end-to-end network for 3D detection, tracking, and motion forecasting. It uses spatio-temporal 3D sensing and shared computation to improve robustness and achieve real-time operation.
- Cascaded modules can fail catastrophically because downstream processes cannot recover from errors introduced early in the pipeline.
- FaF jointly performs 3D detection, tracking, and motion forecasting using spatio-temporal information from a 3D sensor.
- Tracking and prediction information can reduce false negatives for occluded or far-away objects, while accumulating evidence over time can reduce false positives.
- The network operates on a bird's-eye-view representation of the 3D world, allowing it to exploit typical object-size priors.
- 3D convolutions over space and time process multiple consecutive frames, producing current and future bounding boxes whose predictions are pooled into tracklets.
- 30 ms: the approach performs detection, tracking, and motion forecasting on a very large-scale North American dataset while significantly outperforming the state of the art.
2. Related Work
Prior work largely treats detection, tracking, and motion forecasting as separate problems or combines only subsets of them. FaF instead uses one temporally informed network to jointly address all three tasks in autonomous driving at real-time speed.
- 2D detection methods commonly use either proposal-based two-stage architectures or one-stage convolutional detectors.
- 3D detection research has used stereo images, voxelized point clouds, or image–point-cloud fusion, including BEV features with handcrafted height inputs.
- Deep-learning tracking methods have used pretrained CNN features, correlation or regression, autoencoders, siamese matching, and inference-time fine-tuning.
- Motion forecasting predicts future object locations from multiple past frames, with prior work modeling long-term trajectories, interactions, or short-term motion.
- Earlier multi-task work jointly modeled detection and tracking, whereas FaF additionally tackles 3D detection, short-term forecasting, and tracking in one network.
- FaF uses temporal information and holistic reasoning to propagate uncertainty while running at 33 FPS.
3. Joint 3D Detection, Tracking and Motion Forecasting
Fast and Furious uses a temporally stacked voxel representation and a single-stage convolutional network to jointly perform 3D detection, tracking, and short-term motion forecasting. The model predicts boxes across timestamps, decodes tracklets by pooling predictions, and shares computation across tasks.
- Data Representation: A 4D tensor of voxelized occupancy grids from multiple frames feeds 3D convolutions over space and time.Past points are transformed into the current vehicle coordinate system before frames are appended along a temporal dimension.
- Data Representation: Single-frame voxelization uses 2D convolutions with height treated as the channel dimension because 3D point-cloud grids are sparse.Each voxel receives a binary occupancy indicator, while the height dimension remains available for learned feature extraction.
- Model Architecture: The one-stage detector directly regresses bounding boxes at the current frame and multiple future timestamps without region proposals.Early fusion aggregates temporal information at the first layer, whereas late fusion gradually merges it to capture higher-level motion features.
- Model Architecture: Two convolutional branches classify vehicle occupancy and predict current and future bounding boxes, enabling short-term motion forecasting from temporal features.The model can learn cues such as velocity and acceleration from multiple input frames.
- Tracking: Tracklets are decoded by averaging current detections with past predictions, avoiding trajectory-based optimization.Each timestamp contains current detections and predictions from earlier timestamps, allowing evidence to be aggregated over time.
- Training: The training objective combines classification and regression losses over current detections and future predictions.Classification uses binary cross-entropy, while regression uses weighted smooth L1 targets matched to predefined boxes by overlap.
4. Experimental Evaluation
The evaluation uses a large LiDAR dataset and tests detection, tracking, and motion forecasting under varying sparsity and distance. The joint model outperforms comparison methods while operating in real time.
- Dataset and setup: The dataset contains 546,658 frames from 2,762 North-American driving scenes, with 5,000 validation samples from 100 geographically separate scenes.Training uses a 144 × 80 meter region, 0.2-meter grid cells, 29 height bins, and the past five timestamps.
- Detection: The detector is compared with SSD, MobileNet, and SqueezeNet using vehicle boxes with at least three 3D points and IoU-based correctness.The evaluation region is 144 × 80 meters, and IoU 0.7 is used for the principal detection comparison.
- Detection: The proposed method outperforms other methods at every tested minimum-point level, including the zero-point setting used to assess temporal information.The minimum-point threshold reflects sparsity caused by occlusion or long-distance vehicles.
- Detection: The model is significantly better at long range, although all methods perform poorly at 100 meters because of limited 3D points.Nearby-vehicle performance is strong across methods, while the long-range gap favors the proposed approach.
- Ablation study: Temporal information improves detection by 3.7% mAP with early fusion and an additional 1.4% with late fusion; prediction loss yields the best reported performance, 6% points better on mAP.Late fusion models more complex temporal features, while prediction loss also enables tracklet decoding and smoother detections.
- Tracking: 80.9% MOTA is achieved, 7.8% better than Hungarian tracking, with 20% better Mostly-Tracked and 10% lower Mostly-Lost performance at similar MOTP.The model’s raw track-id output is evaluated without an additional sophisticated tracking pipeline.
- Motion forecasting: L2 distance remains below 0.33 meter when forecasting 10 frames ahead, evaluated on true positives with 92.5% recall.The study measures average L1 and L2 distances of vehicle-center locations.
5. Conclusion
The paper concludes that a holistic model can jointly reason about detection, prediction, and tracking for autonomous driving while running in real time and achieving strong accuracy across tasks.
- Conclusion: The proposed holistic model jointly reasons about detection, prediction, and tracking, runs in real time, and achieves strong accuracy across all tasks.Future work includes RoI align, additional object categories, and longer-term predictions.