Source-linked AI summary
PnPNet: End-to-End Perception and Prediction with Tracking in the Loop
Ming Liang, Bin Yang, Wenyuan Zeng, Yun Chen, Rui Hu, Sergio Casas, Raquel Urtasun
TL;DR
The paper addresses joint perception and motion forecasting for self-driving vehicles, where temporal context is important for estimating and predicting surrounding objects. PnPNet integrates online tracking with shared end-to-end perception and prediction, and reports improvements in both tasks, including occlusion recovery and future prediction accuracy.
Problem
Joint detection-prediction models without tracking use limited temporal history, which can hinder occlusion handling and temporal consistency in predictions.
Method
PnPNet uses a shared end-to-end model with online discrete-continuous tracking that updates object tracks from detections and supplies trajectory-level features to motion forecasting.
Results
PnPNet shows significant improvements over prior paradigms in perception and prediction on two large-scale driving datasets, with better occlusion recovery and more accurate future predictions.
Takeaways & Limitations
Integrating tracking into the perception-prediction loop yields more complete trajectories, quicker occlusion recovery, and improved future trajectory predictions.
Takeaways & Limitations
Compared with tracking-integrated models, prior joint approaches are limited to up to 1 second of past sensor data because tracking is not included in the loop.
Abstract
from arXiv · showhide
We tackle the problem of joint perception and motion forecasting in the context of self-driving vehicles. Towards this goal we propose PnPNet, an end-to-end model that takes as input sequential sensor data, and outputs at each time step object tracks and their future trajectories. The key component is a novel tracking module that generates object tracks online from detections and exploits trajectory level features for motion forecasting. Specifically, the object tracks get updated at each time step by solving both the data association problem and the trajectory estimation problem. Importantly, the whole model is end-to-end trainable and benefits from joint optimization of all tasks. We validate PnPNet on two large-scale driving datasets, and show significant improvements over the state-of-the-art with better occlusion recovery and more accurate future prediction.
1. Introduction
PnPNet addresses joint perception and motion forecasting for autonomous driving by placing tracking inside an end-to-end model, allowing temporal track context to inform all tasks. It combines trajectory representations with discrete-continuous tracking and improves perception and prediction, including occlusion recovery.
- Joint perception and motion forecasting are crucial for autonomous driving because safe maneuver planning requires anticipating surrounding agents’ future decisions and estimating their current state.
- Traditional autonomy stacks separate detection, tracking, and forecasting into independently developed sequential components, while joint detection-prediction models leave tracking as post-processing.
- Without tracking in the loop, joint models use limited temporal history, which can hinder occlusion handling and produce temporally inconsistent predictions.
- PnPNet updates tracks from sequential detections, using association and trajectory estimation to provide richer temporal information for future trajectory prediction.
- Its trajectory representation combines inferred motion and raw sensor observations through a recurrent network and is used by both tracking and prediction.
- PnPNet is validated on two large-scale driving datasets and reports improvements in perception and prediction, including more complete trajectories, faster occlusion recovery, and more accurate future predictions.
2. Related Work
Prior work studies detection, tracking, and motion forecasting separately or combines selected perception and prediction tasks. The supplied passages position trajectory-level representations and explicit temporal modeling as relevant directions for joint systems.
- Related work covers 3D object detection, multi-object tracking, motion forecasting, and joint models for perception and prediction.
- LiDAR-based methods address the depth ambiguity that limits image-only 3D detection in safety-critical applications.
- Multi-object tracking commonly follows tracking-by-detection, combining discrete data association with continuous trajectory estimation.
- Tracking research includes optimization-based association methods, occlusion handling with heuristics or single-object trackers, and alternative affinity representations.
- Motion-forecasting approaches model trajectory proposals, semantic context, interactions, social patterns, and multimodal agent behavior using varied formulations.
- Joint models extend shared representations to detection, forecasting, intentions, and spatial reasoning, while trajectory-level actor representations encode long-term history.
3. End-to-End Perception and Prediction
PnPNet addresses limited temporal history in joint perception and prediction by placing explicit online tracking between detection and prediction. Its trajectory representation combines observations and motion over time, while discrete-continuous tracking supports occlusion handling and trajectory refinement.
- Overview: PnPNet uses explicit object tracking to connect detection and motion forecasting, overcoming joint models’ limited exploitation of temporal history.The prediction module receives tracked object trajectories rather than only convolutional features from the detection backbone.
- Object Detection Module: The model takes multi-sweep LiDAR point clouds and an HD map, producing bird’s-eye-view object detections at each frame.The detector combines up to 0.5 seconds of ego-motion-compensated LiDAR sweeps with map features in a voxel representation.
- Discrete-Continuous Tracking Module: The tracking module solves data association and trajectory estimation, using bipartite matching for detections and single-object tracking for unmatched tracks.Learned affinity and unary scores support matching, while the single-object tracker supplies candidates for objects missed by the detector, including during occlusion.
- Trajectory Representation: Trajectory-level features combine sensor observations with inferred object and ego-motion, then use an LSTM to model temporal dynamics.The representation is formed across object-track frames and is shared by the tracking and prediction modules.
- Discrete-Continuous Tracking Module: Trajectory estimation updates each track’s confidence and recent waypoints to reduce false positives and localization error from detection or association.The updated tracks are filtered with non-maximum suppression before retaining the top tracks.
4. Experiments
PnPNet is evaluated on two large-scale driving datasets using modular and end-to-end system metrics. Across detection, tracking, prediction, and ablations, results show improved accuracy, occlusion recovery, and trajectory quality.
- Datasets and Metrics: PnPNet is evaluated on nuScenes and ATG4D using modular metrics for detection and tracking and system metrics for end-to-end perception and prediction.System evaluation uses real-world inputs, including totally occluded objects; modular metrics evaluate visible objects only.
- Main Results: 0.8% AP improvement at the 0.5 meter threshold over Megvii demonstrates stronger 3D detection at higher localization precision.The comparison is conducted on nuScenes against a leading state-of-the-art detector.
- Main Results: 8.0% AMOTA improvement over StanfordIPRL-TRI shows that PnPNet tracking produces more complete, faster-recovering, and more precise trajectories.The KF tracker baseline improves AMOTA by 2.6%, while the proposed tracker improves it by 8.0%.
- Main Results: Up to 2% AP, 4% recall, and 20% prediction improvements over the no-tracking baseline occur across two object classes and two datasets.Prediction improves by 8% to 20% relatively, while tracking supports long-term occlusion recovery and trajectory-level estimation.
- Ablation Study: Longer object tracks leave perception largely unchanged but reduce prediction errors, with performance plateauing at around 16 frames or 1.6 seconds.All ablations use ATG4D vehicles and system metrics; PnPNet therefore uses T=16.
- Ablation Study: Removing explicit motion features increases prediction error by approximately 6%, while removing single-object tracking reduces occlusion-related recall by 1.7% and increases prediction errors by 2%.These ablations leave other components unchanged and indicate the value of motion history and occlusion tracking for prediction and perception.
5. Conclusion
PnPNet is an end-to-end autonomous-driving model that jointly handles perception and prediction through shared computation and online tracking. It improves both perception and prediction metrics on two large-scale driving datasets.
- PnPNet jointly models perception and prediction instead of using separate models for each traditional-stack subtask.Its tracker generates object trajectories online from detections and provides trajectory-level features for forecasting.
- The model uses shared feature computation and a novel multi-object tracker within an end-to-end architecture.
- PnPNet shows significant improvements in both perception and prediction metrics on two large-scale driving datasets.
- The authors plan to extend the approach to multi-agent behavior prediction and motion planning.
A. Backbone Network Architecture
The backbone represents sequential LiDAR and map information in bird’s-eye view, then processes the combined BEV input with downsampling and multiscale cross-scale modules.
- The backbone uses a bird’s-eye-view occupancy map for LiDAR and concatenates multi-sweep BEV representations along the height dimension.The input combines one current LiDAR frame with N previous frames.
- The map representation incorporates both geometric and semantic priors similar to HDNet.
- The architecture applies three Conv2D layers to down-sample the BEV input by a factor of 4.
- A cross-scale module is applied sequentially three times to exchange information among three feature-map scales.The module is inspired by Inception blocks and includes residual connections.
B. Implementation Details on nuScenes
For nuScenes, the model uses a shared architecture with ten LiDAR sweeps covering 0.5 seconds of history, BEV inputs voxelized around the ego vehicle, and frame-level augmentation during training.
- nuScenes input aggregates 10 LiDAR sweeps—1 current and 9 previous—covering 0.5 seconds of past history.
- The input region spans [−50, 50] × [−50, 50] × [−3, 5] meters around the ego car.A voxel size of 0.15625 × 0.15625 × 0.25 meters produces a 640 × 640 × 320 voxel grid.
- Training augmentation randomly scales, translates, rotates, and flips both LiDAR point clouds and 3D labels.Labels at non-key frames are linearly interpolated from adjacent key frames.
- The model is trained only on cars, excluding labels with no LiDAR points or those beyond 50 meters from the ego car.Training uses Adam, batch size 8, for 4 epochs with scheduled learning-rate decay.
C. Fine-Grained Evaluation of Motion Forecasting
The fine-grained evaluation measures FDE at 1s, 2s, and 3s across all recall rates on nuScenes and ATG4D. PnPNet achieves higher detection recall and consistently outperforms the baseline in prediction.
- FDE is evaluated at 1s, 2s, and 3s prediction horizons across all object recall rates.
- PnPNet achieves higher object detection recall than the baseline across the fine-grained evaluation.
- PnPNet consistently outperforms the baseline in prediction at all recall rates.The evaluation covers both nuScenes and ATG4D datasets.
- Figure 6 presents the backbone architecture, while Figure 7 presents FDE results across prediction horizons and recall rates.