Source-linked AI summary
MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps
Pengxiang Wu, Siheng Chen, Dimitris Metaxas
TL;DR
Autonomous driving requires reliable perception of objects and their motion, while bounding-box-based pipelines can fail on unseen object categories. MotionNet addresses this with joint BEV-based perception and motion prediction, using hierarchical spatio-temporal features and consistency losses; experiments on nuScenes report effectiveness and potential complementary use in motion planning.
Problem
Reliable autonomous driving requires joint perception of environmental objects and prediction of their future trajectories, but bounding-box-based methods struggle with object categories absent from training.
Method
MotionNet converts LiDAR sweep sequences into BEV maps and jointly predicts cell categories, motion, and state using a spatio-temporal pyramid network with spatial and temporal consistency losses.
Results
MotionNet demonstrates effectiveness and superiority on nuScenes against prior scene-flow- and object-detection-based methods.
Takeaways & Limitations
MotionNet has potential as a backup to bounding-box systems and as a source of complementary perception and motion information for autonomous-driving motion planning.
Takeaways & Limitations
Performance is constrained for bicycles by their limited training-set representation and small size in BEV maps, and this issue persists despite increased category weighting.
Abstract
from arXiv · showhide
The ability to reliably perceive the environmental states, particularly the existence of objects and their motion behavior, is crucial for autonomous driving. In this work, we propose an efficient deep model, called MotionNet, to jointly perform perception and motion prediction from 3D point clouds. MotionNet takes a sequence of LiDAR sweeps as input and outputs a bird's eye view (BEV) map, which encodes the object category and motion information in each grid cell. The backbone of MotionNet is a novel spatio-temporal pyramid network, which extracts deep spatial and temporal features in a hierarchical fashion. To enforce the smoothness of predictions over both space and time, the training of MotionNet is further regularized with novel spatial and temporal consistency losses. Extensive experiments show that the proposed method overall outperforms the state-of-the-arts, including the latest scene-flow- and 3D-object-detection-based methods. This indicates the potential value of the proposed method serving as a backup to the bounding-box-based system, and providing complementary information to the motion planner in autonomous driving. Code is available at https://github.com/pxiangwu/MotionNet.
1. Introduction
MotionNet addresses joint environmental perception and motion prediction from LiDAR by representing scenes as BEV maps rather than relying on bounding boxes. Its efficient spatio-temporal architecture and consistency regularization support real-time prediction and complementary use in autonomous-driving systems.
- Motivation: Bounding-box-dependent state-estimation pipelines tend to fail in open-set traffic scenarios when detectors encounter object classes absent from training.This dependency can cause downstream tracking and motion-prediction modules to fail.
- BEV representation: BEV maps encode occupancy, motion, and category information for discretized cells, with displacement vectors representing future positions.The representation can characterize nonlinear dynamics while also describing drivable space.
- MotionNet: MotionNet jointly reasons about cell categories and motion from temporal BEV sequences without bounding boxes.Its outputs include cell classification, state estimation, and motion prediction, while motion predictions are regularized using classification results and spatial-temporal consistency losses.
- Architecture: The spatio-temporal pyramid network extracts hierarchical spatial and temporal features using efficient spatial convolutions followed by pseudo-1D temporal convolutions.The model is reported to run at 53 Hz, supporting real-time deployment.
- Evaluation and role: Experiments on nuScenes compare MotionNet with scene-flow- and object-detection-based methods and report effectiveness and superiority.The authors describe potential collaborative use with other modules and complementary perception and motion information for motion planning.
2. Related Work
Prior environmental-state methods commonly rely on object detection and tracking, which limits robustness to unseen objects. MotionNet instead combines category-aware BEV representation with joint cell-level perception and motion prediction.
- Perception: Perception methods use 2D, 3D, or fusion-based object detection, but shape recognition makes unseen training categories difficult to detect.The BEV-map representation is presented as not relying on shape recognition and as providing complementary information to the motion planner.
- MotionNet: MotionNet processes sequences of LiDAR sweeps as BEV maps and produces a BEV output containing perception and motion-prediction information.Its pipeline uses an STPN backbone with separate cell-classification, motion-prediction, and state-estimation heads.
- Motion prediction: Trajectory-based motion prediction typically depends on accurate object detection and tracking for acquiring object histories.Joint detection, tracking, and forecasting methods remain vulnerable when unexpected objects defeat bounding-box detection.
3. Methodology
MotionNet converts synchronized LiDAR sequences into BEV maps and jointly predicts cell categories, motion, and state. Its spatio-temporal pyramid backbone and consistency-based regularization support efficient, smoother, bounding-box-free predictions.
- Data representation: MotionNet processes synchronized 3D point-cloud sequences represented as BEV maps, using 2D convolutions for efficient processing.Ego-motion compensation aligns past frames to the current vehicle coordinate system, while BEV conversion makes sparse point clouds amenable to standard convolutions.
- Spatio-temporal pyramid network: STPN hierarchically extracts spatial and temporal features using STC blocks that combine 2D convolutions with pseudo-1D temporal convolutions.Temporal resolution is progressively reduced, while multi-scale features are fused through global temporal pooling and lateral connections.
- Output heads: Three task-specific heads classify cells, forecast future cell positions, and estimate whether each cell is static or moving.The motion head outputs N × H × W × 2 future positions, while the state head supplies auxiliary motion information.
- Output heads: Classification and state estimates suppress motion jitters, including spurious background and static-foreground movement.Background motions are thresholded to zero, and estimated static foreground cells receive the same suppression treatment.
- Loss function: Spatial and temporal consistency losses regularize predictions by encouraging nearby cells on the same object and aligned predictions across time to remain coherent.The spatial loss compares adjacent cell pairs, while temporal alignment uses a rigid transformation between predictions from successive time references.
4. Experiments
Experiments on nuScenes evaluate MotionNet’s perception and motion prediction against several baselines, with ablations examining its inputs, synchronization, feature fusion, and training strategies. The results indicate strong accuracy, efficiency, and benefits from consistency losses and design choices.
- Dataset: The nuScenes evaluation uses LiDAR-only data from 850 annotated scenes, split into 500 training, 100 validation, and 250 testing scenes.The official test set is excluded because motion annotations are derived from available bounding-box annotations.
- Comparison with state-of-the-art methods: MotionNet is significantly faster than all baselines and overall achieves the best perception and motion-prediction performance.The comparison includes methods based on scene flow, 3D object detection, and occupancy grids.
- Comparison with state-of-the-art methods: MotionNet is compared with Static Model, scene-flow methods, PointRCNN with Kalman filtering, and a multi-step OGM predictor.Motion predictions extend 1s into the future and are evaluated across static, slow, and fast cell-speed groups.
- Ablation studies: Spatial consistency loss benefits moving-cell prediction, while temporal losses improve static-environment learning; their combination further boosts performance.MGDA enhances motion prediction while mildly sacrificing classification accuracy, and its combination with consistency losses produces the best motion predictions.
- Limitations: Bicycle classification accuracy remains low because bicycles are scarce in training data and occupy small BEV regions.Increasing the bicycle training weight does not solve this issue.
- Ablation studies: Using five frames provides an accuracy-efficiency trade-off, while additional frames improve performance at the cost of computation and eventually saturate.Accuracy gains become small beyond five frames.
- Ablation studies: Ground-truth sweep synchronization substantially outperforms unsynchronized input, while ICP provides only partial ego-motion compensation.This ablation establishes synchronization as important for model performance.
- Ablation studies: Middle fusion with STC blocks achieves the best accuracy-efficiency trade-off among the tested spatio-temporal fusion strategies.Early fusion loses temporal correlation for fast objects, whereas late fusion discards low-level motion cues.
5. Conclusion
The paper presents MotionNet for joint perception and motion prediction from BEV maps. Experiments on nuScenes demonstrate its effectiveness and suggest value as a backup and complementary source of information for autonomous-driving motion planning.
- MotionNet jointly performs perception and motion prediction using BEV maps.
- Experiments on nuScenes demonstrate the effectiveness and superiority of the proposed method.
- MotionNet may serve as a backup system and provide complementary information to motion planning in autonomous driving.