Source-linked AI summary
MP3: A Unified Model to Map, Perceive, Predict and Plan
Sergio Casas, Abbas Sadat, Raquel Urtasun
TL;DR
HD maps are costly to scale and impose demanding localization requirements, motivating mapless driving. MP3 predicts probabilistic online scene representations from raw sensors and high-level commands, then uses them in an interpretable neural planner. The model is reported to be safer, more comfortable, and more robust than baselines across real-world and closed-loop evaluations.
Problem
HD maps are difficult and costly to build and maintain, while their use requires centimeter-level localization and map updates for safe driving.
Method
MP3 uses raw sensor data and a high-level command to predict a planning-centric online map and dynamic occupancy flow for an interpretable neural motion planner.
Results
MP3 is reported to be safer, more comfortable, and more robust than baselines across large-scale real-world evaluation and long-term closed-loop simulation.
Takeaways & Limitations
The approach provides probabilistic intermediate representations that are directly usable as interpretable cost functions for mapless driving.
Takeaways & Limitations
Open-loop evaluation plans from expert states and therefore does not reflect actual performance as well as closed-loop execution; mapless routing also assumes a coarse road network and a supplied high-level command.
Abstract
from arXiv · showhide
High-definition maps (HD maps) are a key component of most modern self-driving systems due to their valuable semantic and geometric information. Unfortunately, building HD maps has proven hard to scale due to their cost as well as the requirements they impose in the localization system that has to work everywhere with centimeter-level accuracy. Being able to drive without an HD map would be very beneficial to scale self-driving solutions as well as to increase the failure tolerance of existing ones (e.g., if localization fails or the map is not up-to-date). Towards this goal, we propose MP3, an end-to-end approach to mapless driving where the input is raw sensor data and a high-level command (e.g., turn left at the intersection). MP3 predicts intermediate representations in the form of an online map and the current and future state of dynamic agents, and exploits them in a novel neural motion planner to make interpretable decisions taking into account uncertainty. We show that our approach is significantly safer, more comfortable, and can follow commands better than the baselines in challenging long-term closed-loop simulations, as well as when compared to an expert driver in a large-scale real-world dataset.
1. Introduction
HD maps provide valuable driving structure but are costly to build and maintain and require centimeter-level localization. MP3 addresses mapless driving by predicting interpretable, probabilistic scene representations for safe, command-following planning.
- Motivation: HD maps encode lane topology, geometry, traffic infrastructure, and lane-specific traffic rules that simplify perception, forecasting, and motion planning.Their lane structure also narrows trajectory generation toward motions that follow traffic rules.
- Motivation: HD maps are difficult to scale because they are costly to generate and maintain and require centimeter-level localization accuracy.Localization failures or outdated maps can create unsafe routes, motivating mapless driving as a potential fail-safe.
- Challenges: Mapless driving expands perception, forecasting, and planning challenges because lane and crosswalk priors are unavailable and safe planning must search all dynamically feasible trajectories.The desired goal is therefore expressed through high-level behaviors such as going straight, turning left, or turning right.
- Challenges: Existing mapless approaches often imitate expert controls without interpretable intermediate representations, making decisions harder to explain and potentially brittle under distributional shift.The passage contrasts these methods with approaches that perform online mapping.
- Approach: MP3 predicts probabilistic spatial layers for a planning-centric online map and dynamic occupancy flow, then uses them directly as cost functions in an uncertainty-aware neural planner.The planner also retrieves dynamically feasible trajectories and predicts a route from an abstract driving goal.
2. Related Work
Prior work spans online mapping, object-centric prediction, direct-control driving, and cost-map planning. MP3 is positioned within this landscape by combining interpretable scene representations with downstream motion planning.
- Online Mapping: Offline mapping methods rely on satellite imagery or repeated scene passes and are often unsuitable for mapless driving.Online mapping instead predicts map elements directly from sensor observations.
- Online Mapping: Online mapping methods predict lane layouts from images or estimate ground height and lanes in 3D using LiDAR and cameras.The related work notes that accurate image estimates do not necessarily provide the precise 3D lane boundaries required by motion planners.
- Perception and Prediction: Perception and prediction pipelines detect actors and model multiple futures using trajectory sets, samples, or temporal occupancy maps.Confidence thresholding and non-maximum suppression can remove unconfident detections of real objects.
- Motion Planning: End-to-end motion-planning methods range from direct neural control commands to cost-map approaches that recover trajectories by searching for local minima.The passage notes that direct-control methods may have stability and robustness issues.
- Motion Planning: MP3 predicts probabilistic scene representations and leverages them as interpretable cost functions for motion planning.Its model overview is referenced in Figure 2.
3. Interpretable Mapless Driving
MP3 represents the unknown environment with an interpretable online map and probabilistic dynamic occupancy flow, then uses these representations for route-aware planning. The model captures drivable space, reachable lanes, intersections, object occupancy, multimodal motion, and uncertainty directly from raw sensor data.
- 3.1. Extracting Geometric and Semantic Features: MP3 extracts geometric, semantic, and motion features from raw LiDAR history to estimate the world around the self-driving vehicle.The model uses 10 past LiDAR point clouds, odometry compensation, and a BEV representation.
- 3.2. Interpretable Scene Representations: The online map represents drivable area, intersections, and reachable lanes, including distance and direction relative to reachable lane centerlines.Reachable lanes encode motion paths the vehicle can follow without breaking traffic rules.
- 3.2. Interpretable Scene Representations: The dynamic occupancy field models current object occupancy and future motion for vehicles, pedestrians, and bicyclists on a 0.4 m/pixel BEV grid.Motion is represented by continuous 2D velocity vectors over 11 future time steps spanning up to 5 seconds.
- 3.2. Interpretable Scene Representations: MP3 represents uncertainty with probabilistic grid-cell variables and categorical multimodal motion distributions for dynamic agents.The online map uses distributions including Bernoulli, Laplacian, and Von Mises variables, while each motion location has K possible velocity vectors.
- 3.2. Interpretable Scene Representations: Future occupancy is computed by iteratively propagating occupancy through the temporal motion field, maintaining consistency between predicted motion and occupancy.Bilinear interpolation distributes motion probability locally, and the resulting representation supports interaction and right-of-way reasoning.
4. Experimental Evaluation
The evaluation combines closed-loop simulation, open-loop comparison with expert demonstrations, and qualitative inspection to assess safety, route following, progress, comfort, and robustness.
- Closed-loop evaluation: Closed-loop evaluation uses 164 challenging 18-second scenarios in a LiDAR-based simulator with diverse reactive actors.A scenario succeeds only when the vehicle avoids collisions, follows the route, stays on-road, and avoids opposite traffic.
- Closed-loop evaluation: MP3 outperforms all baselines across closed-loop success, off-route rate, expert-trajectory distance, and meters per event.The reported comparison says MP3 achieves over 3× the success rate, diverges from the route one-third as often, imitates the expert at least twice as closely, and progresses 3× farther per event than any baseline.
- Open-loop evaluation: Open-loop evaluation measures collisions, route and road violations, progress, imitation distance, jerk, and lateral acceleration against expert demonstrations.The dataset contains geographically non-overlapping train, validation, and test splits, supporting evaluation of generalization.
- Open-loop evaluation: MP3 produces the safest, most progressive, and most comfortable open-loop trajectories, while IL and CIL achieve better imitation but are very unsafe.MP3 achieves map-related metrics similar to the best-performing NMP/CNMP baselines.
- Evaluation scope: Open-loop results are secondary because the vehicle always plans from expert states and therefore do not reflect the full effects of executing its own plans.The authors report that MP3 is more robust than baselines to the distributional shift caused by closed-loop unrolling.
- Qualitative results: Qualitative results show command-conditioned route following, multimodal dynamic occupancy predictions, and safe stopping for pedestrians emerging from occlusion.The examples cover keeping straight, turning left, and turning right in crowded scenes.
5. Conclusion
The paper concludes that MP3 is an end-to-end mapless-driving model whose probabilistic intermediate representations support interpretable planning. Across evaluations, it is reported as safer, more comfortable, more progressive, and more robust than the baselines.
- Conclusion: MP3 produces probabilistic, interpretable intermediate representations that can be used directly as cost functions in a neural motion planner.These representations support mapless driving by describing the environment for planning and decision-making.
- Conclusion: MP3 is reported to be safer, more comfortable, and more progressive than state-of-the-art approaches on a large-scale dataset.The conclusion also reports substantial improvements across all metrics in closed-loop simulation without additional training.
- Appendix: The appendix provides implementation details, additional experiments, and further qualitative results.It supplements the main paper rather than introducing a separate evaluation setting.
A.1. Architecture
The architecture extracts multi-resolution sensor features, predicts an online map and dynamic occupancy flow, incorporates command-conditioned routing, and scores trajectories using interpretable uncertainty-aware costs.
- Backbone Network: The backbone combines multi-resolution convolutional blocks to extract geometric, semantic, and motion information from past LiDAR sweeps.It uses four convolutional blocks and a convolutional header with Group Normalization and ReLU activations.
- Mapping Network: The mapping network outputs six online-map channels at 0.2 m/pixel while combining high resolution with a large receptive field.The map represents drivable area, intersections, and other planning-relevant static structure.
- Perception and Prediction Network: Separate class-specific networks predict current occupancy, motion-mode scores, and future motion vectors for vehicles, pedestrians, and bicyclists.Dilation expands the receptive field for long-horizon motion prediction, and future occupancy is obtained by warping initial occupancy.
- Routing Network: The routing network predicts a spatial map of command alignment, using separate parameterized branches for right turns, left turns, and other commands.Each cell estimates whether driving there from the current location matches the high-level command.
- Dynamic Occupancy Flow: Dynamic occupancy flow marginalizes over motion modes and iteratively propagates occupancy from one spatio-temporal grid to the next.The flow-event probability combines occupancy with motion-field transitions, while future occupancy aggregates inflows from preceding cells.
- Trajectory Retrieval: Retrieved trajectories are conditioned on initial velocity, curvature, and acceleration, producing diverse kinematically plausible samples.The trajectories are clustered into 3,000 sets, with one trajectory retained per cluster prototype.
- Trajectory Scoring: Trajectory scoring penalizes occupancy overlap, distance from reachable-lane centers, misaligned heading, headway violations, and map uncertainty.The uncertainty cost multiplies vehicle velocity by the standard deviation of predicted map distributions, promoting slower maneuvers when uncertainty is high.
A.4. Training details
Training proceeds in two stages: multi-task learning for scene representations and routing, followed by max-margin trajectory scoring against expert behavior and safety constraints.
- Multi-task learning: The first stage linearly combines mapping, occupancy, motion, and routing losses to train the backbone and prediction networks.The loss weights are λK,V = 0.1, λM = 0.5, and λR = 2.0.
- Ablation: Supervising warped future occupancy substantially improves closed-loop simulation performance compared with supervising only the motion field.This result is reported in the future dynamic-occupancy loss ablation.
- Trajectory scoring: The second stage uses a max-margin loss because selecting the minimum-cost trajectory from a discrete set is non-differentiable.The loss encourages the expert trajectory to have lower cost than alternative trajectories that differ from demonstrations or are unsafe.
- Trajectory scoring: The safety task loss accounts for collision occurrence and severity at each trajectory step, separately from other planning subcosts.This ensures the cost margin is achieved for colliding trajectories regardless of other costs at other time steps.
B.1. Dynamic occupancy loss ablation
MP3 predicts multi-modal motion vectors and categorical distributions for each spatio-temporal BEV cell, then flows predicted occupancy into future steps. The ablation motivates optimizing future occupancy directly because errors in motion fields can affect safety-critical predictions.
- MP3 predicts multi-modal motion vectors and their categorical distribution for each spatio-temporal BEV grid cell.
- The predicted motion fields flow initial occupancy forward to produce future occupancy estimates.
- Separate losses for initial occupancy and temporal motion fields do not directly optimize future occupancy, which is critical for motion-planning safety.
- Small motion-field errors can affect the occupancy obtained by flowing the initial prediction into future steps.
B.2. Routing ablation
The routing ablation shows that high-level commands improve route following, while adding approximate distance to the commanded action reduces off-route events further. An HD-map upper-bound experiment evaluates the adequacy of the predicted representations and planning costs.
- Routing ablation: A discrete route command allows the SDV to progress better toward the goal, and noisy longitudinal distance to action provides additional help.
- Routing ablation: 14.63%: adding approximate distance to action and CoordConv reduces MP3 out-of-route events in closed-loop simulation.The Action only ablation records 19.51% out-of-route events, compared with 47.56% for CNMP.
- Routing ablation: 47.56%: CNMP goes off-route more often than MP3 Action only, which records 19.51% out-of-route events.
- Upper bound with access to HD map: Using ground-truth online and route maps gives the motion planner an HD-map upper-bound setting while dynamic occupancy still must be inferred from sensor data.
- Upper bound with access to HD map: The upper-bound experiment supports the adequacy of the proposed online-map, route, and motion-planning cost representations, while motivating better static-environment prediction.
C. Qualitative results
In closed-loop simulation, MP3 represents both static and dynamic scene structure through its online map and dynamic occupancy predictions. These representations support routings and safe maneuvers that remain close to expert demonstrations after several seconds of self-generated execution.
- MP3’s online map and dynamic occupancy predictions capture both static and dynamic parts of the environment.
- The predicted representations translate into good routings and safe maneuvers close to expert demonstrations during closed-loop unrolling.The comparison remains close even after MP3’s own plans cause it to deviate from the expert state.
- The planner visualizes sampled trajectories with costs from blue for lowest cost to red for highest cost, while plotting the lowest-cost trajectory separately.
C.2. Plan comparison against baselines in closed-loop simulation
Closed-loop comparisons visualize expert states, planned paths, commanded routes, methods, and temporal execution across multiple scenarios. The examples show MP3 following routes and handling maneuvers where baselines diverge or collide.
- Each comparison column represents a method, each row a video frame sampled every 2.5 seconds, with expert states in black, plans in blue, and commanded paths in orange.
- Figures 11 and 12 provide additional closed-loop qualitative results for MP3’s predicted scene representations and motion plans.
- The qualitative scenarios include yielding, cruising, left-turn, and right-turn cases with trajectory costs and velocity-adapted samples.
- MP3 is the only method that follows the route and avoids collision in one closed-loop example.
- MP3 is the only method that completes the unprotected left turn in another closed-loop comparison.
- MP3 closely imitates the expert, while CIL diverges and TC and CNMP collide with other vehicles despite staying on route.