Source-linked AI summary
Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations
Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, Raquel Urtasun
TL;DR
The paper addresses the challenge of producing safe, interpretable motion plans whose costs remain consistent with perception and prediction. It jointly learns perception, prediction, and planning through differentiable semantic occupancy representations used directly by the planner. On large-scale manual-driving data and closed-loop simulation, the method more closely imitates human trajectories while producing substantially safer trajectories.
Problem
Existing autonomy stacks can lose information through detection thresholding, while neural planners may produce inconsistent perception, prediction, and planning estimates.
Method
The model jointly learns perception, occupancy forecasting, and motion planning using differentiable semantic occupancy layers over space and time as interpretable planning costs.
Results
40% less collisions at 5s compared to PLT, while the proposed model outperforms baselines in almost all planning metrics and imitates human trajectories more closely.
Takeaways & Limitations
Semantic occupancy layers preserve low-probability and arbitrary-shape objects while keeping planned trajectories consistent with perception and prediction outputs.
Abstract
from arXiv · showhide
In this paper we propose a novel end-to-end learnable network that performs joint perception, prediction and motion planning for self-driving vehicles and produces interpretable intermediate representations. Unlike existing neural motion planners, our motion planning costs are consistent with our perception and prediction estimates. This is achieved by a novel differentiable semantic occupancy representation that is explicitly used as cost by the motion planning process. Our network is learned end-to-end from human demonstrations. The experiments in a large-scale manual-driving dataset and closed-loop simulation show that the proposed model significantly outperforms state-of-the-art planners in imitating the human behaviors while producing much safer trajectories.
1 Introduction
The paper targets safe, interpretable end-to-end autonomy by addressing information loss and inconsistent estimates in existing neural planning stacks. It introduces differentiable semantic occupancy representations and reports more human-like, safer trajectories.
- Traditional autonomy stacks use thresholded detections and NMS, which can discard information needed for safety-critical decisions.
- Existing end-to-end neural planners often prioritize human-like control outputs without producing interpretable intermediate representations.
- Shared feature representations with separate task headers can yield inconsistent perception, prediction, and planning estimates, allowing planners to ignore detections or forecasts.
- The proposed semantic layer represents probabilistic occupancy over space and time, including low-confidence and potentially occluded vehicles, bicyclists, and pedestrians.
- The differentiable occupancy representation is used directly to penalize risky maneuvers, while training remains end-to-end from raw sensor data to planned trajectory.
- Experiments show closer imitation of human trajectories and much lower collision rates than existing approaches.
2 Related Work
Prior work spans direct-control networks, cost-map planners, imitation learning, multi-task learning, and occupancy-based prediction. The paper distinguishes its approach through future semantic occupancy layers that remain interpretable, differentiable, efficient, and able to retain low-probability objects.
- End-to-end self-driving: End-to-end self-driving methods either directly predict control commands or produce cost maps from which trajectories are recovered.
- Perception and Motion Prediction: Some end-to-end perception-and-planning approaches omit prediction of how the scene evolves, whereas this planner uses predicted semantic occupancy maps as interpretable cost terms.
- Imitation learning and inverse reinforcement learning: Imitation learning regresses driving commands from human demonstrations, while max-margin planning instead reasons about trajectory costs and makes human trajectories least costly.
- Multi-task learning: The framework uses multi-task learning with a joint objective covering object detection, occupancy forecasting, and motion planning.
- Perception and Motion Prediction: The model directly produces scene-level occupancy layers rather than actor-specific outputs and avoids detection-score thresholding, allowing low-probability objects to reach the planner.
3 End-to-End Interpretable Neural Motion Planner
The planner jointly learns perception, semantic occupancy forecasting, and motion planning through interpretable representations designed to support safe decision-making. Semantic occupancy predictions directly inform sampled trajectory costs while preserving uncertainty, object shape, and route-dependent distinctions.
- 3 End-to-End Interpretable Neural Motion Planner: The model uses LiDAR, HD maps, and the intended route to generate interpretable semantic occupancy representations over present and future space-time.The representation captures dynamic agents by semantic class and supports arbitrary shapes and low-probability objects.
- 3.1 Perceiving and Forecasting Semantic Occupancies: Occupancy layers cover vehicles, bicyclists, pedestrians, and occluded objects, with route-dependent subclasses that support distinct planning costs.The hierarchy distinguishes interactions such as oncoming and crossing traffic and can support different safety buffers for parked vehicles and oncoming traffic.
- 3.1 Perceiving and Forecasting Semantic Occupancies: Occupancy forecasting models semantic groups with non-parametric distributions, avoiding detection and tracking while scaling better to crowded scenes.The representation is instance-free and represents groups of actors with a single future occupancy distribution.
- 3.1 Perceiving and Forecasting Semantic Occupancies: The forecasting network fuses local and coarse multi-scale features, then uses recurrent occupancy updates to predict logits across future horizons.The recurrent design uses a larger receptive field for farther horizons, while lower-resolution recurrence reduces memory impact.
- 3.2 Motion Planning: Motion planning samples diverse ego trajectories and selects the minimum-cost trajectory using occupancy-based safety costs together with comfort, traffic-rule, and route-related costs.The sampled set is kept small enough for real-time computation while covering varied maneuvers.
- 3.3 Learning: The planner is trained end-to-end with losses for semantic occupancy, imitation, and safety-aware max-margin planning.The max-margin objective encourages the human trajectory to have lower cost than alternative trajectories that differ behaviorally or are unsafe.
4 Experimental Evaluation
The evaluation compares P3 with established planners in open-loop, ablation, qualitative, and closed-loop settings. Across these tests, semantic occupancy and joint training support safer, human-like, and cautious driving behavior.
- Dataset and baselines: The dataset contains 6,100 training, 500 validation, and 1,500 test scenarios, each lasting 25 seconds.Compared with KITTI, it provides 33x more driving hours and 42x more objects.
- Dataset and baselines: P3 is compared with ACC, imitation learning, NMP, and PLT using collision rate, human-trajectory L2 distance, jerk, lateral acceleration, and route progress.These metrics assess safety, behavioral imitation, comfort, and advancement along the route.
- Open-loop results: 40% fewer collisions at 5s than PLT were achieved by P3 in open-loop evaluation, while P3 outperformed nearly all baselines across planning metrics.The evaluation assumes each generated trajectory is executed for a 5-second planning horizon without replanning.
- Ablation study: End-to-end training of perception and planning improved safety, while occupancy representations produced better driving metrics without making route progress overly cautious.The best model, M5, jointly trains semantic occupancy and motion planning; M3 rasterizes predicted trajectories into occupancy for planning.
- Qualitative results: The semantic occupancy layers represent multiple actor classes, route-related vehicle categories, occlusions, multiple prediction modes, and uncertainty across time horizons.Qualitative examples include conflicting and oncoming vehicles, recognized occluded regions, and larger uncertainty for a low-velocity oncoming vehicle.
- Closed-loop evaluation: In closed-loop simulation, P3 safely reacts to initially occluded, route-conflicting non-compliant vehicles and produces fewer collisions than M2.The test covers 80 challenging scenes over 15 seconds, with trajectories replanned every 100ms; cautious behavior is also reflected in jerk and acceleration metrics.
5 Conclusion
The proposed end-to-end model generates safe trajectories while keeping its ego-vehicle trajectories consistent with interpretable perception and prediction outputs. Experiments show human-like driving behavior and significantly greater safety than state-of-the-art learnable planners.
- The model jointly performs perception, prediction, and motion planning from raw sensor data while producing interpretable intermediate representations.
- Semantic occupancy layers preserve information about low-probability objects for motion planning instead of using thresholded detections and trajectory predictions.
- Closed-loop simulations and large-scale challenging scenarios show human-like driving behavior and significantly safer trajectories than state-of-the-art learnable planners.
Supplementary Material
The supplementary material expands the paper’s technical and experimental documentation, covering trajectory generation, planning costs, training, and qualitative results.
- The supplement provides detailed trajectory parameterization and sampling procedures.
- It also overviews the planning cost functions and adds further training details and qualitative results.
A Trajectory Parametrization and Sampling
The planner represents trajectories using lane-relative Frenet coordinates and samples longitudinal and lateral behaviors to produce higher-quality lane-based bicycle-model trajectories.
- Planner outputs are sequences of bicycle-model states containing position, heading, velocity, curvature, and acceleration.Curvature can be converted to steering angle.
- Frenet-frame sampling represents trajectories by longitudinal position and lateral offset relative to the desired lane’s driving path.Higher-order derivatives are included in the lane-relative representation.
- The procedure samples longitudinal velocity profiles including stopping, accelerating to a specific velocity, and maintaining current velocity.
- Lateral samples include nudging, lane-changing, and following the driving path, yielding lane-based trajectories with maneuver variations.
- Longitudinal and lateral trajectories are represented with quartic and quintic polynomials, respectively, using sampled mid- and end-conditions.Longitudinal acceleration is fixed at zero at the specified intermediate and terminal times.
B Motion Planner Cost Functions
The motion planner evaluates candidate trajectories with costs for safety, lane and road adherence, traffic rules, route progress, future requirements, dynamics, and comfort.
- Collision, Safety-margin, and Headway: Collision and safety-distance costs penalize spatio-temporal overlap with predicted actor trajectories and violations of safety margins.Distances are computed between the SDV polygon and predicted actor polygons at each timestep.
- Driving-path, Lane and Road Boundaries: Lane and road-boundary costs penalize leaving lane or road boundaries and deviating from the lane’s driving path.
- Speed-limit, Traffic Lights, and Stop Signs: Speed-limit and stop-line costs penalize regulated-speed violations and failure to stop for red lights or stop signs.
- Route, Progress, and Cost-to-go: Route costs penalize lane changes needed to return to the high-level route and divergence toward dead-end lanes, while progress receives a reward.
- Route, Progress, and Cost-to-go: Cost-to-go penalizes the deceleration required beyond the planning horizon for upcoming speed limits, stop signs, or red lights.
- Dynamics and Comfort: Dynamics and comfort costs prune infeasible trajectories and penalize aggressive acceleration, steering, jerk, lateral acceleration, curvature, and derivative violations.
C Training details
Training jointly optimizes perception, occupancy forecasting, and motion planning, using separate optimization strategies and class-imbalance handling for semantic occupancy.
- Adam updates the perception backbone and occupancy forecasting networks with a base learning rate of 1e−5.
- Exponentiated gradient descent optimizes planning parameters while keeping subcost weights greater than zero, using α = 1e −3.
- Both learning rates are linearly scaled with batch size.
- The semantic occupancy loss has weight λS = 1, while the motion-planning max-margin loss has weight λM = 1e −3.
- Hard negative mining selects all positives and the 10 · |Post,c| hardest negatives to form the training pixel subset.
D Architecture details
The architecture recurrently updates multi-scale occupancy predictions using parallel convolutional streams and concatenated previous occupancy logits.
- The recurrent occupancy-update architecture uses two parallel fully convolutional streams with different dilation rates for multi-scale context fusion.
- One stream applies two regular convolutions at dilation 1 to 128-channel features, while the other uses dilation 2 on coarser features.
- At each future timestep, occupancy logits are updated by concatenating interpolated previous logits with occupancy features and applying Uθ.Uθ is a two-layer CNN whose first convolution upsamples by 2 and whose second is a regular convolution.
- The logits l_t,c represent the root-class occupancy prediction at future timestep t.
E Additional Qualitative Results
Additional qualitative figures show current and forecast occupancies across scenarios involving occlusion, obstacles, lane relevance, pedestrians, and lane changes.
- Figures 10–14 include occupancy at the current time and forecasts for future timesteps.
- The figure legend identifies actor subcategories, while the camera view illustrates lane regions occluded by an oncoming truck.
- An oncoming truck and stationary vehicles are handled by nudging around a parked vehicle while continuing along the route.
- When a garbage truck enters the right lane, the SDV nudges around it, while the occupancy representation also covers a nearby person.
- A vehicle in a left-turn lane is categorized as irrelevant to the planner as it approaches the intersection.
- The qualitative examples depict cautious SDV behavior while pedestrians cross the street.
- With large vehicles occupying the oncoming lane and a truck encroaching slightly, the planner lane-changes right instead of continuing in the same lane as the human driver.