Source-linked AI summary
Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients
Oliver Scheel, Luca Bergamini, Maciej Wołczyk, Błażej Osiński, Peter Ondruska
TL;DR
Urban driving requires policies that handle complex maneuvers and interactions, yet conventional planning depends on hand-engineered costs that scale poorly. The paper trains an offline imitative policy with policy gradients through a differentiable, data-driven simulator built from real-world logs, mid-level representations, and HD maps. It reports strong generalization and performance, including control of a real-world self-driving vehicle and better driving performance than other state-of-the-art ML methods.
Problem
Urban driving requires diverse maneuvers and interactions, while expert-defined planning costs scale poorly with complex situations and rare events.
Method
The method trains imitative driving policies offline with policy gradients in a closed-loop differentiable simulator built from real-world demonstrations, mid-level representations, and HD maps.
Results
The approach generalizes well to complex urban driving, performs varied maneuvers, controls a real-world self-driving vehicle, and yields better driving performance than other state-of-the-art ML methods.
Takeaways & Limitations
Closed-loop training from demonstrations can produce complex urban-driving policies without complex state perturbations or additional on-policy training data.
Takeaways & Limitations
The authors identify further extension toward production-grade L4 and L5 requirements, especially novel or rarely seen scenarios and improved sample efficiency.
Abstract
from arXiv · showhide
In this work we are the first to present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This is achieved by building a differentiable data-driven simulator on top of perception outputs and high-fidelity HD maps of the area. It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients. We train our proposed method on 100 hours of expert demonstrations on urban roads and show that it learns complex driving policies that generalize well and can perform a variety of driving maneuvers. We demonstrate this in simulation as well as deploy our model to self-driving vehicles in the real-world. Our method outperforms previously demonstrated state-of-the-art for urban driving scenarios -- all this without the need for complex state perturbations or collecting additional on-policy data during training. We make code and data publicly available.
1 Introduction
Urban driving decision-making remains difficult because expert-designed planning costs scale poorly to complex situations, while urban policies must handle diverse interactions and maneuvers. The paper addresses this with offline policy gradients, a differentiable simulator, and vectorized mid-level representations learned from real-world demonstrations.
- Urban planning systems still rely heavily on expert-defined trajectory-optimization costs, whose engineering scales poorly with complex situations and rare events.
- Urban driving requires varied maneuvers and interactions with traffic lights, other cars, and pedestrians.
- The paper presents offline policy-gradient learning for imitative urban-driving policies using mid-level representations and a closed-loop simulator.
- The differentiable simulator synthesizes realistic driving episodes from real-world logs and HD maps, while vectorization enables policy gradients through backpropagation through time.
- The method is evaluated quantitatively and qualitatively, including simulation-trained control of a real-world self-driving vehicle, generalization, and optimization of imitation and auxiliary costs.
2 Related Work
Related work spans optimization-based planning, reinforcement learning, imitation learning, neural motion planning, mid-level representations, and data-driven simulation. The paper positions its approach as using real-world logs and mid-level representations to address limitations of hand-engineered simulators and behavioral cloning.
- Trajectory-based optimization remains dominant in industry, using manually defined costs with interpretable methods and functional guarantees.
- Reinforcement learning reduces some human engineering through reward signals, but real-time disengagement rewards are impractical because of low sample efficiency and risk.
- Hand-engineered simulators can limit coverage of long-tail real-world scenarios, while prior sim-to-real work did not focus on typical urban interactions with other agents.
- Imitation learning scales through expert demonstrations, but naive behavioral cloning suffers from covariate shift.
- Mid-level representations combine perception outputs with HD maps, compactly capturing urban-scene nuance for large-scale training on real driving situations.
- Data-driven simulators are intended to capture more diverse real-world behavior than simulators based on vehicle-control heuristics.
3 Differentiable Traffic Simulator from Real-world Driving Data
The simulator generates new closed-loop driving experiences from real-world logs by transforming vectorized scene states according to learner actions and differentiable vehicle dynamics. This enables efficient policy-gradient computation through backpropagation, but the approximation is effective only when generated poses remain sufficiently close to the original pose.
- The differentiable simulator approximates new driving experiences from collected real-world experiences for closed-loop policy evaluation and policy-gradient computation.
- Each vectorized state contains static map elements and dynamic traffic participants represented relative to the self-driving vehicle pose.Static elements include lanes, stop signs, and pedestrian crossings; dynamic elements include traffic lights and detected road users.
- The simulator generates states for a different action sequence by computing the corresponding self-driving trajectory and locally transforming logged states.
- Updated vehicle poses follow differentiable kinematic dynamics, after which scene-element positions are recomputed for the new state.
- The simulator approximation is effective only when the distance between the original and generated self-driving-vehicle poses is not too large.
- Differentiability with respect to states and actions enables efficient policy gradients through backpropagation through time.
4 Imitation Learning Using a Differentiable Simulator
The method learns a deterministic driving policy through closed-loop interaction with a differentiable simulator built from expert demonstrations. It estimates policy gradients by differentiating through simulated rollouts while discarding biased initial steps.
- Objective: The imitation objective minimizes the L1 pose distance between expert and learner trajectories over collected real-world demonstrations.The poses are represented as 3D vectors (x, y, θ).
- Gradient computation: Differentiability of the transition model enables policy-gradient computation by backpropagation through time without separately estimating a value function.The recursive gradient is computed along the current policy rollout.
- Trajectory sampling: The simulator generates policy trajectories by unrolling the current policy for T steps from a sampled expert state.Expert trajectories are sampled from the demonstration dataset before policy rollout.
- Trajectory sampling: Discarding the first K timesteps removes bias caused by initializing rollouts from the expert rather than the policy distribution.The remaining T − K steps are used to estimate the policy gradient.
- Gradient computation: Training uses one forward pass of length T and one backward pass of length T − K to update policy parameters recursively.Equations (4) and (5) are applied from t = T to t = K.
5 Experiments
Experiments evaluate the proposed policy on urban driving data against three baselines using normalized imitation, safety, and comfort metrics. The method achieves the best overall simulated performance and is also deployed on a real self-driving vehicle.
- Baselines: The study compares the proposed method with naive behavioral cloning, behavioral cloning with perturbations, and MS Prediction.All baselines use the same network backbone, with model-specific differences.
- Implementation: The policy uses a PointNet-like local encoder followed by scaled dot-product attention for global aggregation and trajectory prediction.The architecture contains around 3.5 million trainable parameters.
- Metrics: Table 1 reports normalized mean and standard deviation over 3 runs, with lower values preferred for every metric.The metrics include L2 error, off-road events, collisions, comfort failures, and I1K.
- Evaluation: The evaluation uses 3600 test sequences, each 25 seconds long, to unroll every policy and measure planning performance.Metrics include imitation accuracy, safety, and comfort.
- Results: The proposed method yields the best overall results and the lowest I1K, while imitation-only training leaves all models with many comfort failures.The comfort issue arises because imitation training optimizes positional accuracy rather than comfort.
- Real-world deployment: A Ford Fusion equipped with 7 cameras, 3 LiDAR, and 11 radar sensors was used to deploy the planner in real-time on a private test track.The vehicle used the same sensor setup as the data-collection platform and processed scene representations onboard.
6 Conclusion
The paper concludes that closed-loop training with mid-level representations, a data-driven simulator, and real-world demonstrations can produce urban driving policies with good generalization. The trained policy controls a real self-driving vehicle and handles diverse traffic situations qualitatively.
- 6 Conclusion: The proposed approach combines closed-loop training, mid-level representations, a data-driven simulator, and a large corpus of real-world demonstrations.It is presented as a method for learning autonomous driving policies in urban settings.
- 6 Conclusion: The method shows good generalization and performance for complex urban driving.The conclusion specifically connects these outcomes to the combined training and simulation approach.
- 6 Conclusion: The learned policy controls a real-world self-driving vehicle with better driving performance than other state-of-the-art machine-learning methods.This result is stated as part of the paper’s concluding evaluation.
- Future scope: The paper identifies extending the approach to production-grade L4 and L5 requirements as future work, especially for novel scenarios and scaling data efficiency.The stated direction includes improving rarely seen scenarios and scaling toward millions of driving hours.
Appendix A: Qualitative results
The method handles diverse and challenging traffic situations in simulation and is deployed to control self-driving vehicles in real-world scenes.
- Qualitative simulation: The method handles diverse, complex traffic situations well in qualitative simulation results.Figure 6 depicts scenes with the self-driving vehicle, other agents, crosswalks, and traffic-light states over time.
- Real-world deployment: The trained planner navigates a multitude of challenging scenarios during real-world self-driving vehicle deployment.Figure 7 presents real-world scenes read from left to right.
- Real-world deployment: Additional deployment material includes different camera angles, scene understanding, and the vehicle’s planned trajectory.
Results for Optimizing Auxiliary Costs
The method combines imitation with auxiliary objectives and can trade off imitation performance against comfort and collision-related costs during closed-loop training.
- Objective: The mixed objective combines L1 imitation loss with auxiliary losses to tailor behavior toward comfort or safety.This formulation blends imitation learning with reinforcement learning.
- Acceleration and comfort: The method is the only evaluated approach that successfully trades off imitation and comfort as the acceleration weight α changes.Increasing α slightly raises I1K while allowing comfort failures to be reduced to arbitrary levels.
- Collision cost: Collision-loss results are closer across methods, which the authors hypothesize follows from setting α = 0 and permitting one-step corrections.
- Training factors: Performance improves with more training data and with larger K, although the data-size curve appears to approach a plateau.Figure 9 identifies proper sampling through K as important for closed-loop training.
Discussion on Used Metrics
The appendix tests metric sensitivity and finds that reported trends are relatively robust to threshold changes, while finer comfort metrics reveal additional behavior differences.
- Sampling sensitivity: Proper sampling matters because performance increases with growing K.
- Threshold sensitivity: At a 4m off-road threshold, off-road failures increase while other metrics improve, yet differences remain small and relative trends hold.Table 4 repeats the main evaluation with the wider threshold.
- Comfort metrics: Longitudinal jerk and lateral acceleration provide finer-grained comfort measures than the paper’s simpler acceleration metric.Table 5 reports these additional comfort values.
- Comfort metrics: Most measured discomfort comes from longitudinal acceleration and jerk, while lateral movement is smoother across methods.
- Comfort trade-offs: The method remains the only one able to jointly optimize performance and comfort, with larger α yielding smoother driving.The number of jerk failures exceeds the number of acceleration failures, motivating explicit jerk penalties in future experiments.
- Collision interpretation: Rear collisions can reflect other traffic participants or non-reactive simulation, but may also indicate severe misbehavior such as unsafe braking.The evaluation therefore includes rear collisions in its aggregation.
Appendix C: Policy architecture and state representation
The policy represents urban scenes as sets of point-based static and dynamic elements, processes them with PointNet and attention, and outputs trajectories for closed-loop driving. Its differentiable collision loss models vehicles with overlapping circles.
- Policy architecture: Three PointNet layers independently embed each object’s points into a 128-dimensional descriptor before attention-based aggregation.Sinusoidal point embeddings encode order, and the resulting object descriptors are processed by scaled dot-product attention.
- State representation: Each state element contains multiple points representing features of agents, lanes, and other scene entities.The state uses variable numbers of points per element, with feature vectors padded to a fixed dimension.
- Policy architecture: Attention uses the ego descriptor as the key and all object descriptors as keys and values before an MLP projects features to a trajectory.Object-type embeddings allow attention to incorporate the types of attended entities.
- State representation: The model input includes static and dynamic elements within a 35m circular field of view around the SDV and is queried once before unrolling.This state includes up to the listed maximum number of elements and remains fixed during the rollout.
- Differentiable collision loss: The collision loss approximates each vehicle with three circles and evaluates pairwise intersections over the relevant rollout timesteps.The pairwise term uses the maximum intersection among circle pairs for the SDV and another vehicle.
- Differentiable collision loss: Circle intersection loss is zero without overlap and increases linearly to one as the intersection grows.The calculation depends on center distance and circle radii.