Source-linked AI summary
Urban Driving with Conditional Imitation Learning
Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemyslaw Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, Alex Kendall
TL;DR
Real-world urban autonomous driving remains difficult because roads are unstructured, situations are diverse, and effective policies must combine high-level reasoning with low-level control. This paper develops conditional imitation learning for route-following with lateral and longitudinal control on a real vehicle, using 30 hours of demonstrations and evaluating generalisation to unseen urban routes.
Problem
Urban driving involves highly unstructured roads, a large state space, and the need to combine hierarchical reasoning with low-level control.
Method
The method is an end-to-end conditional imitation learning policy that maps monocular camera inputs and route commands to full-vehicle speed and steering control.
Results
30 hours of demonstrations collected over six months produced a policy that generalised to routes it had not been trained to perform in European urban streets.
Takeaways & Limitations
Intermediate computer-vision representations, diverse data, and increased scene observability are supported as important components for learned urban driving, while multiview models reduce single-view failure modes.
Takeaways & Limitations
The method lacks long-term dependencies, road-scene reasoning, and predictive long-term planning for safe interaction with occluded dynamic agents.
Abstract
from arXiv · showhide
Hand-crafting generalised decision-making rules for real-world urban autonomous driving is hard. Alternatively, learning behaviour from easy-to-collect human driving demonstrations is appealing. Prior work has studied imitation learning (IL) for autonomous driving with a number of limitations. Examples include only performing lane-following rather than following a user-defined route, only using a single camera view or heavily cropped frames lacking state observability, only lateral (steering) control, but not longitudinal (speed) control and a lack of interaction with traffic. Importantly, the majority of such systems have been primarily evaluated in simulation - a simple domain, which lacks real-world complexities. Motivated by these challenges, we focus on learning representations of semantics, geometry and motion with computer vision for IL from human driving demonstrations. As our main contribution, we present an end-to-end conditional imitation learning approach, combining both lateral and longitudinal control on a real vehicle for following urban routes with simple traffic. We address inherent dataset bias by data balancing, training our final policy on approximately 30 hours of demonstrations gathered over six months. We evaluate our method on an autonomous vehicle by driving 35km of novel routes in European urban streets.
I. INTRODUCTION
The paper addresses autonomous driving in complex urban environments by learning from human demonstrations rather than relying solely on hand-crafted rules. It presents conditional imitation learning for route following, full vehicle control, and simple traffic interaction, evaluated on real urban roads.
- Urban driving combines a large state space, irregular road structure, varied traffic behaviour, and the need for both high-level reasoning and low-level control.
- Prior imitation-learning systems often covered only part of the state or action space, with limited route following, control, observability, or traffic interaction.
- The proposed end-to-end conditional imitation-learning policy follows user-prescribed routes while controlling both steering and speed on a real vehicle in complex urban scenarios.
- 30 hours of demonstrations collected over six months supported a policy that generalised to unseen routes in challenging European urban streets.
- 35km of public-road driving provided a comprehensive evaluation of the learned policy in real urban environments.
- The architecture learns semantic, geometric, and motion representations from camera inputs, fuses them, and produces a short steering-and-speed motion plan conditioned on a route command.
II. RELATED WORK
Related work spans imitation learning, simulation, bird’s-eye-view representations, and reinforcement learning. This paper extends prior driving policies to full control, conditional route following, traffic interaction, and real urban-road evaluation.
- Earlier imitation-learning systems addressed lane following, conditional navigation, or limited control, while some learned both steering and speed only in structured toy-vehicle settings without traffic.
- Simulation benchmarks enable extensive driving-policy research, but simulation lacks real-world complexity and offers a lower-entropy state space with near-infinite data.
- Bird’s-eye-view approaches use fused scene representations or LiDAR but require additional perception, sensor-fusion, or high-definition mapping infrastructure.
- Reinforcement-learning approaches had shown results in simulation or limited real-world settings but had not yet been evaluated in real urban environments.
- This work learns full lateral and longitudinal control with conditional route following and traffic behaviour, evaluating novel routes over 35km of urban driving.
III. METHOD
The method uses conditional imitation learning to map observations and a high-level route command to driving actions. Its fully end-to-end network is conceptually organized into perception, sensor fusion, and control.
- Imitation learning trains a policy to map observations to actions that imitate an expert, while minimizing the distance between predicted and demonstrated actions.
- Conditional imitation learning adds a route command to disambiguate multiple valid behaviours at situations such as intersections.
- The proposed policy learns directly from images and outputs a local motion plan containing speed and steering.
- The architecture is fully end-to-end but is conceptually divided into perception, sensor fusion, and control components.
1) Perception:
The perception system learns compact scene representations from camera images, combining semantics, geometry, and motion information. Multiple views improve observability, while feature corruption addresses causal confusion from motion inputs.
- Representation learning: A deep encoder-decoder reconstructs RGB, depth, and segmentation, producing features that encode appearance, semantics, and distance information.
- Representation learning: Optical-flow features are concatenated with visual features to provide temporal motion information without relying on concatenated past-frame representations.
- Multiview perception: The system can process front, left, and right camera views independently before sensor fusion aggregates them into a driving representation.
- Multiview perception: Multiview inputs improve state observability, but naive fusion can make the model over-rely on spurious side-view information.
- Causal confusion: Causal confusion can make additional motion information harmful by encouraging spurious correlations, such as an identity mapping from current speed to predicted speed.
- Causal confusion: Random noise and dropout on flow features break correlations during training, encouraging the model to use motion information without learning the trivial speed-and-steering identity mapping.
3) Control:
The control module conditions local speed-and-steering motion plans on route commands and trains them against future expert actions. Data balancing addresses skewed driving distributions across steering and speed.
- Route conditioning: Route commands are injected into the control module as one-hot vectors for going straight, turning left, or turning right.The command is input at multiple network stages to improve robustness.
- Motion planning: The controller outputs locally linear motion plans for both speed and steering, parameterized by the current prediction and its slope.The method assumes vehicle motion is locally linear.
- Training objective: Training minimizes discounted mean squared error between predicted motion plans and expert speed-and-steering actions over N future timesteps.Future discounting weights errors across the prediction horizon.
- Training objective: Predicting action trends into the future produces smoother vehicle motion and improves performance in both closed-loop and open-loop testing.The stated benefit comes from forecasting future expert controls rather than only matching the current action.
- Data balancing: The training sampler balances data uniformly across lateral and longitudinal control dimensions to counter the dominance of near-straight, mid-lane driving.Steering bins are constructed using both sample counts and bin widths, then balanced recursively with respect to speed.
IV. EXPERIMENTS
The experiments test perception, motion, data-diversity, and end-to-end training choices for urban driving. Models are evaluated in closed loop on real European streets, where simulation substantially simplifies the task.
- Experimental questions: The experiments assess explicit visual representations, multiview observability, optical flow, and training-data diversity as factors affecting driving performance.These questions motivate the model variants evaluated in the study.
- Model variants: SV is the single-forward-view baseline with pretrained frozen perception and a local linear speed-and-steering plan predicted one second ahead.The baseline trains sensor fusion and control while keeping perception fixed.
- Model variants: MV integrates forward, left, and right camera views, while MVF adds optical-flow information to the multiview model.These variants isolate wider spatial observability and temporal motion cues.
- Model variants: SV75, SV50, and SV25 use progressively less training data, while SVE2E and SVE2EFT test end-to-end training and fine-tuning alternatives.SV75 discards the latest 25% of data, with analogous truncation for SV50 and SV25.
- Evaluation setting: All models are evaluated in closed loop on real-world European urban streets, and SV performs significantly better in simulation than in the real world.The simulation comparison highlights the simplification of simulated driving conditions.
A. Procedure
The procedure uses six-month human-driving data and novel urban-route trials, measuring safety-driver interventions during lane following, turning, and traffic interaction. Outdoor tests retain unavoidable time-dependent variation.
- Data collection: Human demonstrations were collected for 30 driving hours over six months in a densely populated European-style urban environment.Drivers chose random routes without explicit instructions.
- Data collection: The dataset includes synchronized front, left, and right cameras together with driver speed and steering measurements.Images were captured at 15Hz, while scalar controls were received at 100Hz.
- Evaluation routes: Evaluation uses two approximately 1km routes absent from training to test lane following, intersection turns, and interaction with other road agents.Each route is driven in both directions, with each direction counted as an attempt.
- Traffic interaction: Pace-car trials measure interventions for stopping behind a stationary vehicle or failing to follow a vehicle that periodically stops.These tests directly assess interaction with another vehicle.
- Evaluation metric: Safety-driver intervention number and type are used because multiple driving behaviors can complete the same task correctly.The safety driver takes control when behavior is poor.
- Evaluation boundary: Testing routes and the test vehicle were unseen during training, although some route sections overlap with training data.The outdoor environment also varies in weather, lighting, vehicles, objects, and other agents.
B. Training Procedure
The network is trained jointly with stochastic gradient descent for 200k iterations, using a decaying learning rate and fixed momentum and weight decay. Architecture size varies with perception encoder and camera count.
- Optimization: Joint training uses stochastic gradient descent with batches of 256 for 200k iterations.The initial learning rate is 0.01 and decays linearly during training.
- Optimization: Momentum is set to 0.9 and weight decay to 0.0001 throughout training.These values accompany the stated SGD schedule.
- Architecture size: The network contains 13–26M trainable parameters, depending on the perception encoder and number of cameras.Parameter count is reported at the whole-network level.
C. Metrics
The evaluation measures route-following, lane-following, turning, traffic-following, stopping, and intervention outcomes across static and interactive driving situations.
- Intervention rate is measured in metres per intervention, including a lane-following-only subset.
- Turning performance is reported as the success rate for left and right manoeuvres.
- Traffic-following evaluation measures intervention rate behind a pace car travelling up to 15 km/h and stopping success behind it.
- Interventions are categorized as obstacles, road position, disobeyed navigation, failed manoeuvres, or poor interaction with dynamic vehicles.
- The learned policy is evaluated on two approximately 1km urban routes with multiple intersections, in both directions, with route A slightly differing on the return.
- Training data are collected across a European city and are imbalanced, with most driving straight and a significant stationary portion.
D. Results
The results compare representation and temporal-information choices in real-world urban driving. Learned scene understanding and multiview inputs improve robustness, while optical flow particularly improves interaction with other vehicles.
- 34.4km of driving was performed under automation, separating static manoeuvres from traffic following behind a pace car.
- Learned scene understanding is critical for robust driving representation, while fully end-to-end trained models perform very poorly.
- The perception model is more robust to real-world appearance diversity, partly because of additional data from initial perception training.
- End-to-end models perform comparably to SV in shaded environments but poorly in bright sunlight; the authors attribute this to seasonal appearance differences between training and testing.
- Multiview inputs improve scene observability and reduce navigation and kerb-related failures associated with the single-view model.
- Optical flow slightly worsens some metrics but improves responses to other vehicles, raising stopping success from 65% to 81%.
3) Influence of data quantity and diversity on performance:
The policy’s performance depends strongly on training-data quantity and diversity, while broader system limitations still constrain progress toward human-level driving.
- Removing one quarter of the data notably degrades performance, and models trained with less data become almost undriveable.The authors attribute this degradation to reduced data diversity.
- Models perform better under environmental conditions closer to the data-collection period, indicating sensitivity to distributional mismatch.The collection period concluded three months before the experiments.
- The best-case models drove nearly 2km without intervention, but the upper performance limit of additional data remains unknown.The authors expect more data and greater diversity to improve performance.
- The method lacks long-term dependencies, scene reasoning, and predictive long-term planning for safe interaction with occluded dynamic agents.These limitations constrain robust interaction and future progress toward human-level driving.
- Closed-loop evaluation requires substantial time and safety measures, motivating research into robust offline policy evaluation and performance quantification.Learning from corrective interventions is also identified as an important direction for handling real-world complexity across cities.