Source-linked AI summary
Query-Efficient Imitation Learning for End-to-End Autonomous Driving
Jiakai Zhang, Kyunghyun Cho
TL;DR
End-to-end imitation learning must handle states that supervised policies never encounter, while DAgger’s repeated reference-policy queries can be expensive. SafeDAgger adds a safety policy to select hard examples for labeling, and simulated driving experiments report fewer queries and faster, more efficient training.
Problem
Supervised imitation learning can produce unpredictable behavior outside reference-policy states, while DAgger requires costly reference-policy queries.
Method
SafeDAgger extends DAgger with a safety policy that predicts primary-policy error and selects a small subset of examples for reference-policy queries.
Results
SafeDAgger required much fewer reference-policy queries than DAgger and achieved superior driving performance in simulated autonomous driving.
Takeaways & Limitations
SafeDAgger can collect progressively difficult examples while minimizing reference-policy queries and train the primary policy more efficiently.
Takeaways & Limitations
The learned policy’s quality remains limited by the quality of the reference policy.
Abstract
from arXiv · showhide
One way to approach end-to-end autonomous driving is to learn a policy function that maps from a sensory input, such as an image frame from a front-facing camera, to a driving action, by imitating an expert driver, or a reference policy. This can be done by supervised learning, where a policy function is tuned to minimize the difference between the predicted and ground-truth actions. A policy function trained in this way however is known to suffer from unexpected behaviours due to the mismatch between the states reachable by the reference policy and trained policy functions. More advanced algorithms for imitation learning, such as DAgger, addresses this issue by iteratively collecting training examples from both reference and trained policies. These algorithms often requires a large number of queries to a reference policy, which is undesirable as the reference policy is often expensive. In this paper, we propose an extension of the DAgger, called SafeDAgger, that is query-efficient and more suitable for end-to-end autonomous driving. We evaluate the proposed SafeDAgger in a car racing simulator and show that it indeed requires less queries to a reference policy. We observe a significant speed up in convergence, which we conjecture to be due to the effect of automated curriculum learning.
1 Introduction
End-to-end autonomous driving learns a policy that maps sensory inputs to driving actions, commonly through imitation learning. SafeDAgger extends DAgger to reduce expensive reference-policy queries while improving training efficiency in simulated driving.
- End-to-End Autonomous Driving: End-to-end autonomous driving maps sensory inputs, such as front-facing camera images, to driving actions including steering and braking.Because manually developing such systems is highly complex, they are often learned from data.
- Supervised Imitation Learning: Supervised imitation learning trains a neural-network primary policy from image-action pairs collected while a human or rule-based reference policy drives.The reference policy supplies the training examples used to tune the primary policy.
- Limitations of Supervision: Purely supervised imitation learning is suboptimal because the trained policy can reach states outside those visited by the reference policy, causing unpredictable behavior.This mismatch arises because training data do not cover states reached after the primary policy makes errors.
- DAgger: DAgger improves on supervised learning by collecting examples while the primary and reference policies operate together, but it constantly queries the expensive reference policy.The query burden is especially undesirable when the reference policy is costly to execute.
- SafeDAgger: SafeDAgger introduces a safety policy that predicts primary-policy error and selects only a small subset of collected examples for reference-policy labeling.This subset selection reduces the number of reference-policy queries.
- Evaluation: In TORCS experiments, SafeDAgger required fewer reference-policy queries and achieved better average laps without crashes and lower damage than DAgger.The authors conjecture that automated curriculum learning from safety-policy-based subset selection explains the faster convergence.
2 Imitation Learning for Autonomous Driving
The paper formalizes imitation learning for driving around partially observed states, policies, implicit rewards, and policy-query costs. It explains how supervised learning and DAgger address distribution mismatch, while highlighting DAgger’s human-query burden.
- State Transition and Reward: An environment consists of states and available actions, with actions causing transitions through a deterministic or stochastic transition function.The transition function maps a state-action pair to a subsequent state.
- State Transition and Reward: Driving is evaluated with an implicit binary reward: 1 for no crash and 0 otherwise.The reward is observed through success or failure rather than supplied as a continuous signal.
- Policies: A policy maps a state observation φ(s) to an available action, while the underlying state may contain more information than the policy observes.For autonomous driving, φ(s) can be a front-facing camera image, whereas s summarizes road and traffic information.
- Policies: The primary policy learns to drive from pixel-level camera images, while the reference policy is a good policy to imitate and is implemented here as a rule-based simulator controller.The reference controller has access to the true underlying simulator state.
- Policy Cost: Reference-policy queries are treated as costly because human policies can be expensive to query, whereas primary-policy queries are typically cheaper.The paper evaluates imitation-learning algorithms by the number of queries made to the reference policy.
- Supervised Learning: Supervised learning collects observations from reference-policy trajectories and minimizes the difference between primary- and reference-policy actions.The resulting dataset contains the reference policy’s visited observations.
- Supervised Learning: Supervised learning can fail when primary-policy imperfections lead to states outside the reference policy’s reachable set, where training data provide no guidance.The resulting behavior may become unpredictable because the training set is almost confined to reference-policy states.
- DAgger: DAgger augments the training set with states reached by intermediate primary policies, then repeatedly fine-tunes the primary policy on the accumulated data.At each iteration, examples are generated from a mixture of reference and primary policies.
3 SafeDAgger: Query-Efficient Imitation Learning with a Safety Policy
SafeDAgger extends DAgger with a safety policy that predicts when a primary policy may deviate from the reference without querying it. It uses this prediction to trigger reference-policy intervention selectively, reducing queries while focusing training on difficult states.
- Safety Policy: SafeDAgger adds a safety policy that predicts whether the primary policy will deviate from the reference policy without querying it.The safety policy uses the partial state observation and primary policy as inputs and returns a binary safety label.
- Safety Policy: The safety strategy lets the primary policy act when its state is predicted safe and hands control to the reference policy otherwise.This automated intervention is intended to avoid dangerous situations caused by an imperfect primary policy.
- Query-Efficient DAgger: SafeDAgger selects only states marked unsafe for reference-policy labeling, unlike DAgger, which queries the reference policy for all collected states.This subset selection is the main mechanism for reducing reference-policy queries.
- Query-Efficient DAgger: Focusing on difficult states reduces the total number of training examples without losing important examples and makes the algorithm data-efficient.The selected states are described as problematic states for the current primary policy.
- Training Dynamics: Updating the safety policy after each primary-policy update creates an automated curriculum that selects examples of appropriate difficulty at each iteration.The paper connects this procedure to curriculum learning with a mix strategy.
- Training Dynamics: SafeDAgger can inherit much of DAgger’s theoretical guarantees when its threshold is gradually increased, although the paper reports that fixed-threshold training can reduce reference-policy use over iterations.The guarantee condition is τ > ϵ(π, φ(s)) for all s ∈ S.
- Extensions: The safety-policy mechanism is described as adaptable to AggreVaTe and LOLS, but those algorithms are left for future work.The paper does not evaluate these adaptations.
4 Experimental Setting
The experiments evaluate SafeDAgger in TORCS using camera-based driving policies across separate training and test tracks. They compare supervised learning, DAgger, and SafeDAgger under naive and safety-based driving strategies with multiple driving metrics.
- Simulation Environment: The evaluation uses TORCS, a lightweight open-source racing simulator that supports camera-based autonomous-driving research.The simulator can run on an off-the-shelf workstation and interface with Torch.
- Tracks: The ten tracks are split into seven training tracks and three disjoint test tracks, with testing performed only on the held-out tracks.Training and validation examples come exclusively from the training tracks.
- Traffic and Reference Policy: Each training track includes 40 reference-policy-driven cars to simulate traffic during data collection.The reference policy has access to position, heading, speed, and distances to other cars.
- Policy Networks: The primary policy is a deep convolutional network that maps front-facing camera images to steering and braking actions.It also predicts driving affordances as an auxiliary task.
- Policy Networks: Figure 1 plots the histogram of log squared steering-angle errors after supervised learning, with τ = 0.0025 marking the safety threshold.The caption reports that 77.70% of training examples are considered safe.
- Training and Driving Strategies: The study compares supervised learning, DAgger, and SafeDAgger, evaluating each with both naive and safety-based driving strategies.The evaluation includes conditions without traffic and with traffic.
- Evaluation Metrics: Performance is measured by completed laps without leaving the track, damage per lap, and mean squared steering-angle error.Runs cover three test tracks and up to three laps per track.
5 Results and Analysis
SafeDAgger improves driving performance faster than original DAgger while reducing reliance on the reference policy. Its reference-policy usage also declines during testing, including when predicting deviations one second ahead.
- Reference-policy usage: 7.11% and 10.81% of test time used the reference policy without and with traffic, respectively, at the supervised-learning baseline.
- Performance comparison: SafeDAgger makes much faster progress than original DAgger on average laps and damage per lap.After the third iteration, the SafeDAgger primary policy is perfect; the authors conjecture automated curriculum learning explains this progress.
- Reference-policy usage: The portion of test time driven by the reference policy decreases as SafeDAgger iterations continue.The authors attribute this trend to focusing learning on states deemed difficult by the safety policy.
- Performance comparison: Supervised learning alone cannot train the primary policy well despite increasing the number of training examples.
- Reference-policy usage: A one-second-ahead safety policy produces a similar SafeDAgger trend, supporting the algorithm’s practical deployment potential.
6 Conclusion
The paper concludes that SafeDAgger reduces reference-policy queries and trains primary policies more efficiently in simulated autonomous driving. It also identifies dependence on reference-policy quality as a limitation and motivates future reinforcement-learning fine-tuning.
- Conclusion: SafeDAgger uses a safety policy to switch automatically between reference and primary policies without querying the reference policy.During data collection, this selects progressively difficult examples while minimizing reference-policy queries.
- Conclusion: Experiments in simulated autonomous driving show that SafeDAgger queries the reference policy less and trains the primary policy more efficiently.
- Implications: SafeDAgger imitation learning allows the primary policy to learn without catastrophic experience.
- Limitations: The learned policy’s quality is limited by the quality of the reference policy.The paper suggests future research on reinforcement-learning fine-tuning to surpass existing reference policies.
A Dataset and Collection Procedure
The dataset uses front-facing camera observations from simulated racing tracks, with ten variables describing scene state and two control variables. Seven tracks provide training data and three disjoint tracks provide testing data.
- Collection setup: The simulator runs at 30 Hz, which the paper considers sufficient for driving below 50 mph.
- Collection setup: Camera frames are collected from a front-facing racing-car camera and resized or cropped to 160 × 72 pixels with RGB channels.
- Labels: Each image frame receives twelve variables: ten observed only by the reference policy and two control variables.All variables are training targets, but only the final two control variables are used during testing.
B Policy Networks and Training
The primary policy is a deep convolutional network, while the safety policy is a feedforward network using primary-policy convolutional features. The paper also illustrates the architectures, safety classifications, and SGD training procedure.
- Primary policy: The primary policy network has five convolutional layers followed by fully connected layers.
- Primary policy: Each convolutional layer is followed by non-overlapping max pooling, with rectified linear units providing point-wise nonlinearities.Only the shaded portion of the full primary network is used during testing.
- Safety classification: Safety-policy examples are visualized as image frames sorted by the probability of returning 1.
- Safety policy: The safety policy uses two fully connected layers and a softmax classifier over safe and unsafe categories.Its input is the activation of Conv5 in the primary policy network.
- Training: Training uses SGD with batches of 64, momentum 0.9, weight decay 0.001, and an initial learning rate of 0.001.The learning rate is divided by 5 when validation error stops improving, and training early-stops when validation error increases.
C Sample Image Frames
Figure 6 shows sample frames labeled safe or unsafe by a trained safety policy. The policy appears to base these judgments on nearby cars and whether the vehicle is entering a sharp curve.
- Figure 6 presents twenty sample frames classified by a trained safety policy as safe (0) or unsafe (1).The top ten frames are safe, while the bottom ten are unsafe.
- The safety policy considers the existence of other cars when determining the safety of a current state observation.
- The safety policy also considers whether the vehicle is entering a sharp curve.