PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors

Letian Wang, Jie Liu, Hao Shao, Wenshuo Wang, Ruobing Chen, Yu Liu, Steven L. Waslander

arXiv:2305.04412v1cs.ROcs.AIcs.LG

TL;DR

In dense, interactive traffic, reinforcement-learning agents may fail to learn reasonable driving or require substantial data. ASAP-RL learns over parameterized motion skills and incorporates expert priors through recovered skill-space demonstrations and double initialization. Across three challenging dense-traffic scenarios, it outperforms previous methods in learning efficiency and performance.

  • Problem

    In interactive multi-vehicle driving with continuous actions, RL can fail to learn reasonable performance or require extensive data and resources.

    Source: §I · p. 1

  • Method

    ASAP-RL learns over parameterized motion skills, recovers skill parameters from expert demonstrations, and pretrains both actor and critic to leverage expert priors.

    Source: §I · p. 2, §V · p. 9

  • Results

    Across three challenging dense-traffic driving scenarios, ASAP-RL significantly outperforms previous methods in learning efficiency and performance.

    Source: §V · p. 10

  • Takeaways & Limitations

    The evaluations support ASAP-RL as an approach for autonomous driving in complex dense traffic.

    Source: §V · pp. 9–10

Abstract

from arXiv · show

When autonomous vehicles are deployed on public roads, they will encounter countless and diverse driving situations. Many manually designed driving policies are difficult to scale to the real world. Fortunately, reinforcement learning has shown great success in many tasks by automatic trial and error. However, when it comes to autonomous driving in interactive dense traffic, RL agents either fail to learn reasonable performance or necessitate a large amount of data. Our insight is that when humans learn to drive, they will 1) make decisions over the high-level skill space instead of the low-level control space and 2) leverage expert prior knowledge rather than learning from scratch. Inspired by this, we propose ASAP-RL, an efficient reinforcement learning algorithm for autonomous driving that simultaneously leverages motion skills and expert priors. We first parameterized motion skills, which are diverse enough to cover various complex driving scenarios and situations. A skill parameter inverse recovery method is proposed to convert expert demonstrations from control space to skill space. A simple but effective double initialization technique is proposed to leverage expert priors while bypassing the issue of expert suboptimality and early performance degradation. We validate our proposed method on interactive dense-traffic driving tasks given simple and sparse rewards. Experimental results show that our method can lead to higher learning efficiency and better driving performance relative to previous methods that exploit skills and priors differently. Code is open-sourced to facilitate further research.

I. INTRODUCTION

In dense, interactive traffic, reinforcement learning can fail to learn reasonable driving or require substantial data. ASAP-RL combines parameterized motion skills with expert priors to improve learning and driving performance.

  • In interactive multi-vehicle settings with continuous actions, RL can fail to learn reasonable performance or require extensive data and resources.Source: §I · p. 1
  • Learning over temporally extended motion skills can produce consistent control sequences, more informative exploration, and improved reward signaling.Source: §I · pp. 1–2, Figure 1 · p. 1
  • The proposed ego-motion skill parameters aim to avoid the limited coverage of manually designed skills and the distribution and diversity problems of offline motion datasets.Source: §I · p. 2
  • ASAP-RL recovers skill parameters from expert controls and pretrains both actor and critic to use expert priors while addressing suboptimality and early performance drops.Source: §I · p. 2
  • In three challenging dense-traffic scenarios, ASAP-RL outperforms previous methods that consider skills and priors differently.Source: §I · p. 2

II. RELATED WORKS

Prior work on driving skills and expert demonstrations faces coverage, transfer, or training challenges. ASAP-RL learns directly over skill parameters and combines skill-space expert data with actor-and-critic initialization.

II. RELATED WORKS

  • The paper claims to be the first to combine skills and expert priors for autonomous-driving RL, and describes its motion-centric skills as adaptable to other moving robots.Source: §II · p. 2

A. Reinforcement Learning with Skills

B. Reinforcement Learning with Expert Priors

  • Prior methods use demonstrations for initialization, guidance, or replay, but actor-only initialization can lose expert knowledge and guidance can suppress performance with suboptimal experts.Source: B. Reinforcement Learning with Expert… · p. 3

III. APPROACH

  • The approach generates trajectories from skill parameters, recovers those parameters from control-only demonstrations, and pretrains both actor and critic in an actor-critic framework.Source: §III · pp. 3–4

A. Motion Skill Generation

ASAP-RL represents driving actions as parameterized motion skills generated from a path and speed profile. It recovers skill parameters from control-space demonstrations by fitting the skill generator to segmented demonstrated trajectories.

A. Motion Skill Generation

  • Four endpoint parameters determine each motion skill: lateral position, heading, speed, and acceleration; the generator combines a path and speed profile into a trajectory.The path and speed profile use cubic polynomials, and the integrated speed profile is projected onto the path.Source: A. Motion Skill Generation · p. 4, Figure 3 · p. 4
  • The path endpoint’s lateral position and heading encode lateral maneuvers, while terminal speed and acceleration encode temporal intentions.The path’s longitudinal endpoint is fixed to the longest distance reachable within skill duration T; the speed profile’s initial speed and acceleration come from the current vehicle state.Source: A. Motion Skill Generation · p. 4
  • Each skill is a sequence of vehicle states, and conditioning the next skill on the last executed skill’s final state ensures smooth transitions.Planning parameters are restricted to reasonable ranges to enforce acceleration and curvature constraints.Source: A. Motion Skill Generation · p. 4

B. Skill Parameter Recovery

  • Expert trajectories are split into segments of length T, and inverse parameter recovery fits the skill generator to each demonstrated motion skill.Sequential quadratic programming is run with multiple parameter initializations to mitigate local optima and improve recovery accuracy.Source: B. Skill Parameter Recovery · pp. 4–5

C. Expert Prior Learning - Actor and Critic Pretraining

The method converts expert demonstrations into skill-space priors, then pretrains the actor and critic. The actor learns state-conditioned expert skill choices, while its rollouts provide reward-labeled data for critic pretraining.

Expert Prior Learning

  • ASAP-RL uses SAC with an actor π(θ|s) and critic Q(s, θ), initialized with their pretrained weights for subsequent reinforcement learning.The skill-space demonstrations are recovered from raw control-space demonstrations before prior learning.Source: C. Expert Prior Learning - Actor and… · p. 5

Actor Pretraining

  • The actor is pretrained on recovered expert skill parameters to model a state-conditioned Gaussian distribution over skills.This actor provides prior knowledge about which skills are more promising to explore in a given state.Source: C. Expert Prior Learning - Actor and… · p. 5

Critic Pretraining

  • Because demonstrations lack rewards, the pretrained actor is rolled out in the environment to collect skill and reward information for critic pretraining.The critic is pretrained using typical SAC policy evaluation.Source: C. Expert Prior Learning - Actor and… · p. 5

D. RL over parameterized skill with priors

ASAP-RL learns a policy over motion-skill parameters rather than single-step controls, executing each selected skill for a fixed duration. Its maximum-entropy objective combines discounted reward with exploration, and the policy and critic use pretrained weights.

D. RL over parameterized skill with priors

  • The modified maximum-entropy RL objective encourages reward maximization and exploration in skill space.The accumulated discounted reward is calculated after one motion skill of length T is executed, alongside an entropy term weighted by temperature α.Source: D. RL over parameterized skill with… · pp. 5–6
  • The policy outputs skill parameters π(θ|s), which generate a motion skill executed for T steps before the next skill is selected.This fixed-duration execution follows a semi-MDP process with temporal abstraction and accelerated reward signaling.Source: D. RL over parameterized skill with… · p. 6
  • The actor and critic are initialized with pretrained weights, while extending the fixed-length policy to dynamic skill lengths is left for future work.The authors report that a fixed-length policy can achieve strong performance.Source: D. RL over parameterized skill with… · p. 6

IV. EXPERIMENTS

The experiments examine ASAP-RL’s learning efficiency and driving performance, the effect of skill duration, and how its double initialization compares with other ways of incorporating expert priors.

Performance

  • The performance evaluation asks whether ASAP-RL learns driving strategies more efficiently and performs better than methods using skills and priors differently.The comparison is associated with Fig. 5.Source: §IV · p. 6

Influence of the length of skill

  • The skill-length experiment examines how the motion-skill duration T influences ASAP-RL’s performance.The request identifies Fig. 6 for this evaluation.Source: §IV · p. 6

Influence of expert prior

  • The expert-prior experiment tests whether prior knowledge accelerates reinforcement learning and compares double initialization with other prior-integration methods.The request identifies Fig. 7 for this evaluation.Source: §IV · p. 6

1) Environment:

ASAP-RL is evaluated in randomized dense-traffic scenarios using BEV observations and sparse rewards for progress, destination arrival, collisions, and overtaking.

1) Environment:

  • In MetaDrive [17], ASAP-RL is evaluated in highway, roundabout, and intersection traffic with random lane counts, roadblock orders, vehicle locations, and target speeds; the ego must reach its destination within the required time without collisions or running off the roadway.The RL agent receives a 5-channel BEV image sized 200×200×5, including spatiotemporal information about the ego and surrounding agents.Source: 1) Environment · p. 6
  • BEV input combines the current scene, road and navigation lanes, ego-vehicle waypoint history, and surrounding-object snapshots at t, t −1, and t −2.Source: Figure 4 · p. 6

2) Reward Definition:

  • The reward uses sparse terms: +1 per 10 m completed, +1 for reaching the destination, −5 for a collision, and +0.1 per vehicle overtaken.Source: 2) Reward Definition · p. 6
  • Each skill spans T simulation steps of 0.1 seconds, and its reward is the sum of those step rewards; the method also performed well without destination reward in highway and roundabout settings.Source: 2) Reward Definition · p. 6

3) Expert Demonstration Collection:

The study uses demonstrations from a trained RL expert and compares ASAP-RL with control-space and skill-space baselines, including methods that use expert priors.

3) Expert Demonstration Collection:

  • The trained RL expert was selected for data collection because its demonstrations were more diverse and agents trained from them performed more strongly than those trained from rule-based demonstrations.The rule-based planner performed better after time-consuming tuning, but needed perturbations to collect diverse, danger-responsive actions.Source: 3) Expert Demonstration Collection · p. 6

4) Baselines:

  • PPO and SAC train from scratch over single-step control actions, while Constant SAC repeats the same action for the skill horizon T.Source: 4) Baselines · pp. 6–7
  • SPiRL learns latent skills from demonstrations and uses an expert-prior KL term, whereas TaEcRL uses task-agnostic ego-centric skills without expert priors.Source: 4) Baselines · p. 7

5) Evaluation metric:

Evaluation combines five driving metrics with ablations of skill length and expert-prior initialization.

5) Evaluation metric:

  • The evaluation metrics are episode reward, destination success rate, road completion ratio, collision rate, and cars passed per episode.Source: 5) Evaluation metric · p. 7
  • In the roundabout ablation, increasing skill length from T = 1 to T = 10 improves performance, while T = 20 delays reactions to accidents; T = 10 is a reported trade-off.Source: Figure 6 · p. 8
  • In the roundabout prior ablation, ASAP-RL starts well and reaches the highest final performance without the early drop or expert-suboptimality suppression seen in the compared initialization variants.Source: Figure 7 · p. 8

6) A three-stage strategy for performance evaluation:

ASAP-RL outperforms TaEcRL on all metrics in the first 10k iterations and passes substantially more cars, while the evaluation strategy progressively examines more detailed driving measures.

6) A three-stage strategy for performance evaluation:

  • The evaluation first compares reward, then success rate and road completion when rewards are similar, and finally collision rate and passed cars if earlier metrics remain close.Passed cars and collision rate can trade off, so the latter-stage measures add context to earlier comparisons.Source: 6) A three-stage strategy for… · p. 8

B. Performance Comparison

  • PPO, SAC, and SPiRL have significantly lower reward than the other three methods; SPiRL agents mostly drive straight, barely maneuver, and have high collision rates.Source: B. Performance Comparison · p. 8
  • Constant SAC earns slightly lower reward than TaEcRL and ASAP-RL, but its conservative, slow driving yields close-to-zero success and passed cars despite decent road completion and low collision rates.Source: B. Performance Comparison · pp. 8–9
  • In the first 10k iterations, ASAP-RL performs better than TaEcRL on all metrics and passes many more cars, with better or similar success, road completion, and collision results.Source: B. Performance Comparison · p. 9

1) Influence of the length of skill:

Across three challenging dense-traffic scenarios, ASAP-RL outperforms previous methods in learning efficiency and driving performance. The ablations examine skill horizon and expert-prior initialization, while visualizations show scenario-specific maneuvers.

1) Influence of the length of skill:

  • T = 10 provides a good skill-length trade-off: longer decisions improve performance, but T = 20 delays replanning and limits skill-set richness.T = 1 performs poorly as a single-step method.Source: 1) Influence of the length of skill · p. 9

2) Influence of expert prior:

  • Double initialization achieves the highest final performance without the actor–critic mismatch drop or the performance suppression associated with expert suboptimality.Behavior Cloning does not improve further, while Initiate Actor drops during its first 5K iterations.Source: 2) Influence of expert prior · p. 9

D. Visualizations

  • Visualizations show lane changes, overtaking, and cautious slowing across highway, intersection, and roundabout traffic scenarios.The roundabout examples include cautious overtaking and slow driving when vehicles block the way.Source: D. Visualizations · p. 9, Figure 8 · p. 10

V. CONCLUSION

  • ASAP-RL significantly outperforms previous methods in learning efficiency and performance across three challenging dense-traffic driving scenarios.Source: §V · p. 10
Loading 2305.04412v1…