PaperlayerView exact revision on arXiv

Source-linked AI summary

CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Driving

Dongkun Zhang, Jiaming Liang, Ke Guo, Sha Lu, Qi Wang, Rong Xiong, Zhenwei Miao, Yue Wang

arXiv:2502.19908v3cs.ROcs.CVcs.LG

TL;DR

Large-scale driving RL planners face training inefficiency and performance challenges, while imitation learning is vulnerable to distribution shift and causal confusion. CarPlanner uses consistent-mode autoregressive generation with model-based training, expert-guided rewards, and invariant-view inputs. The authors report that it surpasses RL-, IL-, and rule-based state-of-the-art planners on nuPlan.

  • Problem

    RL planners struggle with training inefficiency and performance on large-scale driving, while IL planners are vulnerable to distribution shift and causal confusion.

    Source: §1 · pp. 1–2, §2.1 · p. 3

  • Method

    CarPlanner uses consistent-mode autoregressive RL planning with a model-based formulation, expert-guided reward, and invariant-view inputs.

    Source: §1 · p. 2

  • Results

    CarPlanner surpasses RL-, IL-, and rule-based state-of-the-art approaches on nuPlan.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The results demonstrate the potential of RL for learning-based planning on the nuPlan benchmark.

    Source: §5 · p. 8

    The expert-aided reward relies on demonstrations and may constrain RL’s full potential or hinder discovery of solutions beyond human expertise.

    Source: §5 · p. 8

Abstract

from arXiv · show

Trajectory planning is vital for autonomous driving, ensuring safe and efficient navigation in complex environments. While recent learning-based methods, particularly reinforcement learning (RL), have shown promise in specific scenarios, RL planners struggle with training inefficiencies and managing large-scale, real-world driving scenarios. In this paper, we introduce \textbf{CarPlanner}, a \textbf{C}onsistent \textbf{a}uto-\textbf{r}egressive \textbf{Planner} that uses RL to generate multi-modal trajectories. The auto-regressive structure enables efficient large-scale RL training, while the incorporation of consistency ensures stable policy learning by maintaining coherent temporal consistency across time steps. Moreover, CarPlanner employs a generation-selection framework with an expert-guided reward function and an invariant-view module, simplifying RL training and enhancing policy performance. Extensive analysis demonstrates that our proposed RL framework effectively addresses the challenges of training efficiency and performance enhancement, positioning CarPlanner as a promising solution for trajectory planning in autonomous driving. To the best of our knowledge, we are the first to demonstrate that the RL-based planner can surpass both IL- and rule-based state-of-the-arts (SOTAs) on the challenging large-scale real-world dataset nuPlan. Our proposed CarPlanner surpasses RL-, IL-, and rule-based SOTA approaches within this demanding dataset.

1. Introduction

CarPlanner targets RL training inefficiency and weak performance in large-scale driving by conditioning autoregressive trajectory generation on consistent modes. It combines model-based training with an expert-guided reward and an invariant-view module, and reports results surpassing RL-, IL-, and rule-based nuPlan state of the arts.

1. Introduction

  • RL trajectory planning faces training inefficiency and poor performance in large-scale driving, while IL is vulnerable to distribution shift and causal confusion.The paper proposes a model-based approach using neural transition models suited to GPUs, with time cost described as comparable to IL-based methods.Source: §1 · pp. 1–2
  • CarPlanner conditions autoregressive generation on consistent modes to address long-term inconsistency and unnecessarily broad exploration from random action sampling.The longitudinal-lateral mode representation provides stable guidance across time steps.Source: §1 · p. 2
  • A universal reward combines expert-trajectory displacement error with task-oriented driving terms, avoiding scenario-specific reward designs.The task-oriented terms include collision avoidance and adherence to the drivable area.Source: §1 · p. 2
  • The Invariant-View Module transforms agent, map, and route information into ego coordinates and clips distant information to provide time-agnostic policy input.The module is intended to ease feature learning and support generalization.Source: §1 · p. 2
  • CarPlanner surpasses RL-, IL-, and rule-based state-of-the-arts on the nuPlan benchmark.The authors describe this as the first demonstration that an RL-based planner outperforms IL and rule-based approaches on this large-scale dataset.Source: §1 · p. 2

2. Related Work

Prior work highlights limitations in both imitation learning and reinforcement learning for autonomous driving. The paper motivates model-based RL for large-scale datasets and identifies a trade-off in trajectory-based exploration.

2.1. Imitation-based Planning

  • IL leverages scalable human-driving data, but improving open-loop performance does not resolve the challenge of closed-loop driving-oriented evaluation.The cited evaluation concerns safety, traffic-rule adherence, comfort, and progress.Source: §2.1 · pp. 2–3
  • IL planners can face a substantial gap between training and closed-loop testing, with vulnerability to distribution shift and causal confusion.Distribution shift can produce suboptimal decisions in unrepresented scenarios; causal confusion can lead networks to learn shortcuts from expert imitation loss.Source: §2.1 · p. 3

2.2. RL in Autonomous Driving

  • Trajectory actions can improve RL training efficiency over high-frequency control actions, but horizon length trades reactivity and data volume against control-space-like challenges.The passage attributes this trade-off to ASAP-RL [40].Source: §2.2 · p. 3
  • The paper proposes a model-based formulation with a transition model to facilitate RL training on large-scale datasets and preview the world during multi-step decisions.The proposed formulation is presented as a way to address the trajectory-horizon trade-off.Source: §2.2 · p. 3

3. Method

The method formulates planning as sequential decision-making and decomposes autoregressive trajectory generation into policy and transition components. CarPlanner conditions the policy on a mode selected from the initial state, while a non-reactive transition model predicts traffic-agent futures in one shot.

3.1. Preliminaries

  • The MDP formulation represents sequential decision-making with state, action, transition, reward, initial-state distribution, discount factor, and time horizon.Its objective is to maximize expected discounted return.Source: §3.1 · p. 3
  • The state representation includes map information and histories of the ego vehicle and other traffic agents, encoded as polylines and polygons.The ego vehicle has index 0, while traffic agents have indices 1 through N.Source: §3.1 · p. 3

3.2. Problem Formulation

  • Defining each action as the ego vehicle’s next pose turns autoregressive planning into a state-sequence formulation that can be decomposed into policy and transition models.The decoded poses are collected as the ego-planned trajectory.Source: §3.2 · p. 4
  • Vanilla autoregressive sampling can produce inconsistent behavior across time because the policy samples randomly from the action distribution.The proposed formulation introduces mode information that remains unchanged across time steps.Source: §3.2 · p. 4
  • CarPlanner selects a mode from the initial state and conditions the trajectory policy on that mode, yielding a generation-selection framework for multimodal trajectories.The mode does not affect the transition model when the focus is ego trajectory planning.Source: §3.2 · p. 4
  • The non-reactive transition model predicts all future traffic-agent poses from the initial state in one shot because per-step transition-model execution is time-consuming and showed no performance improvement.This replaces repeated per-timestep traffic-agent prediction in the described formulation.Source: §3.2 · p. 4

3.3. Planner Architecture

CarPlanner uses a generation-selection architecture to score driving modes, generate corresponding trajectories, and select a planner output. Its components include a non-reactive transition model, mode selector, trajectory generator, and rule-augmented selector.

3.3. Planner Architecture

  • The planner scores all possible modes, generates a corresponding trajectory for each in parallel previewed worlds, and uses a trajectory predictor to model other agents.The rule-augmented selector chooses the highest-scoring trajectory.Source: §3.3 · p. 4

3.3.1. Non-reactive Transition Model

  • The non-reactive transition model encodes map and agent information from the initial state and decodes agent features into future trajectories.PointNet encoders produce map and agent features, which a self-attention Transformer fuses.Source: §3.3.1 · pp. 4–5, §3.3 · p. 4

3.3.2. Mode Selector

  • The mode selector combines longitudinal speed modes with map-derived lateral routes and outputs a probability for each mode.The combined mode representation is fused with map and agent features to score the modes.Source: §3.3.2 · p. 5

3.3.3. Trajectory Generator

  • The trajectory generator recurrently decodes the next ego pose from the current state and consistent mode information.Its policy and value heads produce action-distribution parameters and value estimates; inference uses the distribution mean.Source: §3.3.3 · p. 5

3.3.4. Rule-augmented Selector

  • At inference, the rule-augmented selector scores candidate trajectories using driving-oriented metrics and mode scores, then outputs the highest-scoring trajectory.The metrics include safety, progress, and comfort.Source: §3.3.4 · p. 5

3.4. Training

  • Training freezes the non-reactive transition model, assigns positive modes from the ground-truth endpoint, and rewards displacement accuracy, collision avoidance, and drivable-area compliance.The generator uses PPO loss; the selector uses cross-entropy and a trajectory-regression side task.Source: §3.4 · pp. 5–6

4. Experiments

On nuPlan benchmarks, CarPlanner performs best across non-reactive metrics, while its reactive-setting performance trails PDM-Closed. Ablations, IL comparisons, and a pedestrian scenario further characterize the method’s behavior and design choices.

4.1. Experimental Setup

  • Evaluation uses nuPlan’s 1,500-hour, four-city logs, with Test14-Random (261 scenarios) and Reduced-Val14 (318); CLS covers safety, drivable-area compliance, progress, and comfort.The benchmarks support comparisons with other methods and analysis of CarPlanner design choices.Source: §4.1 · p. 6

4.2. Comparison with SOTAs

  • CarPlanner scores highest across non-reactive metrics, improving by 4.02 and 2.15 versus PDM-Closed and PLUTO, respectively.It also improves progress over PDM-Closed while achieving comparable S-CR.Source: Table 2 · p. 6, §4.2 · pp. 6–7

4.3. Ablation Studies

  • S-CR rose 97.49 to 99.22 and S-Area 96.91 to 99.22 with quality reward; coordinate transformation raised overall closed-loop score 90.78 to 94.07 and S-PR 91.37 to 95.06.Quality reward alone encourages static trajectories; the authors report that coordinate transformation and KNN in IVM improve closed-loop metrics and generator loss.Source: Table 3 · p. 7, §4.3 · p. 7

4.2. Comparison with SOTAs

  • In reactive settings, CarPlanner falls slightly short of PDM-Closed after training only with non-reactive traffic agents.The authors attribute lower robustness to disturbances from reactive agents at test time.Source: §4.2 · p. 7

4.4. Extention to IL

  • Mode dropout and selector side tasks benefit both IL and RL, whereas ego-history dropout and backbone sharing are less suitable for RL.Ego-history dropout degrades RL advantage estimation; backbone sharing increases generator and selector losses, which the authors link to interfering gradients.Source: §4.4 · p. 7

4.5. Qualitative Results

  • In the pedestrian right-turn scene, RL starts before a returning pedestrian and avoids the hazard, unlike PDM-Closed’s collision and the IL variant’s close pass.The figure marks the ego vehicle in green, traffic agents in sky blue, and the planned ego trajectory in blue.Source: §4.5 · pp. 7–8, Figure 3 · p. 8

5. Conclusion

CarPlanner is a consistent auto-regressive planner for large-scale RL training, and the paper reports that it outperforms existing RL-, IL-, and rule-based SOTAs. The authors also identify reliance on expert-aided rewards as a limitation that may constrain RL’s potential.

5. Conclusion

  • CarPlanner outperforms existing RL-, IL-, and rule-based SOTAs, according to the authors.The conclusion describes it as a consistent auto-regressive planner for large-scale RL training.Source: §5 · p. 8
  • The authors’ analysis highlights RL’s potential to take a further step toward learning-based planning.Source: §5 · p. 8
  • Expert-aided reward design may constrain RL’s full potential because it relies on expert demonstrations and may hinder solutions surpassing human expertise.Source: §5 · p. 8

A. Training Procedure

CarPlanner trains a transition model, then trains a mode selector and trajectory generator; the generator can be trained with RL or IL, which the paper studies separately. The selector combines mode classification with a trajectory-regression side task, while the RL generator uses PPO losses.

A. Training Procedure

  • The training procedure supports either RL or IL for the trajectory generator, but this work studies them separately rather than combining them.Source: A. Training Procedure · p. 11
  • The non-reactive transition model is trained to simulate agent trajectories from the initial state.Source: A. Training Procedure · p. 11
  • The mode selector uses cross-entropy for the positive mode and a side task that regresses the ego ground-truth trajectory.Source: A. Training Procedure · p. 11
  • For RL, the generator uses PPO policy, value, and entropy losses; for IL, it minimizes L1 error between planned and ground-truth ego trajectories.Source: A. Training Procedure · pp. 11–12
  • The training algorithm updates the transition model first, then trains the selector and generator using simulated agent trajectories and assigned modes.Source: A. Training Procedure · p. 11

B. Implementation Details

Implementation uses an eight-second trajectory horizon with one-second intervals and interpolates trajectories to 0.1-second intervals at test time. The replanning frequency varies by benchmark.

B. Implementation Details

C. Ablation Study on RL Training

The ablations report better closed-loop performance for the consistent than vanilla auto-regressive framework, and substantially higher sampling efficiency than a model-free baseline. Results also compare reactive and non-reactive transition models and examine training and testing horizons.

C. Ablation Study on RL Training

  • CarPlanner’s sampling efficiency exceeds ScenarioNet’s by two orders of magnitude in the comparison of model-based and model-free training.Source: C. Ablation Study on RL Training · p. 12, Table 6 · p. 12
  • The consistent framework outperforms the vanilla framework in closed-loop performance and generates more consistent trajectories than vanilla and IL-based methods.Displacement error also serves as an effective guide function for policy training, further enhancing closed-loop performance.Source: C. Ablation Study on RL Training · p. 13
  • Except for S-Area, the non-reactive transition model outperforms the reactive model in the current implementation.The reactive model assumes agents negotiate with ego, whereas the non-reactive model assumes they do not respond and effectively have higher priority.Source: C. Ablation Study on RL Training · p. 13, Table 9 · p. 13
  • Longer training and testing horizons generally improve performance; a 1-second training horizon yields poor performance at every tested horizon.The reported exploration is limited to horizons up to 8 seconds because the prepared data lacks map information or ground-truth trajectories beyond that.Source: C. Ablation Study on RL Training · p. 13, Figure 4 · p. 13

D. Comparison with Differentiable Loss

This section compares differentiable losses with RL rewards for optimizing collision and drivable-area metrics. RL rewards achieve the strongest overall trajectory-set quality, while the two approaches differ in their open- and closed-loop effects.

D. Comparison with Differentiable Loss

  • IL converts Col and Area into differentiable losses, while RL treats them as rewards contributing to the quality reward.The implementations are identical except that gradient flow is enabled for differentiable losses.Source: D. Comparison with Differentiable Loss · p. 14
  • RL with Col and Area quality rewards yielded the highest overall trajectory-set quality, with smaller open-loop Mean and Max metrics than differentiable losses.The open-loop metrics summarize candidate multi-modal trajectories.Source: D. Comparison with Differentiable Loss · p. 14
  • Col loss benefited open-loop Col and improved closed-loop S-CR, whereas Area loss improved open-loop Area but deteriorated closed-loop performance.Source: D. Comparison with Differentiable Loss · p. 14
  • RL reward-to-go can let rewards at time t influence earlier trajectories, unlike the current timewise-aligned differentiable-loss optimization.The authors aim to combine differentiable loss’s low-variance gradients with RL’s long-term foresight using model-based RL.Source: D. Comparison with Differentiable Loss · pp. 14–15

E. Effect of Mode Representation

With random sampling disabled, the experiments compare single-modal vanilla trajectories against consistent mode representations. The results favor consistency, especially the Lon-Lat representation.

E. Effect of Mode Representation

Related papers

Loading 2502.19908v3…