PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Multi-Modal Planning with Reward-Guided Preference Optimization for Autonomous Driving

Chenglin Chen, Lujia Wang, Xinhu Zheng, Jun Ma, Haoang Li

arXiv:2609.38862v1cs.ROcs.AI

TL;DR

End-to-end driving planners face challenges in computational efficiency, safety alignment, and using preference data beyond strict trajectory pairs. EMPlan combines sparse anchors and offset refinement with reward-guided fine-tuning using rule-based rewards and unpaired preference supervision. On non-reactive NAVSIM, it achieves state-of-the-art performance with competitive inference efficiency.

  • Problem

    End-to-end imitation-learning policies can face causal confusion and distribution shifts, while rule-based scoring and strict pairwise preference methods present efficiency, objective-alignment, or data-utilization challenges.

    Source: §I · p. 1

  • Method

    EMPlan combines sparse-anchor trajectory generation and offset refinement with a two-stage training strategy that fine-tunes a pretrained policy using reward signals and unpaired preference supervision.

    Source: §I · pp. 1–2

  • Results

    On non-reactive NAVSIM, EMPlan achieves state-of-the-art performance with competitive inference efficiency.

    Source: §V · p. 8

  • Takeaways & Limitations

    The hybrid architecture and reward-guided fine-tuning provide real-time multi-modal trajectory generation while improving planning performance beyond pure imitation learning.

    Source: §V · p. 8

    Reward-guided training relies on simulator-based evaluation, limiting direct scaling to benchmarks without compatible simulation environments.

    Source: §V · p. 8

Abstract

from arXiv · show

Safe and efficient trajectory planning is essential in autonomous driving. However, existing end-to-end approaches often fall short in both computational efficiency and safety guarantees. Methods based on imitation learning suffer from causal confusion, while rule-based scoring approaches often incur heavy computational overhead and suffer from objective misalignment. Additionally, preference-based methods rely on strict pairwise annotations, limiting data utilization. To overcome these limitations, we propose EMPlan, an efficient multi-modal trajectory planning method powered by reward-guided fine-tuning. We design a hybrid architecture that combines sparse anchors with an offset refinement module for efficient multi-modal trajectory prediction. Sparse anchors provide coarse trajectory candidates with low latency, which are subsequently refined by the offset module for higher prediction accuracy. To enhance safety without incurring additional inference costs, we adopt a two-stage training paradigm consisting of pretraining and reward-guided fine-tuning. During fine-tuning, we leverage rule-based reward signals and unpaired preference supervision to refine the pretrained policy toward safer trajectory selection. We evaluate EMPlan on the non-reactive NAVSIM benchmark, where it strikes a favorable balance between planning accuracy and efficiency, demonstrating superior performance under real-time constraints.

I. INTRODUCTION

Existing end-to-end driving methods face challenges in safety, efficiency, and data utilization. EMPlan combines sparse-anchor trajectory generation and offset refinement with reward-guided fine-tuning that uses unpaired preference supervision.

I. INTRODUCTION

  • Imitation learning can fail in long-tail situations amid causal confusion and distribution shifts, while its mimicry objective may neglect safety, comfort, and traffic-rule compliance.Rule-based scoring approaches can require dense anchors and incur high latency; strict pairwise preference labels use only limited trajectory pairs.Source: §I · p. 1
  • Sparse anchors generate coarse candidates that a lightweight offset module refines, preserving action-space coverage while reducing computational overhead.Reward-guided fine-tuning aligns trajectory selection with safety preferences without additional inference costs.Source: §I · p. 1, Figure 1 · p. 1
  • Unpaired desirable and undesirable trajectories provide preference supervision without strict pair construction, allowing available trajectory data to be used more fully.The strategy combines rule-based reward signals with relative preference signals during fine-tuning.Source: §I · p. 2
  • EMPlan combines sparse anchors with offset refinement and reward-guided fine-tuning to improve trajectory-planning accuracy and efficiency while supporting real-time inference.The offset module refines coarse candidates; the fine-tuning strategy uses reward and unpaired preference supervision.Source: §I · p. 2

II. RELATED WORKS

Prior end-to-end autonomous-driving planners span multi-modal architectures, scoring methods, and diffusion models, while policy fine-tuning includes reinforcement-learning and preference-based approaches. EMPlan uses a lightweight hybrid planner and unpaired preference data.

A. End-to-End Autonomous Driving

  • Score-based planners such as VADv2 [4] and HydraMDP [11] assign supervision to anchors individually, while diffusion approaches, can incur high memory overhead and inference latency.EMPlan instead uses a lightweight hybrid architecture with reward-guided fine-tuning.Source: A. End-to-End Autonomous Driving · p. 2

B. Policy Fine-Tuning in Autonomous Driving

  • Policy fine-tuning includes reinforcement learning and preference optimization, but RL methods can depend on reward engineering and unstable training, while DriveDPO [17] relies on strict pairwise labels.EMPlan uses unpaired preference data so candidate trajectories can contribute independently to optimization.Source: B. Policy Fine-Tuning in Autonomous… · p. 2

III. METHOD

EMPlan frames multi-modal planning as selecting among trajectory anchors and targets efficient candidate generation with relative reward signals for safer selection. Its architecture refines sparse-anchor trajectories and selects the candidate with the highest aggregated reward.

III. METHOD

  • The policy predicts a T-step ego-vehicle trajectory and selects among a predefined vocabulary of trajectory anchors to handle multi-modal driving behavior.Each anchor is a T-step waypoint sequence, and the highest-scoring anchor is selected.Source: §III · p. 2
  • Standard supervised training scores anchors independently, ignoring relative candidate quality and requiring dense anchors that increase computational cost.EMPlan's stated goals are to avoid exhaustive anchor enumeration and use relative reward signals to refine safer trajectory selection.Source: §III · pp. 2–3
  • The network extracts BEV representations from multi-view images, refines sparse-anchor trajectories with an offset head, and outputs the trajectory with the highest aggregated reward.Training first uses imitation and rule-based supervision, then reward-guided fine-tuning with reward and preference losses.Source: Figure 2 · p. 3

A. Hybrid Architecture of Our Policy Model

EMPlan’s policy combines scene perception with a trajectory decoder that represents maneuvers using sparse anchors and refines them with scene-conditioned offsets. A scorer evaluates the refined candidates for final reward-based selection.

A. Hybrid Architecture of Our Policy Model

  • The trajectory decoder combines coarse discrete maneuver modes with continuous offset refinement, reducing anchor quantization error without the larger vocabulary and decoding computation of dense-anchor methods.Unlike diffusion planners, refinement occurs in a single feed-forward pass without iterative denoising.Source: A. Hybrid Architecture of Our Policy… · p. 3
  • Multi-view RGB features form BEV representations at two scales; one supports auxiliary semantic segmentation, while the other becomes scene tokens for decoding.The scene tokens provide scene-level context to the trajectory decoder.Source: A. Hybrid Architecture of Our Policy… · p. 3
  • Sparse trajectory anchors are clustered from NAVSIM training data and transformed into high-dimensional embeddings for trajectory decoding.Ego state and high-level driving commands are fused with anchor embeddings to form planning tokens, which attend to scene tokens.Source: A. Hybrid Architecture of Our Policy… · p. 3
  • An offset head predicts per-anchor residuals to refine coarse trajectories that may not align with the predefined anchors.A separate scorer evaluates each refined trajectory, and inference selects the candidate with the highest predicted reward.Source: A. Hybrid Architecture of Our Policy… · pp. 3–4

B. Score-Based Policy Learning

First-stage policy learning combines trajectory imitation, score supervision, and auxiliary BEV segmentation. The losses address human-like prediction, rule-based safety and physical constraints, and imbalanced map categories.

B. Score-Based Policy Learning

  • The first-stage objective combines winner-takes-all trajectory regression, imitation-score and rule-based score supervision, and auxiliary BEV focal-loss supervision.The respective loss terms are balanced in the total loss.Source: B. Score-Based Policy Learning · p. 4
  • Auxiliary BEV segmentation uses focal loss to emphasize hard and minority categories amid scenes dominated by background and road.This differs from the cited prior approaches that use cross-entropy loss.Source: B. Score-Based Policy Learning · p. 4
  • Rule-based supervision covers collision, drivable-area compliance, time-to-collision, comfort, and ego progress, using sigmoid focal loss for skewed scores.The conditional ego-progress mask applies that loss only when collision and drivable-area scores exceed 0.5.Source: B. Score-Based Policy Learning · p. 4

C. Reward-Guided Fine-Tuning

The section addresses the limits of absolute score regression by combining reward supervision over candidate trajectories with unpaired preference optimization. Its fine-tuning process aggregates rewards, partitions trajectories into preference sets, and uses relative rewards to guide policy updates.

C. Reward-Guided Fine-Tuning

  • Stage II combines trajectory-level reward supervision across all candidates with preference loss from unpaired preference data.The fine-tuning objective adds reward supervision as an auxiliary signal alongside preference optimization.Source: Figure 3 · p. 4, C. Reward-Guided Fine-Tuning · p. 4
  • The reward aggregates imitation and rule-based scores in log space, sharply penalizing trajectories with near-zero collision or drivable-area scores.This aggregation is intended to reflect the conjunctive nature of safety criteria, so high progress, time-to-collision, or comfort cannot compensate for near-zero collision or drivable-area scores.Source: C. Reward-Guided Fine-Tuning · p. 4
  • KTO uses unpaired samples to increase the relative likelihood of desirable trajectories and suppress that of undesirable ones against the reference distribution.Its objective uses sample weights and a sigmoid function whose sensitivity is controlled by β.Source: C. Reward-Guided Fine-Tuning · pp. 4–5
  • Preference construction evaluates generated candidates offline, then partitions the top 25% by expert L2 similarity into desirable, undesirable, and reference sets using reward thresholds −0.6 and −1.0.D+ represents strictly safe behaviors, D− safety-violating behaviors, and Dref safe but suboptimal behaviors.Source: C. Reward-Guided Fine-Tuning · p. 5, Figure 4 · p. 5
  • Relative reward measures a trajectory’s preference under the current policy against the frozen reference policy, whose trajectory probabilities are normalized over anchors.The current policy probability is obtained by softmax-normalizing aggregated rewards over the trajectory-anchor set.Source: C. Reward-Guided Fine-Tuning · p. 5
  • The reference point is estimated from average relative reward over Dref because infeasible or unsafe samples can bias an estimate over all predicted trajectories.The section notes that anchor-based prediction uses a discrete candidate set, unlike the continuous output space discussed for the original KTO setting.Source: C. Reward-Guided Fine-Tuning · p. 5

A. Experimental Setup

Experiments use NAVSIM’s non-reactive benchmark and evaluate driving performance with PDMS. The setup uses multi-view camera inputs, predefined trajectory anchors, and a two-stage training schedule.

A. Experimental Setup

  • NAVSIM provides 1,192 navtrain scenes for training and validation and 136 navtest scenes for testing, with 2 Hz sensor inputs.It also provides semantic BEV maps and 3D object bounding boxes.Source: A. Experimental Setup · p. 5
  • PDMS is the primary non-reactive evaluation metric, combining NC, DAC, TTC, C, and EP as PDMS = NC × DAC × (5EP + 5TTC + 2C)/12.Table I compares representative methods on navtest and distinguishes camera (C) from lidar (L).Source: A. Experimental Setup · p. 5, Table I · p. 6
  • The model uses concatenated 1024×256 multi-view RGB input, a V2-99 backbone, and 256 predefined trajectory anchors.Inference predicts 8-waypoint trajectories over a 4-second horizon.Source: A. Experimental Setup · pp. 5–6
  • Figure 5 qualitatively compares predicted trajectories with TransFuser [2], Hydra-MDP [11], and WoTE [12] across four challenging scenarios.EMPlan is reported to execute all depicted maneuvers successfully.Source: Figure 5 · p. 6
  • Training uses 35 pretraining epochs followed by 10 iterative fine-tuning epochs, with batch size 16 per GPU in both stages.Experiments use two NVIDIA RTX A6000 GPUs.Source: A. Experimental Setup · p. 6

B. Comparisons with State-of-the-art Methods

On NAVSIM navtest, EMPlan achieves the best reported PDMS among the compared methods. It also combines low inference latency with qualitative successes in challenging maneuvers.

B. Comparisons with State-of-the-art Methods

  • 89.6 PDMS is the best reported score on NAVSIM navtest, ahead of Transfuser (84.0), Hydra-MDP (86.5), WoTE (88.3), and DiffusionDrive (88.1).DiffusionDrive achieves the highest EP but underperforms EMPlan on NC and DAC.Source: B. Comparisons with State-of-the-art… · pp. 6–7
  • In the construction-obstruction scenario, EMPlan completes a right turn, while Transfuser and Hydra-MDP continue straight and WoTE does not commit to the turn.The reported comparisons also describe lane deviations among baselines and stagnation by scoring-based methods in a U-turn scenario.Source: B. Comparisons with State-of-the-art… · p. 7
  • 21.4 ms is EMPlan’s lowest latency among the compared anchor-based methods, while its PDMS of 89.6 is the highest.Table II reports inference efficiency for models tested on an NVIDIA RTX A6000.Source: Table II · p. 6, B. Comparisons with State-of-the-art… · p. 7

C. Ablation Study

The ablations examine reward-guided fine-tuning, offset refinement, anchor count, and learned reward distributions. Their results show how these choices relate to planning score, efficiency, and trajectory ranking.

C. Ablation Study

  • Adding offset refinement raises PDMS from 81.6 to 87.8, and adding SFLoss further raises it to 88.3.Offset refinement adjusts trajectories beyond the discrete sparse-anchor set; SFLoss emphasizes hard samples.Source: C. Ablation Study · p. 7
  • A denser anchor set slightly raises PDMS from 88.2 to 88.4 but increases inference latency from 17.5 ms to 24.8 ms; N = 256 is adopted as a favorable trade-off.Source: C. Ablation Study · p. 7
  • 89.6 PDMS with reward supervision outperforms score supervision at 89.0, and KTO with auxiliary supervision consistently improves PDMS over no fine-tuning.The passage attributes complementary roles to preference optimization and auxiliary training signals.Source: C. Ablation Study · p. 7
  • Figure 6 shows higher rewards for feasible trajectories near ground truth and lower rewards for collision or off-road trajectories, while preserving multimodal feasible solutions.In the left-turn example, intended turning trajectories receive higher rewards than non-turning alternatives.Source: C. Ablation Study · pp. 7–8, Figure 6 · p. 7

V. CONCLUSION

EMPlan combines a hybrid architecture with reward-guided fine-tuning and reports state-of-the-art performance with competitive inference efficiency on non-reactive NAVSIM. Its reward-guided training relies on simulator-based evaluation, constraining direct scaling to benchmarks without compatible simulators.

V. CONCLUSION

  • On non-reactive NAVSIM, EMPlan achieves state-of-the-art performance with competitive inference efficiency while enabling real-time multimodal trajectory generation.The conclusion reports improved planning performance beyond pure imitation learning.Source: §V · p. 8
  • Reward-guided training relies on simulator-based evaluation, preventing direct scaling to benchmarks that lack compatible simulation environments.The authors propose simulator-free reward modeling and reactive closed-loop evaluation as future work.Source: §V · p. 8
Loading 2609.38862v1…