PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning

Xirui Li, Zhe Liu, Xiaoqing Ye, Wenhua Han, Yifeng Pan, Junyu Han, Hengshuang Zhao

arXiv:2606.24231v1cs.AI

TL;DR

Multimodal planners have lacked a framework combining dense reward supervision with generative proposal modeling. FlowR2A learns a reward-conditioned action distribution from dense trajectory-reward pairs and achieves state-of-the-art performance on NAVSIM v1 and v2 with substantially higher-quality multimodal proposals than prior methods.

  • Problem

    Existing scoring-based and anchor-based planners separate dense reward supervision from generative proposal modeling, leaving their combination an open planning question.

  • Method

    FlowR2A learns a reward-conditioned action distribution from dense trajectory-reward pairs using flow-based decoding, fine-grained rewards, and controllable test-time sampling.

  • Results

    FlowR2A achieves state-of-the-art performance on NAVSIM v1 and v2, with substantially higher-quality multimodal proposals than prior methods.

  • Takeaways & Limitations

    FlowR2A generates high-quality multimodal proposals consistently within the feasible action distribution.

  • Takeaways & Limitations

    FlowR2A depends on reward-signal fidelity from the NAVSIM rule-based simulator, whose inaccurate subscores can propagate label noise into training.

Abstract

from arXiv · show

Multimodal driving planning faces a long-standing tension between two paradigms: scoring-based methods benefit from dense reward supervision but are confined to a fixed action vocabulary, while anchor-based methods generate proposals dynamically yet suffer from sparse supervision constrained to a single ground-truth trajectory. In this work, we propose FlowR2A, which resolves this tension by reframing simulation-based rewards from discriminative targets into generative conditions. By learning the reward-conditioned action distribution from dense trajectory-reward pairs with a flow-matching decoder, FlowR2A unifies the dense supervision of scoring-based methods with the proposal generation of anchor-based methods in a single generative model, forcing the model to internalize the correlation between an action and its outcomes in safety, progress, comfort, and rule compliance. To balance hard safety constraints against soft progress objectives, we introduce fine-grained per-timestep reward conditioning and reward noise augmentation. The generative formulation naturally supports controllable test-time sampling via reward guidance and anchored sampling, producing high-quality proposals. FlowR2A achieves state-of-the-art results on the NAVSIM v1 and v2 benchmarks, with multimodal proposals of substantially higher quality than prior methods.

1 Introduction

FlowR2A addresses the tension between dense reward supervision and generative proposal modeling by learning a reward-conditioned action distribution from dense trajectory-reward pairs. It further enables controllable sampling and achieves state-of-the-art performance with higher-quality multimodal proposals on NAVSIM v1 and v2.

  • Existing paradigms: Scoring-based planners [5] [32] [28] [33] evaluate a fixed action vocabulary with dense reward labels, whereas anchor-based planners [17] [35] [52] [24] dynamically decode proposals but supervise only the closest one to GT.The resulting tension makes dense action-reward supervision and generative proposal modeling mutually exclusive in prior paradigms.
  • Limitations: Anchor-based winner-takes-all supervision leaves many anchors unsupervised, producing low-quality or degenerate proposals and inheriting imitation-learning pathologies [9] [10].These include shortcut learning from ego status and unawareness of action consequences.
  • Motivation and approach: FlowR2A learns p(a|r) from dense trajectory-reward pairs, unifying scoring-based dense supervision with anchor-based generative proposal modeling.Simulation-based rewards characterize safety, progress, comfort, and rule compliance across the action space, while the formulation supports controllable test-time sampling.
  • Reward conditioning: Fine-grained per-timestep safety and compliance rewards sharpen hard constraints, while reward noise augmentation helps the decoder approach high-reward feasible boundaries without crossing into infeasibility.The two designs address the practical challenge that high-reward actions often lie near the feasible-region boundary.
  • Results: FlowR2A achieves state-of-the-art performance on NAVSIM v1 and v2 [10] [3] and produces substantially higher-quality multimodal proposals than prior methods.The result supports modeling the entire conditional action distribution rather than relying on fixed candidates or single-GT supervision.

2 Preliminaries

Flow matching defines a probability path from Gaussian noise to data and trains a velocity field to follow that path. FlowR2A uses this framework to generate clean action samples by integrating the learned dynamics with a 20-step Euler solver.

  • Flow Matching: Flow matching [36] [1] [38] linearly combines data x ∼ p_data(x) and Gaussian noise ϵ ∼ N(0, I) into noisy samples z_t.With a straight rectified-flow path, z_0 is pure noise and z_1 is the clean sample.
  • FlowR2A Pipeline: FlowR2A injects reward embeddings through AdaLN into a flow-based action decoder conditioned on scene features, supervised by velocity matching.The training pipeline samples action-reward pairs to construct noisy samples z_t.
  • Flow Matching: The model v_θ(z_t, t) is trained with velocity matching to predict the flow velocity along the noise-to-data path.
  • Sampling: At inference, clean samples are generated by solving d z_t = v_θ(z_t, t) d t from z_0 ∼ N(0, I) to z_1 using a 20-step Euler solver [12].

3 Method

FlowR2A learns a reward-conditioned action distribution from dense simulated action-reward pairs, combining fine-grained reward conditioning with flow-based trajectory generation. Its perception, reward encoding, proposal decoding, and mode selection support controllable multimodal planning at inference.

  • Dense Action-Reward Pairs: FlowR2A constructs dense training supervision by pairing 8192 clustered four-second trajectories with simulated reward labels spanning safety, progress, comfort, and rule compliance.The trajectories are clustered from 700K nuPlan trajectories and simulated in NAVSIM to produce action-reward pairs.
  • Fine-grained Reward Signals: Fine-grained reward signals expose submetrics and per-timestep hard-constraint arrays, resolving ambiguities in scalar scores while strengthening collision and road-compliance conditioning.The per-timestep arrays encode projected collision time and whether the ego vehicle remains on-road and on-route.
  • Encoders: The model encodes multi-view images, BEV LiDAR, ego status, and driving commands, while independently embedding reward signals and randomly dropping them to enable subset conditioning and classifier-free guidance [19].The perception encoder also uses auxiliary detection and semantic-segmentation losses, and an imitation head provides an inference anchor.
  • Flow-based Action Decoder: A flow-based action decoder denoises noisy continuous trajectories conditioned on scene and reward features, using transformer blocks with scene cross-attention and reward-conditioned adaptive normalization.Training corrupts trajectories with Gaussian noise at uniformly sampled times and optimizes the flow-matching decoder loss.
  • Mode Selector: The mode selector ranks decoder proposals using predicted NAVSIM subscores, and two-stage training finetunes it on simulated online proposals while mixing vocabulary trajectories for calibration.Stage one trains the full model jointly; stage two freezes the decoder and trains the selector on proposal distributions matching inference.
  • Inference: At inference, classifier-free guidance steers ODE-based denoising toward high-reward actions while varying the target reward and anchor noise level to control proposal sampling.The high-reward condition fixes all reward entries at maximal values except the target PDM score, which remains a sampling control.

4 Experiments

FlowR2A achieves state-of-the-art performance on NAVSIM v1 and v2 while generating feasible, high-quality multimodal proposals. Ablations show that fine-grained reward conditioning, reward noise, guidance, and mode selection respectively support safety, robustness, and objective balancing.

  • Results on NAVSIM-v1: FlowR2A achieves 92.8 PDMS on NAVSIM-v1, outperforming prior methods by ≥1.1 PDMS, with margins of ≥0.9 TTC and ≥2.1 EP.It leads on both safety and progress despite their typical tension, while single-proposal sampling remains competitive on safety metrics.
  • Results on NAVSIM-v2: FlowR2A leads NAVSIM-v2 with 88.9 EPDMS and the best reported safety and progress scores, but underperforms on extended comfort.The extended-comfort gap is attributed to the current mode selector scoring frames independently, limiting inter-frame consistency.
  • Proposal quality: FlowR2A surpasses iPad’s 64-proposal PDMS within 4 proposals and exceeds average proposal PDMS by +11.5 over iPad and +29.1 over DiffusionDrive.Averaging all proposals reveals the largest advantage, supporting the claim that sampled actions occupy a feasible distribution rather than only the scorer-selected trajectory.
  • Reward conditioning: Finer reward conditioning improves TTC from 88.8 to 94.9, while reward noise prevents precise rewards from acting as trajectory identifiers during high-score inference.Noise encourages mapping reward bands to feasible high-quality actions, and finer conditioning exposes where trajectory constraints are violated.
  • Mode selection and guidance: The mode selector primarily improves progress, raising EP from 84 to 90 as scored proposals increase, while safety remains saturated; CFG is essential and saturates at scale 5.These findings indicate that sampling absorbs hard constraints, whereas selection refines soft objectives.
  • Sampling strategy: With smax = 1.0 and tmin ≤0.5, FlowR2A maintains consistently high performance; lowering smin produces more conservative proposals at the same 0.05 range length.Inference-time reward-range adjustments trade off objectives without changing the range length.

5 Related Works

Prior multimodal driving planners use either fixed-vocabulary scoring or dynamically generated anchor-based proposals. FlowR2A bridges these paradigms by combining dense supervision with proposal generation and by conditioning action distributions on per-timestep multi-signal rewards rather than a single scalar reward, return, or goal.

  • End-to-End Autonomous Driving: End-to-end autonomous driving maps raw sensor input directly to planning output through a single differentiable model.UniAD [20] integrates perception, prediction, and planning; VAD [22] uses vectorized scene representations; and Transfuser [7] fuses multi-view images with LiDAR.
  • Multimodal Driving Planning: Multimodal planners divide into scoring-based methods that select from fixed action vocabularies and anchor-based methods that dynamically decode proposals.Scoring-based methods use dedicated scorers and simulation-based supervision, while anchor-based methods generate proposals from action anchors.
  • Multimodal Driving Planning: FlowR2A unifies the dense supervision of scoring-based methods with the generation ability of anchor-based methods.The related anchor-based methods include truncated diffusion from fixed anchors, reinforcement-learning post-training with dense trajectory scoring, goal-point-conditioned flow matching, and iterative proposal refinement.
  • Reward-Conditioned Policies and Offline RL: Earlier reward-conditioned and offline-RL methods learn policies from logged data conditioned on a single scalar reward, return, or goal.The cited families include reward-conditioned behavioral cloning [25] [45], return-conditioned sequence modeling [4], goal-conditioned supervised learning, and diffusion-based planning [21].

6 Conclusion

FlowR2A learns a reward-conditioned action distribution for multimodal driving planning, unifying dense reward supervision with generative action modeling through fine-grained reward signals. Its decoder consistently generates high-quality multimodal proposals within the feasible action distribution.

  • FlowR2A learns the reward-conditioned action distribution for multimodal driving planning.
  • The framework unifies dense reward supervision with generative action modeling through fine-grained reward signals.
  • Experiments show that FlowR2A consistently generates high-quality multimodal proposals within the feasible action distribution.

A Limitations and Future Directions

FlowR2A’s action-distribution quality is limited by the fidelity of its reward signals, while future extensions could replace NAVSIM’s rule-based simulator with faster learned or proxy reward functions. Because reward biases may affect driving safety, practical use would require runtime monitoring and human oversight, and the paper makes no real-world deployment claims.

  • Limitations: The learned p(a|r) is bounded by reward-signal fidelity because inaccuracies in simulator subscores propagate into supervision.NAVSIM’s ego-area check can produce false detections from gaps between adjacent area polygons, introducing label noise into the per-timestep ego-area array.
  • Future Directions: FlowR2A can potentially extend beyond NAVSIM by replacing rule-based simulation with a learned reward model, hand-crafted proxy metrics, or a pretrained trajectory scorer such as GTRS [33].These alternatives are described as substantially faster than rule-based simulation and capable of supporting online labeling.
  • Broader Impacts: Reward-signal bias or blind spots could propagate into autonomous-driving behavior, requiring independent runtime monitoring and human oversight for practical use.Evaluation is restricted to closed-loop NAVSIM simulation, and the paper makes no claims about real-world deployment.

B Comparison with Prior Generative Planners · C Additional Experiments

FlowR2A differs from prior generative planners by learning reward-conditioned action distributions from dense action-reward pairs rather than relying on single-ground-truth supervision with external multimodality mechanisms. Additional evidence indicates that prior generative policies can collapse toward single-trajectory regression, while FlowR2A requires little inference-time denoising.

  • B Comparison with Prior Generative Planners: FlowR2A trains its decoder on dense action-reward pairs, unlike prior flow or diffusion planners that use one ground-truth trajectory per scene and inject multimodality externally.The prior mechanisms include fixed anchors, goal-point selection, or scorer outputs.
  • B Comparison with Prior Generative Planners: Naive diffusion or flow policies can collapse into single-trajectory regression because deterministic ground-truth supervision gives the noise input little incentive to encode information.Different noise samples therefore converge toward one mode at convergence.
  • B Comparison with Prior Generative Planners: DiffusionDrive [35] uses 20 trajectory anchors, winner-takes-all reconstruction, and a separate classification head to force proposals across a multimodal anchor space.Only the anchor closest to the ground truth receives reconstruction supervision.
  • B Comparison with Prior Generative Planners: GoalFlow obtains diversity by varying a separately constructed high-scoring goal-point condition, while its flow decoder remains trained against a single ground-truth trajectory.Its flow is consequently subject to the same degeneration as the naive policy.
  • B Comparison with Prior Generative Planners: A single denoising step suffices at inference, indicating that the noise input carries little information; Table 5 reports the corresponding NAVSIM navtest latency breakdown.The latency measurements use a single NVIDIA H20 GPU, batch size 1, and 1000 samples.
  • B Comparison with Prior Generative Planners: CATG [37] adds trajectory, endpoint, and ego-progress conditioning, but still trains the flow decoder against a single ground-truth trajectory, making per-condition mode collapse more likely.Its target endpoint is selected from the top-100 candidates of a pretrained GTRS [33] scorer.

C.1 Latency Analysis … D.2 Reward Score Distribution

FlowR2A’s latency is dominated by sequential denoising, while denoising-step and reward-weight controls expose speed–quality and progress–safety trade-offs. Qualitative sampling demonstrates controllable proposal diversity, and reward labels are highly non-uniform across trajectories.

  • C.1 Latency Analysis: Denoising exceeds 75% of per-frame latency, while perception, reward encoding, and mode selection together remain below 22 ms; proposal count changes latency by only 9 ms.Sequential denoising is therefore the dominant computational bottleneck.
  • C.2 Number of Denoising Steps: Using K=10 denoising steps gives a 1.7× speed-up with a 0.6 PDMS regression, whereas PDMS saturates at K=20, the default setting.Latency grows roughly linearly with K.
  • C.3 Mode Selector Aggregation Weights: Changing aggregation weights keeps PDMS within 0.3 while trading progress for safety, raising TTC from 96.0 to 97.1 as EP falls from 90.1 to 88.0.The weights provide a retraining-free inference-time control knob, and official NAVSIM weights remain robust.
  • D.1 Sampling Space Visualization: Increasing rhigh from 0.80 to 1.00 shifts proposals toward higher ego progress, while increasing tinit from 0.75 to 0.95 anchors them closer to imitation-learning outputs.Higher tinit reduces the spatial spread of sampled proposals.
  • D Qualitative Results: Qualitative sampling controls reshape proposal progress and spatial diversity, illustrating controllable generation across the sampled action space.The rhigh and tinit sweeps provide the demonstrated qualitative control mechanisms.
  • D.2 Reward Score Distribution: Across the 8192-trajectory vocabulary, most candidates receive low aggregate PDMS while high-quality actions form a sparse subset, with reward signals carving complementary regions.EP and HC express soft progress and smoothness preferences, whereas per-timestep TTC-time and ego-area labels localize hard constraints.

D.3 Failure Cases · D.4 Extended Comparisons · E NAVSIM Benchmark and PDM Score

FlowR2A’s failures expose perception coverage and, more importantly, mode selection as remaining bottlenecks, while extended scenes and benchmark details broaden evaluation context. The NAVSIM section defines the dataset, simulation pipeline, subscores, and PDMS/EPDMS aggregation differences.

  • D.3 Failure Cases: FlowR2A fails through missed mid-trajectory vehicles, insufficient TTC or DAC behavior, and unsafe mode selection despite feasible alternatives.Figure 11 presents four representative navtest failures, including perception misses, close following, edge drifting, and ranking errors.
  • D.3 Failure Cases: The primary limitations are perception coverage and mode-selector quality, with the selector identified as the more actionable bottleneck.When the decoder produces mostly feasible proposals, the selector can still promote a failing proposal, leaving recoverable headroom.
  • D.3 Failure Cases: Figure 9 visualizes how sweeping high-reward targets and initial denoising times changes 60 sampled proposals per configuration on one navtest scene.All other inference settings remain fixed across the sampling-space grid.
  • D.3 Failure Cases: Figure 10 maps PDMS, NC, EP, HC, TTC-time, and ego-area reward signals across 8192 vocabulary trajectories in a single navtest scene.Brighter colors indicate higher scores for PDMS, NC, EP, and HC, while TTC-time and ego-area selectively show violations.
  • D.4 Extended Comparisons: Figures 12–14 extend the main-paper qualitative comparison to additional navtest scenes.These figures provide further qualitative comparisons beyond the primary examples.
  • E NAVSIM Benchmark and PDM Score: The NAVSIM [10] [3] benchmark section specifies the dataset, simulation pipeline, evaluation subscores, and reward-construction framework.It also defines PDMS aggregation for v1, EPDMS aggregation for v2, and the differences between the two versions.

E.1 Dataset and Simulation Pipeline … F.3.2 Trajectory Sampling

The paper details NAVSIM’s simulation-based evaluation and FlowR2A’s implementation, including dense reward construction, reward-conditioned flow decoding, mode selection, and rebalanced trajectory sampling. NAVSIM v1/v2 differ in safety gates and comfort/compliance terms, while training uses simulated state sequences and score-density-aware sampling.

  • E.1 Dataset and Simulation Pipeline: NAVSIM evaluates 4-second plans by replaying eight waypoints through an LQR-controlled kinematic bicycle at 10 Hz, producing 41 simulated states for scoring.The dataset contains 103k navtrain and 12k navtest frames with multimodal observations, driving commands, agent tracks, maps, and future trajectories.
  • E.2 Subscores; E.3 Aggregation; E.4 Differences Between v1 and v2: NAVSIM combines multiplicative safety/compliance gates with weighted progress, collision, comfort, and lane-quality terms, with v2 adding DDC and TLC gates, replacing C with HC, and adding LK and EC.The PDM reference trajectory normalizes ego progress; EC is omitted when undefined, and v2 applies a human penalty filter to forgive failures also made by the human reference.
  • F.1 Reward Construction: FlowR2A creates dense action-reward pairs by simulating an 8192-trajectory vocabulary for every navtrain scene, using scalar NAVSIM v2 rewards plus higher-resolution per-timestep TTC-time and ego-area arrays.The scalar set includes NC, DDC, TLC, EP, LK, HC, and PDM; EP is safety-gated and normalized, while DAC and TTC are replaced by temporal arrays.
  • F.1 Reward Construction: Reward construction is an offline CPU process costing about one day on 32 machines for scalar rewards and approximately eight hours on four eight-machine nodes for per-timestep rewards.Gaussian noise with σ = 0.05 is added during training to EP and PDM scores but not at inference.
  • F.2.2 Reward Encoder: The reward encoder embeds discrete, scalar, and per-timestep rewards with type-specific modules, while scheduled reward dropout teaches both unconditional and partially conditioned distributions.All rewards are retained with probability 0.5, jointly dropped with probability 0.1, or independently dropped with probability 0.4 and per-reward probability 0.5.
  • F.2.3 Flow-based Action Decoder; F.2.4 Mode Selector: A four-block transformer flow decoder generates 4-second trajectories represented as eight (x, y, θ) waypoints, conditioning AdaLN modulation on diffusion time and the 256-dimensional reward embedding.The mode selector scores proposals with a lightweight transformer and predicts NAVSIM subscores plus auxiliary TTC-time and ego-area arrays, using continuous TTC labels with tbound = 2 seconds.
  • F.3.2 Trajectory Sampling: Trajectory sampling counteracts score imbalance by weighting samples inversely to per-scene PDM-score density, using KDE rebalancing with α = 0.6 and separately drawing 20 trajectories for the decoder and selector.Uniform sampling would underrepresent rare high-score trajectories; α = 0 is uniform over trajectories, whereas α = 1 is uniform over score bins.

F.3.3 Stage 1: End-to-End Training … F.4.1 Default Sampling Configuration

The training pipeline uses jointly optimized stage-1 components followed by selector-only finetuning on decoder proposals and sampled vocabulary trajectories. Inference uses a fixed Euler-based configuration with classifier-free guidance, anchored sampling, and randomized reward and noise controls.

  • F.3.3 Stage 1: End-to-End Training: Stage 1 weights the velocity-matching loss Ldec at 40, while Lperc assigns weights 10, 1, 14, and 10 to agent class, box, BEV semantic, and IL-head losses, respectively, and Lsel receives weight 10.The weights are chosen to balance gradient scales.
  • F.3.3 Stage 1: End-to-End Training: Stage 1 uses reduced selector supervision as an auxiliary scene-feature signal, predicts v2 subscores NC, DAC, EP, TTC, and HC, and excludes per-timestep TTC-time and ego-area heads.This defines the selector targets before the stage-2 finetuning procedure.
  • F.3.3 Stage 1: End-to-End Training: To stabilize Ldec near t → 1, training clips its denominator to max(1 − t, 0.05); this loss-only clip does not affect the underlying sampling formulation.The clipped conversion is vθ = (xθ − zt) / max(1 − t, 0.05).
  • F.3.4 Stage 2: Mode Selector Finetune: Stage 2 freezes every component except the mode selector and finetunes it for two epochs to bridge stage-1 vocabulary trajectories and inference-time decoder proposals.Training uses a batch size of 256 across eight H20 GPUs, while other settings match stage 1.
  • F.3.4 Stage 2: Mode Selector Finetune: Stage 2 trains the selector on decoder proposals sampled as at inference, 32 inverse-density-weighted random vocabulary trajectories, simulator-derived subscores, and per-timestep TTC-time and ego-area arrays.The selector predicts these targets under Lsel defined in App. F.2.4.
  • F.4.1 Default Sampling Configuration: Default sampling uses an Euler scheduler with 20 denoising steps, CFG scale wg=5, and the IL-head output as the anchor for zero-shot anchored sampling.The controls rhigh and tinit are independently and uniformly sampled per proposal, with target scores [0.9, 1.0] and initial noise levels [0.5, 0.9].

F.4.2 Per-Experiment Sampling Settings · F.4.3 Reward Subset for Classifier-Free Guidance

Per-experiment inference settings vary only in proposal count, tmin lower bounds, and single-proposal construction. Classifier-free guidance prescribes selected rewards that combine a randomized target score with hard safety, drivable-area, and comfort constraints.

  • F.4.2 Per-Experiment Sampling Settings: Inference settings differ across experiments only in proposal count, tmin lower bound, and single-proposal construction, as summarized in Table 8.Table 8 defines deviations from the default configuration in Appendix F.4.1.
  • F.4.2 Per-Experiment Sampling Settings: The single-proposal Tab. 1 evaluation fixes rhigh and tinit to representative values to avoid noise from random sampling.Other single-proposal experiments sample both controls from the default ranges, matching multi-proposal inference.
  • F.4.2 Per-Experiment Sampling Settings: Tab. 3 uses 64 proposals to match the comparison setting of iPad [17].
  • F.4.3 Reward Subset for Classifier-Free Guidance: Classifier-free guidance conditions on three scalar rewards—NC, HC, and target PDM score—and two per-timestep rewards, TTC-time and ego-area.All entries except the target score are set maximally, while the target score is randomized over [smin, smax].
  • F.4.3 Reward Subset for Classifier-Free Guidance: The prescribed subset leaves most scene-dependent rewards empty, allowing the learned p(a|r) to balance them during inference.The PDM target score is used as an overall indicator for high-quality actions.
  • F.4.3 Reward Subset for Classifier-Free Guidance: NC and the per-timestep safety and drivable-area rewards are pinned to enforce hard constraints across all proposals.These rewards encode no collision, safety, and drivable-area compliance at every timestep.
  • F.4.3 Reward Subset for Classifier-Free Guidance: HC is included as a stable, easy-to-satisfy regularizer that smooths the resulting trajectory dynamics.
Loading 2606.24231v1…