DriveMLP: When Training Strategy Outperforms Model Complexity in Autonomous Driving

1 Introduction

• Open-loop IL can suffer distribution shift and causal confusion, with compounding errors pushing policies into unseen states during closed-loop execution.On nuPlan reactive evaluation, surrounding agents follow rule-based IDM policies rather than replayed logs, and open-loop IL methods show substantial degradation.Source: §1 · pp. 1–2
• Standard RL for driving often relies on engineered rule-based rewards or simpler rules paired with extremely large-scale simulation.Demonstration-derived reward alternatives include adversarial or occupancy-matching methods that are computationally expensive and difficult to stabilize at scale [14].Source: §1 · p. 2
• The paper investigates whether a simple policy trained in closed-loop simulation with demonstration-derived supervision can compete with state-of-the-art methods on Highway and nuPlan benchmarks.It also tests whether the recipe can let a lightweight MLP compete with recent generative imitation-based planners.Source: §1 · p. 2
• DriveMLP achieves SOTA performance among purely learned imitation-based planners on evaluated nuPlan reactive splits while running > 15× faster than recent generative planners.The authors attribute much of the performance gain to the training procedure rather than increased model size or complexity alone.Source: §1 · p. 2
• EGRL uses standard RL in a non-reactive log-replay simulator, guided by reward derived from deviations from logged expert trajectories.The paper also proposes an exponential reward formulation and reports improved robustness to action noise and data efficiency versus state-of-the-art open-loop IL.Source: §1 · p. 2

2.1 Problem Formulation

• AD planning is formulated as an MDP with state, action, transition, reward, and discount factor.The state includes map and route information, nearby agents, and static objects; the policy additionally receives ego dynamics, while actions are normalized steering and acceleration commands.Source: §2.1 · p. 3

2.2 Non-Reactive Simulation

• In the non-reactive simulator, the learned policy controls the ego vehicle while other agents replay recorded trajectories and the static environment remains fixed.This keeps the logged expert ego trajectory a consistent reference for dense, step-wise supervision without adversarial reward learning or expert policy access.Source: §2.2 · pp. 3–4

2.3 Trajectory-Based Reward Function

• The reward measures position and speed deviations from expert behavior, with termination conditions defining where replay-based imitation remains valid.Episodes end for off-road motion, collisions, or significant deviation from the expert trajectory; the reward is meaningful only near the expert path where agent interactions remain approximately consistent with logs.Source: §2.3 · p. 4
• Position deviation is normalized by expert displacement to account for speed-dependent tolerance, then mapped to [0, 1] with an exponential function.A small constant prevents division by zero when the expert vehicle is stationary.Source: §2.3 · p. 4

2.4 Policy Learning

• The policy and value function use separate MLPs, with permutation-invariant set encoders aggregating variable-length inputs before policy and value heads.The policy head parameterizes a diagonal Gaussian over continuous actions; the method tests SAC and PPO.Source: §2.4 · pp. 4–5

3.1 Experimental Setup

• The evaluation covers Highway-Env and nuPlan, including reactive and non-reactive settings, and intentionally omits common training augmentations to isolate closed-loop RL with trajectory-based rewards.Source: §3.1 · p. 5

3.2 Controlled Training Strategy Comparison

• With the same DriveMLP architecture, the proposed method beats BC on Highway-Env and performs comparably to AIRL and BC-RL; AIRL takes three times more training steps to converge.On nuPlan, AIRL is described as especially challenging to scale because of instability.Source: §3.2 · p. 5, Table 1 · p. 6

3.3 Transfer Across Planner Backbones

• Applying EGRL to encoders derived from existing planners consistently improves reactive performance, matching or exceeding their original pipelines.Source: §3.3 · p. 6, Table 2 · p. 6

3.4 Comparison with State of the Art

• A roughly 2.3M-parameter MLP outperforms several state-of-the-art imitation-learning baselines in reactive settings and has a smaller non-reactive-to-reactive performance gap.The section identifies Table 3 as the source of the overall nuPlan comparison.Source: §3.4 · p. 6, Table 3 · p. 7
• The authors suggest that closed-loop training may help establish more appropriate links between ego dynamics and planning decisions, consistent with ego-state features helping DriveMLP but hurting open-loop imitation methods.Source: §3.4 · pp. 6–7

4.1 Reward Formulation Ablation

• In the NuPlan Val14 exponential-reward ablation, replacing exponential normalization with inverse distance or raw distance tests components that contribute to performance; normalization is particularly important for stable and effective training.Source: §4.1 · p. 7, Table 4 · p. 7

4.2 Training Dynamics

• DriveMLP’s validation performance plateaus at approximately 50M steps, before training reward plateaus around 80M steps, after early surpassing of the BC baseline.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Mitigating Distribution Shifts

• Across tested action-noise settings, DriveMLP has substantially smaller closed-loop performance degradation than Diffusion Planner, including despite the latter’s training-time data augmentation.The perturbations model execution errors, not general semantic distribution shift.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 Data Efficiency Analysis

• DriveMLP maintains strong performance with significantly reduced demonstration sets, which the authors identify as valuable when expert data is scarce in long-tail scenarios.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5 Inference and Training Cost

• >15× speedup over Flow Planner, with higher Val14-R performance, is reported for DriveMLP.Training requires approximately 112 NVIDIA L4 GPU hours (~4.7× BC); normalized by TFLOPS, DriveMLP uses ~3–5× less compute than Diffusion Planner.Source: §4.5 · p. 8, Table 5 · p. 8

• The trajectory-distance reward remains anchored to expert demonstration quality, although closed-loop exploration partially compensates for scarce data.Source: §5 · p. 9
• EGRL requires a non-reactive log-replay simulator and a dynamic vehicle model for the ego vehicle.Source: §5 · p. 9
• Sim2Real transfer for object-level AD planners remains challenging, and the paper lists overshooting turns, late braking, and missed gap acceptance as failure cases.Source: §5 · p. 9

6 Conclusion

• The authors present non-reactive RL training with trajectory-based rewards as a way to reduce BC distribution shift and standard RL’s reward-engineering burden.They describe DriveMLP as an easy-to-implement competitive baseline for reconsidering emphasis on model complexity.Source: §6 · p. 9
• DriveMLP achieves SOTA reactive performance on nuPlan while using a simpler architecture and >15× faster inference than the previous best IL model.EGRL also outperforms BC, AIRL, and BC-RL and improves reactive performance of other SOTA models.Source: §6 · p. 9
• DriveMLP maintains strong performance with substantially reduced demonstration sets, outperforming Diffusion Planner in data efficiency.The authors attribute mitigation of accumulated execution errors to training that exposes the model to execution-time deviations.Source: §6 · p. 9

A Extended Related Work

• DriveMLP’s specific combination of design components is novel relative to the methods compared in Table 6.The comparison covers closed-loop training, demonstration supervision, lightweight architecture, absence of adversarial reward learning and post-processing, single-stage training, and nuPlan evaluation.Source: A Extended Related Work · p. 15, Table 6 · p. 15

A.1 RL Benchmarks and Baselines for Autonomous Driving

• Existing AD RL baselines use traffic-rule or multi-component handcrafted rewards, whereas this method uses a single trajectory-distance reward without explicit traffic rules.Waymax’s RL baselines underperform BC in their evaluation; Scaling Is All You Need uses BC pre-training and 2.5 billion agent steps.Source: §A.1 · p. 15

A.2 Data Augmentation as an Alternative to Closed-Loop Training

• Unlike fixed trajectory perturbations, RL exploration adapts to current policy weaknesses; under significant action noise, DriveMLP stays above 82 Val14-R while Diffusion Planner falls to nearly 40.Trajectory perturbation augmentation is substantially cheaper per sample because it does not query the current policy.Source: §A.2 · p. 15

A.3 Supervised Fine-Tuning; A.4 Other Hybrid Approaches

• Related hybrid and imitation approaches add distinct mechanisms, while this method directly optimizes a lightweight control policy with standard RL and no pre-trained model or rule-based refinement.CAT-K uses discrete pre-computed trajectories and a two-stage pipeline; CIMRL and RITP add safety constraints or learned priors, and RITP includes test-time rule-based refinement.Source: §A.3 · p. 15, §A.4 · p. 16

B.1 Compounding Error in Open-Loop BC

• Open-loop BC trains on expert-distribution states, but deployment shifts to policy-induced states; the stated performance-gap bound grows quadratically with planning horizon.BC is also agnostic to transition dynamics, so it supplies no corrective signal when the agent drifts into off-distribution states.Source: §B.1 · p. 16

B.2 On-Policy Optimization Mitigates the Distribution Mismatch; B.3 Connection to DAgger

• On-policy closed-loop rollouts compute policy gradients under the states the policy encounters, reducing distribution mismatch by construction.In the non-reactive simulator, other agents replay recorded trajectories, allowing the logged expert ego trajectory to serve as a consistent reference without interactive expert access.Source: §B.2 · p. 16, §B.3 · p. 17

B.4 Relationship to Inverse RL; B.5 Interpretation as Dynamics-Aware Regularized BC

• The trajectory-distance reward acts as a smooth surrogate for occupancy matching and supplies dense, continuous feedback, unlike SQIL’s binary rewards.The paper also interprets the closed-loop objective as imitation plus dynamics regularization, since multi-step rollouts evaluate actions by their consequences.Source: §B.4 · p. 17, §B.5 · p. 17

B.6 Properties of the Exponential Reward

• On nuPlan Val14, the exponential reward beats inverse distance by +5.5 NR / +3.8 R points and the unnormalized linear penalty by +39.9 NR / +38.9 R points.The paper explains that the exponential reward favors committing to one trajectory mode more strongly as the separation between modes increases.Source: §B.6 · p. 17

C.1 Implementation Details; C.2 Evaluation Metrics; C.3 Hyperparameters; C.4 Datasets

• Experiments use PPO on nuPlan and SAC on Highway-Env, evaluate nuPlan under reactive and non-reactive settings, and train on approximately 270K closed-loop scenarios.The reported nuPlan training budget is 100M environment steps; the reward uses the same scaling values across experiments and is reported not to be sensitive to scaling.Source: §C.1 · p. 18, §C.2 · p. 18, §C.3 · p. 18

D InterPlan Results

• On InterPlan’s 80-scenario split, DriveMLP scores close to Flow Planner but below PLUTO; on the full 335-scenario set, it is close to PLUTO and below Flow Planner.It exceeds reported PlanTF and Diffusion Planner scores on both splits.Source: D InterPlan Results · p. 19, Table 8 · p. 19

E.1 MLP Capacity Scaling

• Across tested MLP capacities, EGRL improves reactive performance over BC by approximately 53–55 points, while larger models beyond 2.3M parameters add no observed score gain.Larger models incur higher inference latency and training cost; capacity increases under BC yield only marginal gains.Source: §E.1 · p. 19, Table 9 · p. 19

E.2 Reduced Transformer Policy

• On a reduced 500K-parameter transformer, EGRL achieves approximately +21/+47 (NR/R) points over the BC baseline.The result supports benefits beyond MLP policies, including a reduced transformer architecture.Source: §E.2 · p. 20, Table 10 · p. 20

F.1 Reward Components and Hyperparameter Sensitivity

• Episode termination conditions improve performance while restricting rollouts to safe and approximately log-consistent states.Collision and off-road termination prevent physically invalid overlaps and restrict exploration to drivable areas; deviation termination limits interaction states inconsistent with logged scenarios.Source: §F.1 · p. 20, Table 11 · p. 20
• The trajectory-based reward is reported as substantially more robust to mini-batch-size variation than AD rewards that can drop near zero when batch size changes.The experiments also report sensitivity analyses for discount factor and terminal penalty.Source: §F.1 · p. 20

F.2 Action Repetition Ablation

• At 1 Hz, action repetition balances broader state coverage and a shorter effective RL horizon against fidelity to 10 Hz evaluation; the comparison tests 0.5, 1, and 2 Hz.The authors report that 1 Hz provides the best balance.Source: §F.2 · pp. 20–21, Table 15 · p. 21
• Uniform 1 Hz action repetition is effective, but dynamically varying the control frequency is identified as a future direction.Variable-rate repetition could allow higher-frequency adjustments during critical maneuvers and sparse control in nominal driving.Source: §F.2 · p. 21

F.3 Ego-state Ablation

• Ego-state information improves DriveMLP by +11 NR points but degrades both BC MLP and BC Transformer performance.The authors relate the contrast to the hypothesis that closed-loop RL better captures interactions between ego dynamics and planning decisions.Source: §F.3 · p. 21, Table 16 · p. 21

• Injected noise leaves DriveMLP largely unaffected, while Diffusion Planner gradually drifts from the expert trajectory.Figure 6 contrasts the two planners’ trajectories under noise.Source: G Noise Robustness Visualization · p. 21, Figure 6 · p. 22
• Diffusion Planner can approach a neighboring agent without a corrective response or miss a turn under noise.Source: Figure 6 · p. 22

Hardware and Software

• Experiments use Python 3.9, PyTorch 2.7.1, CUDA 12.6, four NVIDIA L4 GPUs, and 48 CPU cores.Source: H Reproducibility and Simulation… · p. 22

Training Parallelism and Runtime

• Training runs for 100M environment steps in approximately 30 hours using 45 parallel environments.PPO spends less than half of wall-clock time on data collection; the runtime exceeds 100× the throughput of a naive nuPlan wrapper.Source: H Reproducibility and Simulation… · p. 22
• Table 17 reports performance for single-environment reset and step functions.Source: Table 17 · p. 22

Simulation Optimizations

• Five simulation optimizations reduce overhead, including offline preprocessing, contiguous NumPy buffers, spatial indexing, and batched Numba-JIT kernels.The runtime also replaces generic orchestration and map interfaces with domain-specific stepping.Source: H Reproducibility and Simulation… · p. 22
• The simulation stack remains entirely in pure Python, without C/C++ extensions.Source: H Reproducibility and Simulation… · p. 22

Inference Benchmarking Protocol

• Inference latency comparisons use the same batch size and timing procedure on an Intel Xeon 6136 CPU and NVIDIA Quadro P6000 GPU.Source: H Reproducibility and Simulation… · p. 23

• Representative Val14-Reactive failures include turn overshoot into the opposite lane, stalled progress, side collisions, and late braking for lane obstacles.Figure 7 compares expert trajectories in orange with ego trajectories in blue.Source: Figure 7 · p. 23
• On the unseen Starting Left Turn scenario, DriveMLP closely matches Diffusion Planner across Val14 and Test14-Hard NR/R scores.Val14: 88.55/74.08 vs. 86.71/72.49; Test14-Hard: 65.36/72.21 vs. 65.73/67.01.Source: I Failure Cases · p. 23
• DriveMLP scores 86.51 versus 82.02 for replaying the logged expert trajectory on Val14-Reactive.Source: I Failure Cases · p. 23
• Across Val14 and Test14-Hard, DriveMLP exceeds Diffusion Planner on TTC and Drivable Area Compliance but trails slightly on Driving Direction Compliance.Table 18 gives the per-metric and per-scenario comparison.Source: I Failure Cases · p. 23, Table 18 · p. 24