Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negatives

1 Introduction

• Trajectories with nearly identical imitation errors can have different safety outcomes, while conventional imitation learning does not explicitly model unsafe behavior or establish clear safety boundaries.The introduction describes one trajectory as recoverable and another as resulting in collision.Source: §1 · p. 2
• Prior scoring methods can identify unsafe candidates, but discard them after scoring rather than using their failures to optimize the policy.The introduction frames learning from expert-proximate failures as the question BeyondDrive addresses.Source: §1 · p. 2
• Hard negatives must remain spatially close to expert demonstrations while containing explicit safety-critical behavior, making their generation a central challenge.The introduction describes this as a contrast between meaningful proximity and informative unsafe behavior.Source: §1 · p. 3
• BeyondDrive generates unsafe, expert-proximate trajectories with flow matching and diversity-aware sampling, then uses Repulsive Distance Loss to attract predictions to experts and repel them from failures.The loss is described as enabling contrastive safety optimization in trajectory space.Source: §1 · p. 3
• 89.7 PDMS on NAVSIM v1 is achieved by BeyondDrive with Latent TransFuser, alongside reported improvements across multi-modal planners and zero-shot transfer to HUGSIM.The authors report that it outperforms numerous recent methods on NAVSIM v1.Source: §1 · p. 3

2.1 End-to-End Autonomous Driving

• Existing end-to-end methods predominantly imitate successful demonstrations, while trajectory-scoring approaches discard unsafe candidates instead of using their failures in policy optimization.The review also notes that these approaches can recognize undesirable behaviors but do not directly learn safety boundaries from them.Source: §2.1 · p. 4

2.2 Negative Sample Learning in Autonomous Driving

• Prior negative-sample methods use pre-collected or predefined trajectory sets, whereas BeyondDrive trains with unsafe, expert-proximate negatives in continuous trajectory space.The review identifies this as a limitation of preference- and takeover-based methods as well as fixed candidate sets.Source: §2.2 · p. 4

3.1 Flow Matching.

• Flow matching trains a time-dependent vector field along a probability path from a simple prior to target data, using a conditional linear interpolation and its target velocity.The objective regresses the predicted velocity toward x_1 − x_0.Source: §3.1 · p. 5
• After training, flow-matching samples are generated by solving an ODE from random noise; the review describes this approach as offering faster sampling and simpler objectives than diffusion models.The ODE is integrated from t = 0 to t = 1.Source: §3.1 · p. 5

3.2 Classifier-Free Guidance.

• Classifier-Free Guidance combines conditional and unconditional velocity estimates, with its guidance scale balancing conditional fidelity against sample diversity.The paper notes that this flexibility suits generating diverse trajectory candidates close to expert demonstrations.Source: §3.2 · p. 5

4 Method

• 89.7 PDMS is not reported in this section; BeyondDrive’s method has a generation stage followed by joint imitation and repulsive-distance policy optimization.The first stage constructs unsafe yet expert-proximate hard negatives; the second trains the policy to learn expert strategies while avoiding unsafe behaviors.Source: §4 · pp. 5–6, Figure 2 · p. 6

4.1 Problem Formulation

• The planner maps front-view images, LiDAR, ego state, and navigation commands to an eight-waypoint trajectory over four seconds at 2 Hz.Waypoints use ego-frame position and heading.Source: §4.1 · p. 6

4.2 Guided Trajectory Generation via Flow Matching

• Classifier-free guidance and expanded inference noise generate diverse candidates; scoring selects low-safety candidates, then chooses the one closest to the expert trajectory.For each conditioning input, the method samples 64 candidates and uses a safety-and-driving-quality score for filtering.Source: §4.2 · p. 7

4.3 Negative Sample Learning

• Trajectory targets use first-order coordinate differences and sine/cosine heading components, producing a smoother learning target than direct angle regression.The trajectory loss is the ℓ1 distance between predicted and expert differentials.Source: §4.3 · p. 8
• Repulsive Distance Loss pushes predictions away from unsafe, expert-similar negatives while imitation pulls them toward demonstrations, combining both with a semantic loss.The repulsive term uses a clipped ℓ1 distance between predicted and negative trajectory differentials.Source: §4.3 · p. 8

5.1 Benchmark / 5.2 Implementation Details

• NAVSIM v1 uses short-horizon, non-reactive scene unrolls and scores NC, DAC, TTC, comfort, and ego progress; training used 100 epochs and five flow-matching sampling steps.All models used batch size 32; negative-sample generation used CFG scale 0.5.Source: §5.1 · p. 9, §5.2 · pp. 9–10

5.3 Main Results

• 89.7 PDMS on NAVSIM v1 for LTFv7, outperforming numerous recent methods; TransFuser v7 scores 89.6 PDMS.Compared with LTF, LTFv7 gains 5.9 PDMS, with NC +1.0 and DAC +5.1.Source: Table 1 · p. 10, §5.3 · p. 10
• Both TransFuser v7 and LTFv7 exceed 90 EPDMS on reactive NAVSIM v2, with improved DAC and LK (+0.7).The authors associate the LK improvement with enhanced long-range planning capability.Source: Table 2 · p. 11, §5.3 · p. 11
• On four HUGSIM datasets, BeyondDrive achieves state-of-the-art results at easy, medium, and extreme difficulty, improving overall RC by +4.8 and HD-Score by +10 versus LTF.The authors also report gains in DDC and LK, metrics not used in PDMS-based negative-sample selection.Source: Table 3 · p. 11, §5.3 · p. 11

5.4 Ablation Studies and Case Studies

• BeyondDrive brings LTFv7 to 89.7 PDMS after tuning adds +3.6 and normalization adds +1.3; adding its training loss also improves DiffusionDrive, MeanFuser, and WoTE by +1.1, +1.3, and +0.9 PDMS.The normalization uses first-order coordinate differences; the added loss is reported across uni-modal, multi-modal, and world-model methods.Source: §5.4 · pp. 12–13, Table 4 · p. 12
• Increasing CFG from 1.0 to 0.5 raises valid negative-sample scenes from 8.0% to 78.8%; noise scaling to 2.0σ raises them to 96.4%, with Dist_nega 1.68 and LTF at 89.7 PDMS.The reported results highlight CFG and noise scaling as important for producing diverse, expert-proximate hard negatives.Source: Table 5 · p. 12, §5.4 · p. 13
• Increasing λ_rd improves performance only while it remains below λ_imi; otherwise, the model fails to converge.The loss-weight relationship is illustrated in Fig. 4.Source: §5.4 · p. 13, Figure 4 · p. 14
• In three scenarios, negative samples guide LTFv7 away from an unsafe road-boundary departure and a collision-risk forward move, and toward lane-center bus merging.At the stop-required crossroad, LTF* receives NC and TTC scores of 0, while LTFv7 maintains a stationary policy.Source: Figure 3 · p. 13, §5.4 · p. 14

• BeyondDrive enforces safety boundaries using unsafe trajectories that remain close to expert demonstrations.The framework synthesizes these trajectories with flow matching and optimizes a contrastive distance-based objective.Source: §6 · p. 14
• Its objective preserves imitation fidelity while explicitly avoiding safety-critical behaviors.Source: §6 · p. 14
• Experiments on NAVSIM and HUGSIM show consistent improvements across uni-modal and multi-modal planners, with improved closed-loop driving performance.Source: §6 · p. 14

• BeyondDrive does not explicitly model long-horizon interactions.The paper defers this to future work.Source: §7 · p. 14
• High-level semantic constraints are also outside BeyondDrive’s explicit modeling scope.The paper defers these to future work.Source: §7 · p. 14
• Integration with vocabulary-based approaches such as VADv2 is limited by the need for a refinement module after scoring to support safety-oriented training.Source: §7 · p. 14

8 Model Training and Sampling

• Algorithm 1 trains the flow-matching generator with CFG using trajectory-point and dimension counts, dropout probability, map-loss weight, and networks Eθ and vθ.The training loop samples noise and time, may drop conditioning, and adds a weighted map-reconstruction loss.Source: §8 · p. 19
• Algorithm 2 samples N candidates using guidance, noise, ODE-step, and score-threshold settings.Source: §8 · p. 19
• The sampler retains candidates below the safety-score threshold and selects the one nearest the expert trajectory, skipping selection when none qualify.Source: §8 · p. 19
• Table 6 covers BeyondDrive applications to uni-modal LTF, multimodal DiffusionDrive and MeanFuser, and world-model WoTE.It also distinguishes fine-tuned LTF hyperparameters, normalization, and the LTF variant incorporating BeyondDrive.Source: Table 6 · p. 20

9.1 Ablation studies Details on NAVSIMv2 Benchmerk.

• BeyondDrive improves performance across the uni-modal TransFuser, multimodal DiffusionDrive, and world-model WoTE categories on NAVSIMv2.LTFv7 gains or maintains parity across sub-metrics; DiffusionDrive has slight degradation on some metrics, attributed primarily to its trajectory-selection module, while WoTE improves on almost all metrics.Source: §9.1 · p. 20

9.2 Ablation studies on Loss Weights.

• When imitation and RD loss weights are equal, training fails to converge; increasing the imitation-loss weight supports stable convergence and consistent gains.Source: Table 7 · p. 21, §9.2 · p. 21

9.3 Ablation studies on score functions.

• Either DAC-only or TTC-only filtering yields 89.6 PDMS, while each favors its corresponding safety metric.DAC-only filtering raises DAC performance, whereas TTC-only filtering yields superior TTC scores.Source: Table 8 · p. 21, §9.3 · pp. 21–22

9.4 Experimental Results Details on HUGSIM Benchmerk.

• BeyondDrive achieves state-of-the-art zero-shot HUGSIM performance across easy, medium, and extreme scenarios using models trained on NAVSIM navtrain.Source: Table 9 · p. 22, §9.4 · p. 22

10 Negative Samples Analysis

• Most hard negatives lie close to expert trajectories, but their distance distribution has a long tail.The analysis measures Euclidean distance from expert demonstrations.Source: §10 · p. 23
• Collision-related indicators NC and TTC account for the vast majority of samples with zero-valued sub-metrics.Source: §10 · p. 23
• NC and DAC are weakly correlated, while NC, DAC, and EP correlate most strongly with PDMS and dominate failure modes.Source: §10 · p. 23

11 Computing Resource Analysis

• Approximately 5.86 hours is the total reported time cost for the flow-matching generator and trajectory safety assessment.The resource breakdown is presented in Fig. 6.Source: §11 · p. 24, Figure 6 · p. 24
• Training spans 100 epochs on 8 GPUs, using 3.9 GB of memory per GPU and taking approximately 3 minutes per epoch.The per-GPU batch size is 4.Source: §11 · p. 24
• Inference uses one GPU with 1.3 GB of memory to generate 64 trajectories per sample, while safety assessment runs on a 96-core CPU for 45.2 minutes.Source: §11 · p. 24

12.1 Visualization on HUGSIM Benchmerk.

• In HUGSIM, successful cases include navigating around an obscuring bus and a car ahead before returning to the lane.The figures also show passing a pedestrian crossing and maneuvering around a stationary roadside car.Source: §12.1 · p. 24, Figure 7 · p. 25, Figure 8 · p. 26
• In a HUGSIM meeting scenario, an oncoming vehicle swerves into the ego lane, and the resulting collision is attributed to the other agent’s excessive aggression.The authors describe this closed-loop environment as significantly adversarially difficult.Source: §12.1 · p. 24, Figure 9 · p. 27

12.2 More Visualization on NAVSIM navtest Benchmerk.

• On NAVSIM navtest, the straight-driving, left-turn, and right-turn comparisons demonstrate improved driving safety with the method.The corresponding comparisons appear in Figs. 10–12.Source: §12.2 · p. 24, Figure 10 · p. 28, Figure 11 · p. 29