PaperlayerView exact revision on arXiv

Source-linked AI summary

Guiding End-to-End Driving Models with Endpoint-Constrained Trajectory Optimization

Brayden Zhang, Mahsa Golchoubian, Igor Gilitschenski, Boris Ivanovic, Kashyap Chitta

arXiv:2609.31383v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Open-loop waypoint metrics can miss physically inconsistent intermediate paths that are hard for controllers to track, despite strong offline scores. The paper introduces ECO, which anchors executed history and the predicted endpoint while refining intermediate waypoints. Across two simulators, ECO improves aggregate closed-loop scores for every evaluated policy; on HUGSIM, VaVAM’s HD-Score rises from 18.1 to 31.0.

  • Problem

    Open-loop behavior cloning and displacement metrics can overlook physically inconsistent intermediate trajectories that are difficult for a controller to follow, despite strong open-loop scores.

    Source: §I · p. 2

  • Method

    ECO is a training-free post-processor that anchors executed history and the policy’s endpoint while optimizing intermediate waypoints.

    Source: §I · p. 2, §IV · p. 3

  • Results

    ECO raises aggregate closed-loop scores for every evaluated policy across two simulators; on HUGSIM, VaVAM’s HD-Score increases from 18.1 to 31.0.

    Source: §I · p. 2, C. ECO Improves Closed-Loop Driving · p. 5

  • Takeaways & Limitations

    The results suggest that closed-loop driving performance depends not only on a policy’s prediction but also on whether its waypoint sequence stays within the controller’s motion limits.

    Source: §VI · p. 7

    Because ECO fixes the endpoint and does not reason about other agents, it cannot add an unpredicted stop, and a separate safety layer may still be necessary.

    Source: E. Limitations · p. 7

Abstract

from arXiv · show

End-to-end driving policies are commonly trained through open-loop behavior cloning, yet they must ultimately operate in closed-loop when deployed on a vehicle, creating a fundamental mismatch between training and execution. Beyond the commonly studied effects of covariate shift and causal confusion, we identify a complementary factor for this open-loop/closed-loop gap: waypoint-based supervision and displacement metrics do not ensure that the intermediate trajectory is physically coherent or easy for the controller to track. We observe that these inconsistencies concentrate primarily at intermediate waypoints, while the predicted endpoint remains comparatively reliable. Based on this observation, we introduce Endpoint-Constrained Optimization (ECO), a lightweight postprocessing layer that anchors the trajectory to the vehicle's executed history, preserves the policy's predicted endpoint, and reshapes the intermediate waypoints to improve feasibility. ECO requires no map, privileged simulator state, or additional training, and can be inserted between a broad range of waypoint-emitting policies and their controllers. Across two closed-loop simulators, it improves the aggregate closed-loop score of all six evaluated generative and regression-based driving policies, and the gains tend to increase with how often the base plans violate motion limits. On HUGSIM, ECO improves VaVAM from 18.1 to 31.0 HD-Score (+71%), achieving 1st place on the HUGSIM Closed-Loop Driving Challenge. Similarly, on AlpaSim, ECO increases the scene scores of VaVAM and DiffusionDrive by 123% and 22%, respectively. These results show that for a broad collection of end-to-end driving models, repairing the intermediate geometry of predicted trajectories without changing the policy's predicted endpoint can substantially improve closed-loop performance.

I. INTRODUCTION

Open-loop waypoint metrics can overlook physically inconsistent intermediate paths that are difficult for controllers to follow, contributing to the open-loop/closed-loop gap. The paper introduces ECO, a training-free post-processor that refines intermediate waypoints while preserving the policy’s endpoint and executed history.

I. INTRODUCTION

  • Open-loop behavior cloning can yield strong trajectory metrics while policies fail in closed-loop, where actions change future observations and predictions.Waypoint metrics may miss accelerations, jerks, and discontinuities in intermediate paths that are difficult for a controller to track.Source: §I · p. 2
  • Most prior work on the open-loop/closed-loop gap focuses on covariate shift and causal confusion, while this paper examines waypoint physical plausibility as a complementary factor.Source: §I · p. 2
  • ECO refines intermediate waypoints while keeping the executed history and policy endpoint fixed, without additional training.The layer is designed for waypoint-emitting policies with generative or regression-based heads.Source: §I · p. 2
  • The authors report increased aggregate closed-loop scores for every evaluated policy across two simulators and first place in the HUGSIM Closed-Loop Driving Challenge.The supplied passage also reports evaluation across five datasets.Source: §I · p. 2
  • The paper’s contributions include analyzing where trajectory violations occur and conducting controlled closed-loop studies of refinement and transfer.Source: §I · p. 2

II. RELATED WORK

Related work addresses closed-loop driving through interactive data collection, policy fine-tuning, training and evaluation analysis, trajectory refinement, and control-level safety methods. ECO instead post-processes emitted trajectories at the policy–controller interface without changing policy training or requiring a learned scorer.

II. RELATED WORK

  • End-to-end driving studies use photorealistic simulators such as HUGSIM [11] and AlpaSim to evaluate online interaction, and the paper reports a gap for generative planning heads.Source: §II · p. 2
  • Interactive data collection and simulation-based closed-loop fine-tuning use expert feedback or closed-loop rewards to modify policies.Source: §II · p. 2
  • Other work analyzes open-loop training biases, including target-point-following shortcuts, while ECO targets physical implausibility at the policy–controller interface without changing training.Source: §II · p. 2
  • Trajectory refinement ranges from classical optimization and driving-specific planning to execution-aware selection, safety filters, learned refinement, and test-time search.Examples include CHOMP, TrajOpt, PDM-Closed, Hydra-NeXt, and TOAD [35].Source: §II · p. 2

III. PROBLEM FORMULATION

The formulation places a post-inference optimizer between a pretrained waypoint policy and its trajectory-tracking controller. It optimizes intermediate waypoints using the emitted plan and executed ego history while retaining the plan’s horizon, timing, and endpoint.

III. PROBLEM FORMULATION

  • ECO prepends two past poses and the current pose, then refines the waypoints between those anchors and the final predicted waypoint before the controller tracks the result.Source: Figure 2 · p. 3
  • The policy predicts N waypoints from images and a navigation command, and the controller tracks the plan, applies its first action, then queries the policy again.Source: §III · p. 3
  • The post-processor maps a predicted plan to an optimized controller reference with the same horizon and timing, using only the plan and executed ego history.Source: §III · p. 3
  • The formulation permits repair of intermediate waypoints while keeping the last predicted waypoint fixed, because motion-limit violations concentrate in the plan interior.Source: §III · p. 3

IV. ENDPOINT-CONSTRAINED OPTIMIZATION

ECO anchors the executed history and predicted endpoint, then optimizes intermediate waypoints for smoothness, turn sharpness, and fidelity to the original plan. The reported nuScenes-subset results show that small paired open-loop changes can accompany closed-loop score gains.

IV. ENDPOINT-CONSTRAINED OPTIMIZATION

  • ECO prepends available executed poses, including up to two recent poses and the current pose, and fixes those anchors and the final predicted waypoint while optimizing the intervening waypoints.At episode start, it anchors only the executed poses available; finite differences over the combined history and predicted trajectory encourage a smooth transition.Source: §IV · p. 3
  • The objective balances smoothness, turn sharpness, and deviation from the predicted plan over the extended trajectory, applying the deviation term to the plan alone.Source: §IV · p. 3, Table I · p. 3
  • L-BFGS-B solves the optimization from the predicted plan in a median 20.7 ms on a Xeon Gold 6238R CPU.The reported median was 31 iterations.Source: §IV · p. 3
  • +16.5 for VaVAM and +9.5 for UniAD in HD-Score, while paired ADE/FDE changes are only centimeters.Table II covers a 34-scene nuScenes subset of the HUGSIM test set; paired deltas compare raw and ECO-processed plans at the same state.Source: Table II · p. 4

V. EXPERIMENTS

The experiments examine where trajectory inconsistencies arise, whether ECO improves closed-loop driving across policies and simulators, and which components drive its gains.

V. EXPERIMENTS

  • The experiments ask where trajectory inconsistencies arise along planned trajectories.Source: §V · p. 4
  • They test whether ECO improves closed-loop driving across policies and simulators.Source: §V · p. 4
  • They use controlled ablations to examine which components drive the gains.Source: §V · p. 4

A. Experimental Setup

The evaluation uses HUGSIM and AlpaSim closed-loop benchmarks, a diverse set of driving policies, and shared protocols for scoring and trajectory-feasibility analysis.

Benchmarks and evaluation splits

  • HUGSIM evaluation uses a 185-scene test split, with nuScenes scenes used for ablations; AlpaSim evaluation uses its 441-scene NuRec validation set.HUGSIM includes four difficulty levels; hard and extreme modes feature aggressive actors that may perform collision-seeking maneuvers.Source: A. Experimental Setup · p. 4
  • AlpaSim uses its own traffic simulation and vehicle dynamics, and tracks plans with an MPC controller rather than HUGSIM’s iLQR-based controller.Source: A. Experimental Setup · p. 4
  • Fig. 3 measures collision overlap at intended rollout frames and compares motion-limit violations for a single plan, eight-plan average, and ECO plan.Source: Figure 3 · p. 4

Policies and controllers

  • The evaluation covers VaVAM, DiffusionDrive, DrivoR, UniAD, VAD, and Latent TransFuser, using the same scenes, controller, and protocol unless specified otherwise.Source: A. Experimental Setup · p. 4

Metrics and evaluation protocol

  • HUGSIM’s HD-Score combines route completion with gated no-collision and drivable-area scores, while time-to-collision and comfort contribute to the remaining score.Source: A. Experimental Setup · p. 4
  • A waypoint counts as violating a motion limit when finite differences exceed a specified yaw-rate, yaw-acceleration, longitudinal-acceleration, or jerk threshold.The limits are used for evaluation, not in ECO’s objective, and are held fixed across policies and simulators.Source: A. Experimental Setup · pp. 4–5

Implementation details

  • ECO’s smoothness, turn, and deviation weights were selected on a small held-out HUGSIM train subset and then frozen for reported evaluations.Moderate threefold increases or decreases changed HD-Score by no more than 2.4 points.Source: A. Experimental Setup · p. 5

B. Where Errors Arise Along Planned Trajectories

Trajectory problems cluster in intermediate waypoints, while good agreement with human trajectories does not by itself indicate closed-loop executability.

B. Where Errors Arise Along Planned Trajectories

  • Collision episodes have lower median displacement from the human path than completed episodes, 1.2 m versus 2.0 m, so geometric agreement alone does not indicate closed-loop ease.Source: B. Where Errors Arise Along Planned… · p. 5
  • ECO raises closed-loop score by +16.5 points for VaVAM and +9.5 for UniAD while changing trajectories relative to the human route by only centimeters.For UniAD, the change is slightly away from the human route, yielding higher ADE.Source: B. Where Errors Arise Along Planned… · p. 5
  • Collision overlap rises from 1.2% at the first predicted waypoint to 8.2% at 2.5 s, then falls to 3.5% at the endpoint; motion-limit violations also concentrate in intermediate waypoints.Fig. 3 compares collision overlap and motion-limit violations along the plan.Source: B. Where Errors Arise Along Planned… · p. 5

C. ECO Improves Closed-Loop Driving

ECO improves aggregate closed-loop scores across policies and simulators, with the largest reported HUGSIM gain for VaVAM and gains that vary with scene difficulty and plan feasibility.

C. ECO Improves Closed-Loop Driving

D. What Causes the Improvement?

ECO’s benefit depends on its boundary conditions and exceeds that of the tested alternative trajectory-shaping operators. Controller replay results suggest the gain reflects improved tracking across the horizon, while ECO’s benefit tends to increase with base-plan motion-limit violations.

Boundary conditions

  • 39.6 HD-Score with executed-history anchoring and a fixed endpoint exceeds the base policy’s 23.1.The boundary-condition comparison shows lower scores when history is not anchored or the endpoint is left free.Source: D. What Causes the Improvement? · p. 6, Table V · p. 6

Base-plan deficit

  • ECO’s benefit tends to increase with the fraction of base-plan waypoints violating a motion limit.The comparison covers VaVAM variants and four other policies on the same nuScenes test split in HUGSIM.Source: Figure 5 · p. 7

Choice of objective

  • 39.6 HD-Score for ECO exceeds the 28.9–34.8 HD-Scores of five tested trajectory-shaping alternatives under the same boundary conditions.The alternatives are cubic smoothing spline, Savitzky–Golay filtering, moving average, constrained bicycle-model optimization, and temporal ensembling.Source: D. What Causes the Improvement? · p. 7

Simplified tracking

  • 8.5% lower controller cost, from 101.2 to 92.6, and fewer steering-rate-limit steps (20.1% to 12.9%) accompany ECO trajectories.Controller cost was lower for 83% of 1,500 paired trajectories, while first-waypoint error barely changed (0.45 m to 0.44 m).Source: D. What Causes the Improvement? · p. 7

E. Limitations

ECO preserves the policy’s endpoint, which limits the changes it can make when that endpoint is poor. It also does not reason about other agents, so a separate safety layer may still be necessary.

Endpoint constraint

  • A poor predicted endpoint limits ECO: it cannot add a stop that the policy did not predict.The fixed-endpoint design constrains what the optimizer can correct.Source: E. Limitations · p. 7

Other-agent safety

  • ECO does not reason about other agents, so a separate safety layer may still be necessary for collisions.The limitation is specifically noted in relation to collisions.Source: E. Limitations · p. 7, Figure 6 · p. 7

VI. CONCLUSION

The paper frames trajectory executability as a complementary aspect of the open-loop/closed-loop gap and introduces ECO to refine intermediate waypoints while preserving the endpoint and executed history. Its results suggest that closed-loop performance depends not only on policy predictions but also on whether the controller can follow the exposed waypoint sequence.

Conclusion

  • ECO is a lightweight, training-free optimizer that anchors executed history and the predicted endpoint while correcting the intervening trajectory.The paper reports consistent closed-loop gains across policies and simulators and presents ECO as complementary to methods that improve high-level planning.Source: §VI · p. 7
Loading 2609.31383v1…