PaperlayerView exact revision on arXiv

Source-linked AI summary

TerraTransfer++: Demonstration-Free End-to-End Planner

Zikang Xiong, Weixin Li, Zhouchonghao Wu, Akshay Rangesh, Saarth Bonde, Grantland Hall, Chen Tang, Yihan Hu, Wei Zhan

arXiv:2610.30319v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Fleet logs rarely capture safety-critical interactions, while photorealistic closed-loop training is costly and self-play policies read vector state rather than raw sensors. TerraTransfer++ trains a self-play policy, distills its rollouts into a trajectory teacher, and aligns image-conditioned students without logged-trajectory supervision. Its students outperform published methods on the reported HUGSim and NAVSIM evaluations, including 59.4 EPDMS on NAVSIMv2 navhard.

  • Problem

    Fleet logs contain few safety-critical interactions, photorealistic closed-loop training incurs per-step rendering and vision costs, and self-play policies consume vector state rather than raw sensors.

    Source: §1 · p. 3

  • Method

    The paper trains a driving policy through multi-agent self-play, distills its rollouts into a trajectory teacher, and aligns image-conditioned students to either teacher using paired image–scene-state frames.

    Source: §1 · p. 3, §5 · p. 17

  • Results

    The closed-loop student exceeds every published end-to-end baseline on every HUGSim split, while the planner reaches 59.4 EPDMS on NAVSIMv2 navhard, above every published method.

    Source: §1 · p. 5, §5 · p. 17

  • Takeaways & Limitations

    The reported results support demonstration-free transfer of self-play driving competence to both closed-loop control and planning students.

    Source: §5 · p. 17

    Alignment data rarely covers severe weather, unusual times of day, or rare scenario geometries, and the teacher may rely on occluded agents that cameras cannot observe.

    Source: §5 · p. 17

Abstract

from arXiv · show

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all existing paradigms: imitation learning requires fleet-scale driving logs whose behavior distribution only thinly covers the safety-critical tail; closed-loop reinforcement learning in a photorealistic simulator, while offering diverse behaviors, is bottlenecked by sample-efficiency, per-step cost of rendering, and a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: at millions of rollout steps per second, one can obtain a state distribution with natural diversity in collisions, near-misses, and recoveries that no driving log contains, and a highly reactive policy emerges from a basic driving-quality reward. Our approach exploits this asymmetry. We pretrain a single policy by self-play, distill its rollouts into a flow-matching trajectory teacher that also plans, and train lightweight adapters on a frozen pretrained vision backbone to reproduce the latent space of either teacher through the action KL divergence, a batch-relational low-rank structural loss, and, for the planner, a trajectory-matching loss. Every target traces back to the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. From this one self-play teacher we obtain two end-to-end students: a closed-loop policy that outperforms every published imitation-based end-to-end method on the photorealistic 3D Gaussian splatting closed-loop benchmark HUGSim, and a planner that reaches 59.4 EPDMS on NAVSIM navhard, above every published method including DriveZero-Scale (57.1), and 95.5 PDMS on navtest, without human demonstrations.

1 Introduction

The introduction presents self-play as a scalable source of diverse driving behavior and proposes transferring that behavior to image-conditioned students without logged-trajectory supervision. The resulting students outperform published methods on the reported HUGSim and NAVSIM benchmarks.

1 Introduction

  • Fleet logs contain millions of driving hours, yet safety-critical events remain rare; photorealistic closed-loop training also incurs rendering and vision-backbone costs at every step.At NHTSA crash rates, a collision occurs only once every ∼16k driving hours; those costs confine image-based closed-loop work largely to fine-tuning pretrained imitation policies.Source: §1 · p. 3
  • Multi-agent self-play in vectorized simulation produces collisions, recoveries, yielding, negotiation, and overtaking at millions of steps per second, using basic driving-quality rewards rather than expert trajectories.One shared policy controls every agent, so policy improvements reshape the traffic each agent faces.Source: §1 · p. 3
  • The three-stage recipe trains a policy from scratch through self-play, distills its realized futures into a flow-matching trajectory teacher, then aligns vision students to either teacher.The self-play policy uses one parameter set to control every agent in each scene; both teachers read vector state.Source: §1 · pp. 3, 5, Figure 1 · p. 4
  • Image-conditioned students learn teacher scene features from paired image–scene-state frames through action-distribution and pairwise-structure losses; the planner also matches the trajectory teacher’s selected plan.Alignment uses no human trajectory targets, online rendering, or expert demonstrations.Source: Figure 1 · p. 4, §1 · pp. 5–6
  • 0.548 aggregate HD-Score exceeds every published end-to-end baseline on every HUGSim split; 59.4 EPDMS on NAVSIMv2 navhard and 95.5 PDMS on NAVSIMv1 navtest exceed the stated published-method comparisons.The navhard score is above every published method, including DriveZero-Scale (57.1); the navtest score is above every published sensor-based method.Source: §1 · pp. 5–6
  • The self-play policy takes 96 hours on 16 A100 GPUs and covers over 2.4B simulated kilometers; its reusable behavior supervises vision frontends without online rendering during alignment.The reported alignment uses 1.83M paired nuPlan frames plus 300K HUGSim domain-adaptation frames, trained on 8 A100 GPUs in 10 hours.Source: §1 · p. 5

2 Method

The method trains a vector-state policy through self-play, distills its behavior into a trajectory-planning teacher, and aligns image-based students to either teacher. The stages use self-play-derived supervision and combine action-distribution matching with relational feature alignment, adding trajectory matching for the planner.

2.1 TerraZero: Self-Play Vectorized Simulator

  • TerraZero spawns vehicles procedurally on compiled nuPlan map topology, assigns reachable-lane goals, and supplies road, traffic-control, and neighboring-entity observations.Training does not use logged trajectories, agent initial states, goals, human actions, or replayed traffic.Source: §2.1 · p. 6

2.2 Self-Play Pretraining: The Teacher

  • Self-play uses one shared policy to control all agents, so their interactions evolve with the policy; reward weights condition it across driver preferences.The reward terms measure goal-reaching, collision, comfort, lane alignment and centering, velocity, and traffic-rule compliance.Source: §2.2 · pp. 6–7

2.3 Trajectory Distillation

  • The trajectory teacher learns from the self-play policy’s actions and realized futures, predicting 80 ego poses over 8 seconds at 10 Hz rather than logged human trajectories.It uses a flow-matching denoiser for plans and an action head trained against the self-play policy.Source: §2.3 · pp. 7–8

2.4 Vision Alignment: The Student

  • The closed-loop student replaces the teacher’s road and partner encoders with image-conditioned encoders while retaining its frozen policy head and ego encoder.The vision encoders share a frozen DINOv3 backbone and use separate adapters.Source: §2.4 · p. 8
  • The method avoids a perception cascade because imperfect recovered state can degrade a teacher trained on clean simulator inputs, and the cascade is not end-to-end differentiable.The student receives images and an ego vector, whereas the teacher also receives vector road and partner state.Source: §2.4 · p. 8
  • Action-distribution matching preserves teacher uncertainty, while structural matching supplies scene-geometry constraints that action logits alone cannot provide.The action loss is forward KL divergence; the two losses are combined with structural weight 0.5.Source: §2.4 · pp. 8–9
  • Structural alignment matches pairwise scene relationships within the teacher’s informative low-rank principal subspace, leaving absolute coordinates and orthogonal noise unconstrained.The method uses k_p=13 and k_r=9, the 80% cumulative-energy cutoffs for partner and road teacher spectra.Source: §2.4 · p. 9
  • The planning student predicts teacher latent tokens from images, aligns actions and scene relations, and adds trajectory matching against a selected teacher plan.Its trajectory loss compares plans over 80 waypoints; at inference it samples one plan without selection.Source: §2.4 · pp. 9–10

3 Experiments

The experiments test whether vision policies distilled from self-play can drive in closed loop and plan in pseudo closed loop without logged human trajectories. They evaluate both students against published driving and planning baselines, then examine the alignment recipe’s components.

3.1 Setup

  • The headline HUGSim evaluation covers 436 scenarios across four dataset splits, reports closed-loop and native HD-Scores, and uses paired nuPlan frames plus 300K HUGSim frames for domain adaptation.The native scorer uses a decel stand-in plan because the single-step controller does not publish a trajectory.Source: §3.1 · p. 10

3.2 Teacher Closed-Loop Results

  • The vector-state teacher reaches 0.562 overall closed-loop HD-Score across HUGSim splits and is treated as a privileged reference, not an image-based baseline.It also scores 94.19 on nuPlan val14 and 70.87 on InterPlan, both above the best prior imitation planner reported there.Source: §3.2 · p. 11

3.3 Closed-Loop Comparison with End-to-End Baselines

  • 0.548 aggregate closed-loop HD-Score: the student exceeds every published end-to-end baseline across HUGSim splits and retains roughly 97.5% of teacher performance.The student also scores 0.528 native HD-Score; its lead holds under both scorers.Source: §3.3 · p. 11, Table 1 · p. 12
  • 0.518 native HD-Score on the shared 345-scenario protocol exceeds DriveZero’s 0.466 and GigaPixel’s 0.385.The comparison uses the student’s comfort-bounded decel stand-in plan, while the other systems are scored on their own plans.Source: §3.3 · p. 12, Figure 2 · p. 13

3.5 Insights and Ablations

  • Road features concentrate 80% of spectral energy in 9 directions and partner features in 13 of 64; performance peaks at the spectrum-predicted truncation and falls on either side.The authors attribute the drops to discarding meaningful directions at low rank and including weakly structured, image-unexplainable variation at high rank.Source: §3.5 · p. 14
  • With identical vector inputs, the privileged probe scores 0.645 versus the teacher’s 0.647; the camera student scores 0.574 versus the probe, rising to 0.636 with HUGSim adaptation frames.The remaining perception gap is linked to agents visible in vector state but not to the camera.Source: §3.5 · pp. 14–16

3.4 Pseudo Closed-Loop Planning on NAVSIM

  • 59.4 EPDMS on NAVSIMv2 navhard and 95.5 PDMS on NAVSIMv1 navtest put the trajectory student above every published sensor-based entry on both splits.On navhard it exceeds DriveZero-Scale by 2.3 points; on navtest it is within 0.4 points of the privileged trajectory teacher.Source: §3.4 · p. 13, Table 2 · p. 15

3.5 Insights and Ablations

  • Under matched data and compute, self-play alignment scores 0.574 versus 0.361 for a modular cascade and 0.206 for imitation; action-only and structural-only losses score 0.434 and 0.373.The combined losses outperform either alone.Source: Table 4 · p. 16, §3.5 · p. 16

4 Related Work

Prior work transfers vector-state self-play competence to images or learns behavior directly from image-based self-play, while TerraTransfer++ retains the self-play policy head and aligns image encoders to its decisions. Its fixed paired-frame alignment uses self-play-derived action, feature-relation, and planning targets without teacher rollouts or logged trajectories.

4 Related Work

  • Vector-state transfer methods differ in supervision and head reuse: GigaPixel uses DAgger, DriveZero uses PPO-teacher trajectories from logged frames, and neither reuses the teacher policy head.GigaPixel transfers through a bounding-box renderer before adapting perception to real images; DriveZero trains a separate student planner.Source: §4 · p. 16
  • Image-space self-play methods avoid a vector-state teacher, but Pictura's policies remain in the non-photorealistic renderer domain, while photorealistic rollout rendering is costly.RAD and World Engine use photorealistic 3D Gaussian splatting simulators only to post-train imitation-pretrained image policies.Source: §4 · pp. 16–17
  • TerraTransfer++ retains the vector-state self-play policy head and trains image encoders to produce its scene features, transferring self-play decisions rather than imitating logged behavior.On fixed paired (image, scene-state) frames, alignment uses action-distribution and feature-relation targets, plus plan targets for the planning student; it needs neither online rendering nor teacher rollouts.Source: §4 · p. 17
  • RAP aligns rasterized counterfactual views from annotated logs to real-image features for imitation learning, whereas TerraTransfer++ aligns image features to self-play teacher decisions.The paired frames are fixed, so alignment requires neither online rendering nor teacher rollouts.Source: §4 · p. 17

5 Conclusion and Future Work

The paper presents a three-stage, demonstration-free recipe: learn driving through multi-agent self-play, distill its rollouts into a trajectory teacher, and align vision students to either teacher. In tested scenarios, the aligned policy retains most of the teacher's competence and the planner exceeds published NAVSIM methods; the authors identify occlusion and limited alignment-scene diversity as future-work boundaries.

5 Conclusion and Future Work

  • The recipe learns behavior through multi-agent self-play, distills rollouts into a planning teacher, and aligns vision students to either teacher using paired image and scene-state frames.No stage trains against logged ego trajectories; alignment targets come from frozen heads reused by the students.Source: §5 · p. 17
  • The trajectory student reaches 59.4 EPDMS on NAVSIM navhard, above every published method, and 95.5 PDMS on navtest, within 0.4 points of the privileged best-of-16 trajectory teacher.In tested scenarios, the teacher outperforms imitation-trained pipelines and the aligned vision policy retains most of its competence.Source: §5 · p. 17
  • Self-play pretraining is a stronger alternative to imitation-based pretraining for learning driving behavior in the matched-data comparison, while imitation remains useful for cues such as brake lights and hand signals.The authors contrast self-play's on-demand collisions, recoveries, and negotiations with imitation's dependence on maneuvers recorded in fleet logs.Source: §5 · p. 17
  • Paired-frame alignment does not resolve the gap when teachers use occluded agents, and alignment data rarely covers severe weather, unusual times, or rare scenario geometries.The authors propose belief-aware privileged distillation and controllable scene generation as extensions.Source: §5 · p. 17

A.1.1 Training Details.

Self-play uses nuPlan map geometry and procedurally generated scenes rather than logged driving trajectories or destinations. The appendix specifies the teacher's PPO setup, observation groups, and reward modification.

A.1.1 Training Details.

  • Self-play uses nuPlan HD maps only for static geometry, while vehicles, goals, vulnerable road users, and obstacles are procedurally generated rather than taken from logs.Controlled vehicles spawn collision-free from lane centerlines; each receives a reachable lane-graph goal 20–100 m away.Source: §A.1.1 · p. 22
  • The decentralized PPO teacher uses V-trace-corrected GAE, clipped PPO, and annealed Adam, with a reference run on 16 A100 GPUs at approximately 3M cumulative agent steps/s.The specified settings include γ=0.99, λ_GAE=0.95, clip coefficient 0.2, and a 5×10^-4 learning rate.Source: §A.1.1 · p. 22

A.1.2 Observations.

  • At each 0.1 s simulator step, observations comprise an ego vector, pooled road elements, and pooled nearby partners represented relative to the ego frame.The ego vector includes kinematics, navigation/goal, vehicle properties, and reward weights; road inputs include lane geometry and controls, while partner inputs encode neighboring entities.Source: §A.1.2 · p. 22

A.1.3 Reward Components.

  • The reward reuses most components and settings from Cusumano-Towner et al. [11], adding a lane-change term that penalizes excessive lane changes.Per-agent, per-episode reward-weight randomization is exposed through the ego observation.Source: §A.1.3 · p. 22

A.1.4 Vehicle Dynamics.

The teacher uses jerk-actuated bicycle dynamics with randomized passenger-car parameters and a 0.1 s step; its integrator also drives HUGSim evaluation. The appendix further defines a closed-loop HD-Score that scores realized rollouts rather than planned futures.

A.1.4 Vehicle Dynamics.

  • The self-play setup uses a discrete 12-action longitudinal/lateral jerk grid and restricts dynamics randomization to passenger-car-sized bodies.Dimensions are sampled as length 3.5–5.5 m, width 1.5–2.5 m, and height 1.2–2.0 m; the 0.1 s step matches the student vision policy's control rate.Source: §A.1.4 · pp. 22–23

A.1.5 From Jerk Actions to Simulator State.

  • Each jerk command is integrated in causal order over Δt: accelerations, speed, steering, then pose, using the bicycle model.The same integrator updates the teacher's training states and supplies states to HUGSim, with no second dynamics model or sim-to-sim conversion.Source: §A.1.5 · p. 23

A.2 Closed-Loop HD-Score

  • The closed-loop HD-Score evaluates the realized rollout, retaining NC, DAC, and route completion while redefining TTC and comfort for the realized trajectory.The native HUGSim score instead evaluates the policy's planned future trajectory.Source: §A.2 · p. 24

A.2.1 Closed-Loop Redefinitions.

The closed-loop scorer evaluates route completion, time-to-collision, and comfort using the realized rollout. It also clarifies how route-completion scores relate to the reference route and native scoring.

A.2.1 Closed-Loop Redefinitions

  • The closed-loop scorer redefines TTC to measure imminent risk on realized rollouts, propagating the ego and visible partners by observed velocities over a short horizon.It tests propagated vehicle boxes and the scene’s static 3DGS background; a collision already present at the keyframe gives TTC_t = 0.Source: §A.2.1 · p. 24
  • COM applies native comfort bounds to the realized ego trajectory within a fixed-duration forward window, requiring all bounds to hold at every sample.The default window is 1.0 s; the scorer derives longitudinal and yaw kinematics from finite differences.Source: §A.2.1 · p. 25
  • Route completion uses the ego’s nearest reference camera pose to compute an index fraction, with the scenario score capped at one.The 0.9 normalizer saturates the score after 90% of reference poses, so a score of 1 does not necessarily mean the route endpoint was reached.Source: §A.2.1 · p. 25
  • Table 6 contrasts native and closed-loop HD-Score subscores: NC, DAC, route completion, and aggregation stay unchanged, while TTC and COM are redefined.Source: Table 6 · p. 25

A.2.2 Native vs. Closed-Loop Summary.

The closed-loop scorer changes TTC and COM relative to native scoring, while NC and DAC remain comparable. The reported baseline ranking on nuScenes is preserved under either scorer.

A.2.2 Native vs. Closed-Loop Summary

  • TTC and COM are not directly comparable to their native counterparts because they use realized rather than planned motion, and closed-loop TTC is continuous rather than binary.Source: §A.2.2 · p. 26
  • NC and DAC remain comparable across the two scorers.Source: §A.2.2 · p. 26
  • On nuScenes, native scoring preserves the closed-loop ranking ECO > LTF > {VAD, UniAD}; every Table 1 method is also scored with the closed-loop variant on its realized rollout.VAD and UniAD differ by at most 0.008 under either scorer.Source: §A.2.2 · p. 26

A.3 Self-Play Policy Baseline Performance

The self-play teacher and aligned student are reported across 436 HUGSim scenarios under native and closed-loop scoring. The teacher’s Extreme-tier scores are limited by route completion on nuScenes and also by drivable-area compliance on KITTI-360.

A.3 Self-Play Policy Baseline Performance

  • Tables 7 and 8 cover all 436 HUGSim scenarios and difficulty tiers under both scorers; the teacher uses privileged vectorized state, while the student consumes camera images.The native block scores a deceleration stand-in plan; the closed-loop block scores the realized rollout, and route completion is shared.Source: Table 7 · p. 26, §A.3 · p. 26, Table 8 · p. 27
  • 0.562 closed-loop HD-Score for the teacher versus 0.548 for the student across the pooled HUGSim scenarios.The aggregates correspond to the all-four-subsets rows in Tables 7 and 8.Source: §A.3 · p. 27
  • On nuScenes, route completion falls from 0.95 on Easy to 0.30 on Extreme, pulling the Extreme HD-Score to 0.28 despite HD-PDMS of 0.89.On KITTI-360, Extreme DAC is 0.569 and HD-PDMS is 0.354; these conditions pull pooled Extreme HD-PDMS down to 0.767.Source: §A.3 · p. 27

B.1 Structural Loss Rank Ablation

The ablation varies the structural-loss truncation rank while holding the rest of the training recipe fixed. It evaluates closed-loop HD-Score over all 436 HUGSim scenarios.

B.1 Structural Loss Rank Ablation

  • The rank ablation compares half the headline ranks (k_p = 7, k_r = 5), twice the ranks (k_p = 26, k_r = 18), and full-coordinate matching against the headline setting.The headline uses k_p = 13 and k_r = 9, corresponding to 80% cumulative-energy cutoffs of the teacher spectra.Source: §B.1 · p. 28
  • Every rank variant keeps the frozen teacher, headline data including 300K HUGSim domain-adaptation frames, action loss, λ = 0.5, and training budget fixed.Source: §B.1 · p. 28
  • All variants are evaluated using closed-loop HD-Score on all 436 HUGSim scenarios; Table 3 reports the results, and the text says performance peaks at the spectrum cutoff.Source: §B.1 · p. 28

B.2 Vision Backbone Ablation

Under a fixed alignment recipe, DINOv3 yields a higher closed-loop score than DINOv2 and ResNet-50. The authors connect the ordering to how much teacher-feature structure the linear adapters can use.

B.2 Vision Backbone Ablation

  • 0.548 for DINOv3 exceeds 0.512 for DINOv2 and 0.499 for ResNet-50 on the closed-loop HD-Score.The DINOv3 score matches the headline student; the alternatives use the same adapter design with different pretrained representations.Source: §B.2 · p. 28, Figure 5 · p. 28
  • The encoder swap accounts for the 0.049 gap between ResNet-50 and DINOv3, with teacher, data, and recipe held fixed.The authors note that the absolute gaps are modest.Source: §B.2 · p. 28
  • The authors relate the ordering to linear adapters: stronger pretrained representations enter alignment closer to the teacher’s feature geometry.The frozen head can use only the structure already carried by its input features.Source: §B.2 · p. 28

C NAVSIM Per-Subscore Breakdown

The NAVSIM per-subscore breakdowns unpack navtest PDMS and navhard EPDMS, including the latter’s two-stage scoring under ego-state perturbations. Baseline entries are published results, while the paper’s rows use the standard protocol.

C NAVSIM Per-Subscore Breakdown

  • Navtest PDMS aggregates no-collision, drivable-area compliance, time-to-collision, comfort, and ego-progress subscores.The table reports PDMS to one decimal; GigaPixel [49] is omitted because it reports navhard results only.Source: C NAVSIM Per-Subscore Breakdown · p. 28, Table 9 · p. 29
  • Navhard EPDMS aggregates nine subscores over two stages, with Stage 2 rescored under Gaussian-Splatting-based ego-state perturbations.The subscores include no-collision, drivable-area and direction compliance, traffic-light compliance, ego-progress, time-to-collision, lane-keeping, heading consistency, and ego-centering.Source: C NAVSIM Per-Subscore Breakdown · p. 28, Table 10 · p. 30
  • The breakdown tables distinguish published baseline values and privileged references from the authors’ rows measured under the standard protocol.Bold indicates the best result in each input group; S. denotes the per-stage EPDM score.Source: C NAVSIM Per-Subscore Breakdown · p. 28, Table 9 · p. 29, Table 10 · p. 30

Related papers

Loading 2610.30319v1…