PaperlayerView exact revision on arXiv

Source-linked AI summary

TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations

Zikang Xiong, Weixin Li, Zhouchonghao Wu, Akshay Rangesh, Saarth Bonde, Grantland Hall, Chen Tang, Yihan Hu, Wei Zhan

arXiv:2606.17386v2cs.CVcs.AIcs.RO

TL;DR

End-to-end driving relies on costly expert logs, while vectorized self-play can learn naturalistic driving without demonstrations but does not produce an image-based policy. TerraTransfer trains a self-play teacher and aligns a vision policy to its action distributions and features using paired image–scene-state frames. The resulting raw-image policy comes within 0.03 aggregate HD-Score of its self-play teacher and outperforms imitation-trained end-to-end pipelines.

  • Problem

    End-to-end driving training depends on costly expert demonstrations and rendering, while self-play learns from vector state and has not itself yielded a deployable raw-sensor policy.

    Source: §1 · pp. 1–2

  • Method

    TerraTransfer trains a planning head by multi-agent self-play, then aligns an image-conditioned student to its frozen action distributions and feature structure without logged-trajectory supervision.

    Source: §1 · pp. 2–3

  • Results

    The raw-image policy comes within 0.03 aggregate HD-Score of its self-play teacher and outperforms imitation-trained end-to-end pipelines, edging out the strongest published baseline on aggregate.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The planning head can provide reusable alignment supervision for different vision frontends, alignment datasets, sensor stacks, and downstream specializations.

    Source: §1 · p. 2

    Training on bounding boxes and vehicle-kinematics models limits exposure to fine-grained visual cues, pedestrians and cyclists, and complex signalized-intersection right-of-way rules.

    Source: §5 · p. 8

Abstract

from arXiv · show

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all stages: collecting and labeling millions of driving frames is costly, and closed-loop RL on images is bottlenecked by the per-step cost of photorealistic rendering plus a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: millions of rollout steps per second, and a state distribution naturally rich in collisions, near-misses, and recoveries that no driving log contains. Our approach exploits this asymmetry by decoupling learning to drive from learning to see. We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone, through the action KL divergence and a batch-relational low-rank structural loss. The action target comes from the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.

1 Introduction

Conventional end-to-end driving depends on costly expert logs and, in some fine-tuning paths, photorealistic rendering; self-play offers efficient, long-tail-rich training but does not itself produce image-based policies. TerraTransfer separates learning to drive from learning to see, using self-play teacher supervision to train a vision policy without logged-trajectory targets.

1 Introduction

  • Conventional training requires costly expert demonstrations, curated rare-event-heavy data, or photorealistic rendering with a heavy vision backbone at each closed-loop RL step.Large-scale imitation pretraining also relies on hundreds of thousands of hours of multi-camera logs and 3D label annotation.Source: §1 · p. 1
  • Self-play in vectorized simulation can produce robust, naturalistic driving without expert demonstrations and naturally exposes policies to collisions, near-misses, and recoveries, but it consumes scene state rather than raw sensors.Source: §1 · p. 2
  • TerraTransfer first trains a planning head by multi-agent self-play, then freezes it and aligns an image-conditioned encoder using the teacher’s per-state action distribution and a feature-level structural loss.The head learns from its own behavior rather than human trajectories; the vision student replaces the teacher’s road and partner inputs.Source: §1 · p. 2
  • The planning head is trained in 96 hours on 16 A100 GPUs over 2.4B simulated kilometers, and its supervision can be reused for vision frontends and downstream specializations.The vision encoder is aligned on 1.83M paired frames in 10 hours on 8 A100 GPUs.Source: §1 · pp. 2–3
  • The proposed structural loss matches pairwise scene similarities induced by the teacher within its low-rank feature subspace, motivated by the teacher’s low-rank, redundant features.Source: §1 · p. 3
  • The resulting end-to-end policy matches or exceeds prior end-to-end methods on aggregate closed-loop HD-Score over photorealistic evaluation benchmarks.The alignment phase uses paired image–scene-state frames and the frozen teacher’s action distribution, without expert demonstrations or logged-trajectory labels.Source: §1 · pp. 2–3

2 Related Work

Prior end-to-end driving methods rely on logged human driving, while vectorized self-play can learn without demonstrations but has not been extended to image observations at comparable scales. TerraTransfer combines a self-play teacher with image-based policy alignment, differing from prior privileged-teacher methods in both its teacher and its relational loss.

2 Related Work

  • Imitation, supervised fine-tuning, and open-loop RL depend directly on logged human driving, while closed-loop RL depends on imitation initialization and logged scenes; vectorized self-play avoids demonstrations but consumes scene state, leaving an image-policy gap.Source: §2 · p. 3
  • Prior privileged teacher–student driving methods use scripted or privileged-state experts and train students through action cloning or hand-designed targets, whereas TerraTransfer uses a self-play teacher and relational structural alignment.Source: §2 · p. 3
  • The pipeline trains a shared-parameter vector policy with PPO, then freezes it as per-frame supervision while a vision student learns from images, ego kinematics, and navigation signals without logged-trajectory targets.The student uses a frozen DINOv3 backbone and two linear adapters; structural and action losses tie it to the teacher.Source: Figure 2 · p. 4

3 Method

The method first trains a shared driving policy through self-play, then teaches image-conditioned encoders to reproduce its scene features and actions. Vision alignment matches teacher-student scene relationships within a low-rank subspace and aligns their action distributions.

3.1 Self-Play Pretraining

  • Self-play uses one shared policy for all agents, pooling their transitions into one on-policy buffer as the multi-agent interaction distribution evolves with the policy.Source: §3.1 · p. 4
  • An ego MLP and DeepSets encoders process ego state, road elements, and nearby partners before a shared MLP and actor produce a discrete action distribution.Source: §3.1 · p. 4
  • Per-step reward is a weighted sum of driving-quality terms, with weights resampled per agent and episode to expose the policy to varied driver preferences.Source: §3.1 · pp. 4–5
  • The policy is trained with clipped PPO, then serves as a frozen teacher for per-frame supervision while its shared MLP and actor head are inherited by the student.Source: §3.1 · p. 5

3.2 Vision Alignment

  • Instead of reconstructing detections, alignment predicts the teacher’s pooled scene features, a fixed-dimensional target that avoids recovering inputs the policy does not directly consume.Source: §3.2 · p. 5
  • Image-conditioned road and partner encoders replace the teacher’s set encoders, while the student inherits its ego encoder, shared MLP, and actor head.Source: §3.2 · p. 5
  • Structural alignment matches pairwise scene relationships within the teacher’s principal subspace, focusing on informative low-rank variation rather than absolute coordinates or the low-variance tail.Source: §3.2 · pp. 5–6
  • Forward KL aligns student and teacher action distributions, adding a policy-level consistency signal to the structural feature loss.Source: §3.2 · p. 6

4 Experiments

The experiments evaluate whether a vision policy aligned to a self-play teacher can drive competitively without logged-trajectory supervision. The aligned policy leads on aggregate HD-Score, needs less paired data than the strongest published baseline, and uses a low-rank structural alignment target motivated by the teacher’s features.

4.1 Setup

  • HUGSim evaluates closed-loop aggregate HD-Score on nuScenes-derived scenarios across Easy, Medium, Hard, and Extreme tiers.Because the policy emits one immediate action at a time, time-to-collision and comfort use closed-loop definitions based on realized rollouts.Source: §4.1 · p. 6

4.2 Closed-Loop Comparison with End-to-End Baselines

  • 0.490 aggregate HD-Score makes the aligned vision policy strongest on HUGSim, 0.130 above LTF and 0.038 above the strongest published ECO variant.It is also close to the self-play teacher at 0.520.Source: Table 1 · p. 7, §4.2 · p. 7
  • ECO leads by 0.105 on Extreme, where out-of-distribution collisions and conservative responses reduce route completion and aggregate HD-Score.The low route-completion score Rc pulls down the aggregate score on this split.Source: §4.2 · p. 7

4.3 Alignment Data Efficiency

  • At ρ = 0.6, the student scores 0.461 versus ECO’s 0.452 using roughly 40% of ECO’s nuPlan frames, and reaches 0.490 with the full paired set.The 1.83M paired frames provide teacher action-distribution targets; logged ego trajectories are never used, and evaluation is on nuScenes-derived scenarios rather than nuPlan.Source: §4.3 · p. 7, Figure 3 · p. 8

4.4 Effective Rank of Latent Features

  • The teacher’s D = 64 features capture 80% of energy in 9 map and 13 partner directions, supporting the low-rank truncation used by the structural loss.Entropy-based effective ranks are about 15 for map and 18 for partner; the 80%-energy truncation is reported to help closed-loop performance versus a full-rank variant.Source: §4.4 · pp. 7–8, Figure 4 · p. 8

5 Limitations

The paper identifies three limitations tied to its simulated training environment: missing fine-grained visual cues, limited mixed-traffic exposure, and incomplete signalized-intersection support.

5 Limitations

  • Training solely on bounding boxes leaves the policy without fine-grained cues such as brake lights or hand signals.The authors plan supervised fine-tuning on real-world data to address this limitation.Source: §5 · p. 8
  • Vehicle-kinematics-only simulation limits exposure to mixed traffic involving pedestrians and cyclists.The authors expect broader supported entities in the simulator to mitigate this constraint.Source: §5 · p. 8
  • Incomplete signalized-intersection support means the system has yet to learn complex right-of-way rules.The authors also identify expanded traffic logic as a future mitigation.Source: §5 · p. 8

6 Conclusion

The conclusion presents a demonstration-free recipe that trains driving competence through self-play, then transfers it to a vision policy by alignment. The resulting policy approaches its teacher and outperforms imitation-trained end-to-end pipelines on photorealistic closed-loop scenarios.

6 Conclusion

  • The two-phase recipe trains a planning head by vector-state self-play, then aligns a frozen teacher to a vision frontend on paired image and scene-state frames.Alignment uses an action-distribution loss and a batch-relational low-rank structural loss, never logged trajectories.Source: §6 · p. 8
  • The raw-image policy comes within 0.03 aggregate HD-Score of its self-play teacher and edges out the strongest published baseline on photorealistic closed-loop scenarios.Source: §6 · p. 8
  • The cost of learning to drive moves upstream into self-play, which the conclusion characterizes as demonstration-free, scalable, one-time, and reusable.Source: §6 · p. 8

Supplementary Contents

The supplementary contents list appendices on the self-play policy, evaluation metrics and baselines, ablations, demonstration quality, and qualitative results.

Supplementary Contents

A.1 Training Details

The appendix details the self-play environment, policy inputs, rewards, and vehicle dynamics, then adapts HD-Score to evaluate realized closed-loop driving. The revised metric preserves native components where possible while redefining trajectory-dependent measures.

A.1 Training Details

  • The self-play simulator uses nuPlan map geometry but no logged trajectories or human actions; vehicles, goals, and initial scenes are procedurally generated.Controlled vehicles are spawned in collision-free, drivable locations, with reachable goals sampled along the map lane graph.Source: §A.1 · p. 14

A.2 Observations

  • At each 0.1-second step, the policy receives ego-state and goal information plus pooled observations of nearby road elements and scene partners.Road and partner sets are represented relative to the ego frame and encoded with DeepSets.Source: §A.2 · p. 14

A.3 Reward Components

  • The reward follows most of Cusumano-Towner et al. [18], including randomized per-agent, per-episode weights, and adds a penalty for excessive lane changes.Table 2 lists the training-run coefficient settings.Source: §A.3 · p. 14, Table 2 · p. 15

A.4 Vehicle Dynamics

  • The policy controls passenger cars with jerk-actuated bicycle dynamics and a discrete 12-action jerk grid; vehicle dimensions and dynamics are randomized at episode reset.The 0.1-second simulator step matches the vision policy’s control rate.Source: §A.4 · pp. 14–15

B Closed-Loop HD-Score

  • The closed-loop HD-Score evaluates realized driving, retaining the aggregation formula and NC and DAC while redefining TTC, COM, and route completion.The native score evaluates planned trajectories, whereas this policy emits one immediate action at a time.Source: B Closed-Loop HD-Score · p. 15, §B.1 · p. 15

B.1 Closed-loop redefinitions

  • Closed-loop TTC propagates the ego and valid partners from their observed velocities, then checks for collisions along a short forward horizon.The checks include partner bounding boxes and the reconstructed static 3D Gaussian-splatting scene background.Source: §B.1 · pp. 15–16
  • Closed-loop COM tests the realized ego trajectory against comfort bounds over a forward time window, while route completion uses distance along the scene route.Route completion is arc-length normalized and undiscounted, so a score of 1 means the route was completed.Source: §B.1 · p. 16

B.2 Native vs. closed-loop summary

  • TTC and COM are not directly comparable to their native counterparts per keyframe because they use realized rather than planned motion; TTC also changes from binary to continuous.NC and DAC remain comparable, and Table 3 summarizes the metric changes.Source: §B.2 · p. 16, Table 3 · p. 17

C Self-Play Policy Baseline Performance

The appendix assesses the privileged self-play teacher’s closed-loop driving across HUGSim difficulty tiers and the realism of its exported nuScenes rollouts. Its primary constraint is route completion on Extreme scenarios, while reported rollout realism is high overall.

C Self-Play Policy Baseline Performance

  • The teacher is evaluated on the 88-scenario HUGSim split by difficulty tier and on exported nuScenes traffic for behavioral realism.It uses privileged vectorized scene state on HUGSim; the WOSAC analysis covers two control settings.Source: Table 4 · p. 17, C Self-Play Policy Baseline Performance · p. 17, Table 5 · p. 18
  • On Extreme scenarios, route completion falls to 0.24 and drives HD-Score to 0.19 despite HD-PDMS of 0.73; safety and on-road compliance remain strong.NC stays above 0.95, DAC is 1.0, and TTC is above 0.83 across all four tiers; the passage associates the Extreme limitation with implausible initial configurations.Source: C Self-Play Policy Baseline Performance · p. 17
  • WOSAC realism is 0.89 in ego-only control and 0.86 in log-initialized multi-agent control, with kinematic likelihood the weakest axis at 0.48 and 0.51, respectively.Interaction and map-compliance likelihoods are near-perfect in both settings.Source: C Self-Play Policy Baseline Performance · pp. 17–18, Table 5 · p. 18

D.1 Loss Type Ablation

The ablations examine the alignment loss’s components and rank choice, then test whether the batch-derived low-rank target captures teacher geometry. The diagnostics support the selected batch size and energy-based ranks as a principled configuration.

D.1 Loss Type Ablation

  • The loss ablation compares action-only, structural-only, and combined alignment objectives, with the expected trade-off that each isolated term loses a property preserved by the other.The combined objective is reported to dominate both isolated terms, and performance is largely insensitive to λ, set to 0.5 by default.Source: §D.1 · p. 18, Table 6 · p. 18

D.2 Structural Loss Rank Ablation

  • The rank ablation compares the headline low-rank similarity loss with intermediate ranks and full-coordinate matching; full-coordinate matching matches or degrades closed-loop HD-Score.The stated rationale is that teacher features are sharply low-rank and redundant, so matching every direction can over-constrain the student on image-unrecoverable directions.Source: §D.2 · p. 18, Table 7 · p. 19

D.3 Low-Rank Subspace Diagnostics

  • At B = 1024, the batch basis tracks dominant teacher modes, with modal coverage dropping only beyond the effective rank.Coverage is measured against the reference teacher subspace; the figure depicts coverage across batch sizes and ranks.Source: Figure 5 · p. 19, §D.3 · p. 19
  • Perturbation-energy preservation depends primarily on rank and weakly on batch size; longitudinal ego shifts have the lowest preservation.The suppressed directions may still be recovered through the action loss; structural alignment sets a lower bound on explicitly enforced geometry.Source: §D.3 · pp. 19–20, Figure 6 · p. 20
  • The selected configuration uses B = 1024 and 80% cumulative-energy ranks kp = 13 and kr = 9, which the diagnostics place before the coverage cutoff while retaining geometric separation.The authors characterize the batch-wise low-rank target as principled rather than arbitrary.Source: §D.3 · p. 20

E Vision Backbone Ablation

The appendix compares image backbones under a fixed alignment recipe and tests whether alignment depends on expert-quality collection trajectories. Random-action observations produce a broadly similar outcome, with collection policy affecting state coverage rather than the supervision target.

E Vision Backbone Ablation

  • The vision-backbone ablation compares DINOv3 with DINOv2 and ResNet-50 under an identical alignment recipe, measuring closed-loop HD-Score and converged alignment losses.The passage identifies Table 8 as the comparison but does not report its cell values.Source: E Vision Backbone Ablation · p. 20, Table 8 · p. 20

F Alignment Decoupled from Demonstration Quality

  • The alignment target is the teacher’s action distribution for each reconstructed scene state, regardless of which policy visited that state.The collection policy changes the distribution of observed states, not the supervision target at those states.Source: F Alignment Decoupled from… · p. 21
  • Randomly sampled HUGSim data yields HD-Score 0.445 versus 0.490 for demonstration-collected data, a broadly similar alignment outcome.The random-action alignment dataset matches the full training dataset in size.Source: Table 9 · p. 21, F Alignment Decoupled from… · p. 21

G Qualitative Results

HUGSim rollouts illustrate the policy handling interactive driving situations, from uncertain cut-ins and time-pressured overtaking to route-aware disengagement and anticipatory slowing. Extreme-tier visual implausibility limits the achievable HD-Score independently of policy quality.

G Qualitative Results

  • Extreme-tier HUGSim scenarios can be visually implausible, imposing a ceiling on achievable HD-Score independent of policy quality.Examples include implausible initial configurations and occluded scenes where cars pass through each other.Source: G Qualitative Results · p. 22, Figure 12 · p. 23
  • During cut-in negotiation, the policy waits for the other vehicle’s intent to resolve, then accelerates and overtakes smoothly.Its speed briefly decreases before a sustained ramp-up.Source: Figure 8 · p. 22, G Qualitative Results · p. 22
  • With oncoming traffic approaching, the policy overtakes a static obstruction and merges back before the oncoming car arrives.The speed profile rises throughout the maneuver.Source: Figure 9 · p. 22, G Qualitative Results · p. 22
  • When a lead vehicle diverges from the ego’s route, the policy turns onto its own path and accelerates.It follows at steady speed until the lead is no longer relevant.Source: G Qualitative Results · p. 22, Figure 10 · p. 23
  • In a narrow lane, the policy slows to pass an oncoming vehicle, then decelerates again for a vulnerable road user ahead.Its speed remains reduced through the pass and does not fully recover.Source: G Qualitative Results · p. 22, Figure 11 · p. 23

Related papers

Loading 2606.17386v2…