PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Self-Play for End-to-End Driving

Luke Rowe, Roger Girgis, Rodrigue de Schaetzen, Daphne Cornelisse, Alaap Grandhi, Felix Heide, Eugene Vinitsky, Christopher Pal, Liam Paull

arXiv:2606.19641v2cs.ROcs.CV

TL;DR

Offline behavior cloning provides limited state coverage and no closed-loop interaction during training. The paper combines pixel-based self-play simulation with self-play DAgger and perception adaptation, reporting strong benchmark performance without human trajectory supervision.

  • Problem

    Offline behavior cloning provides limited state coverage and lacks closed-loop interaction, while prior self-play approaches typically use vectorized observations incompatible with end-to-end policies acting from sensor inputs.

    Source: §1 · pp. 1–2

  • Method

    The paper trains pixel-based policies through self-play DAgger in Gigapixel and transfers them to real sensor data through lightweight perception adaptation.

    Source: §1 · p. 2, §3.2 · p. 5, §5 · p. 10

  • Results

    The approach achieves state-of-the-art HUGSIM and competitive NAVSIM-v2 performance without human trajectory supervision, with proportional gains as training experience increases.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The authors conclude that self-play is a viable, scalable alternative to large-scale behavior cloning for end-to-end driving models.

    Source: §1 · p. 2, §5 · p. 10

    Gigapixel’s abstract scene representation omits cues such as debris, unusual obstacles, weather, and lighting, and real-world adaptation requires paired data.

    Source: §5 · p. 10

Abstract

from arXiv · show

End-to-end autonomous driving models are typically trained on offline human-demonstration datasets that provide limited state coverage and often no closed-loop feedback, making them prone to compounding errors when deployed in closed-loop and brittle to long-tail agent interactions. To overcome these limitations, we propose an alternative strategy for training end-to-end driving models: large-scale self-play directly from pixels in simulation. While prior self-play approaches have shown promising transfer to real-world driving, they typically assume vectorized Bird's-Eye-View (BEV) observations that are incompatible with end-to-end policies operating directly on sensor observations. To this end, we introduce Gigapixel, a high-throughput batched driving simulator with perspective rendering, enabling scalable self-play directly from pixel observations. Rather than targeting compute-costly photorealistic sensor simulation, Gigapixel renders a simplified bounding-box world that preserves essential scene structure while achieving throughput at 50k agent steps per second. Since direct pixel-space self-play RL is prohibitively sample-inefficient at end-to-end model scale, we propose self-play DAgger training: we train pixel-based policies in self-play via on-policy distillation from a privileged RL teacher. To bridge the sim-to-real gap, we subsequently transfer the self-play trained policies to real-world sensor data through lightweight perception adaptation. Policies trained in Gigapixel and adapted to real-world sensor data achieve competitive performance on the HUGSIM and NAVSIM-v2 benchmarks without human trajectory supervision. Moreover, scaling self-play training yields proportional gains in policy performance, establishing self-play as a practical and scalable strategy for training end-to-end models.

1 Introduction

Offline behavior cloning leaves end-to-end driving policies with limited state coverage and no closed-loop interaction during training. The paper introduces pixel-based self-play simulation and a DAgger-based training approach, reporting strong real-world benchmark performance without human trajectory supervision.

1 Introduction

  • Behavior cloning has limited state coverage and lacks closed-loop feedback, leaving policies vulnerable to brittle behavior and compounding errors at deployment.The passage states that these structural limitations are not removed simply by collecting more data.Source: §1 · p. 1
  • Self-play generates closed-loop experience whose coverage scales with compute, unlike fixed offline datasets.Closed-loop interaction lets agents learn the consequences of their actions, while targeted experience generation requires a high-throughput simulator.Source: §1 · pp. 1–2
  • 50k agent steps per second on one GPU enables Gigapixel to support self-play from pixel observations.It uses perspective views of a simplified bounding-box world that preserves essential scene geometry and interaction fidelity.Source: §1 · p. 2
  • Self-play DAgger offers a more sample-efficient route than direct pixel-space RL for training end-to-end policies.A privileged RL teacher trained on vectorized observations is distilled into a pixel-based student in self-play; real-sensor transfer adapts the perception module.Source: §1 · p. 2
  • The approach achieves state-of-the-art HUGSIM and competitive NAVSIM-v2 performance without human trajectory supervision, with proportional gains as training experience increases.The reported transfer to real-world sensor data uses lightweight perception adaptation.Source: §1 · p. 2
  • The authors present self-play as a scalable, practical alternative to offline behavior cloning for end-to-end driving.They describe synthetic experience as a path to continual improvement.Source: §1 · p. 2

2 Related work

Prior work spans fast abstract simulators and slower high-fidelity simulation, while much closed-loop training remains single-agent or uses vectorized observations. This paper positions scalable, multi-agent self-play with pixel observations as a distinct approach.

2 Related work

  • Prior closed-loop driving methods use single-agent settings or slow simulators, limiting behavioral diversity or scalability.The paper characterizes its approach as multi-agent, closed-loop DAgger training in a fast simulator.Source: §2 · p. 3
  • Driving simulators trade throughput against representational fidelity, and Gigapixel is positioned between vectorized BEV and photorealistic simulation.RAP also renders simplified bounding-box worlds, but the passage says it is slower and uses them for data augmentation rather than closed-loop training.Source: §2 · p. 3
  • Earlier driving self-play work largely used vectorized observations; this paper extends self-play to pixels and uses DAgger rather than RL for sample efficiency.The passage describes self-play RL as data-hungry, often requiring billions of experience steps.Source: §2 · p. 3

3 Self-Play for End-to-End Driving

The section presents a three-stage approach: train a vectorized teacher with self-play RL, distill it into a pixel-based student in Gigapixel, then adapt perception to real images. Gigapixel supplies pixel observations, and self-play DAgger trains the student on states induced by joint student behavior.

3.1 Gigapixel

  • Gigapixel renders abstract simulator states as ego-centric pixel views, preserving essential planning information while supporting scalable self-play without photorealistic sensor simulation.It extends PufferDrive with batched GPU rendering using rasterization or ray tracing and simple scene primitives.Source: §3.1 · pp. 4–5

3.2 Self-play DAgger

  • Self-play DAgger confines RL to a cheap vectorized-observation teacher, then distills its closed-loop behavior into the compute-intensive pixel-based student using trajectory targets.Parallel teacher rollouts produce trajectory targets directly, avoiding the mismatch between student trajectory outputs and typical RL control outputs.Source: §3.2 · p. 5
  • Unlike vanilla DAgger, self-play DAgger lets the student control every agent, so training states reflect coevolving interactions and every agent contributes data per joint rollout.The resulting scenarios can surface multi-agent interactions rare in human driving logs, while multiplying training experience per simulator step.Source: §3.2 · p. 5
  • The teacher is a compact decentralized policy trained with PPO on ego-centric vectorized observations and a weighted multi-objective reward.The reward combines individual terms such as collision, off-road, and comfort.Source: §3.2 · p. 5

3.3 Sim-to-real Perception Adaptation

  • Sim-to-real adaptation freezes the planning head and updates only the perception backbone using paired real images and corresponding Gigapixel renderings.Planning and perceptual losses supervise the real-image policy and align its features with the frozen simulation reference, without human trajectory supervision.Source: §3.3 · pp. 6–7
  • Gigapixel models perform competitively across NAVSIM-v2 Stage 1 metrics and improve Stage 2, the closest proxy for closed-loop execution robustness.The caption states this evaluation is conducted without human trajectory supervision.Source: Table 2 · p. 7

4 Experiments

Experiments evaluate self-play DAgger policies in Gigapixel and on HUGSIM and NAVSIM-v2. The policies outperform behavior-cloned counterparts on reported comparisons, and increasing self-play experience yields consistent gains.

Evaluation setup

  • Policies are trained on 335k nuPlan scenarios; Gigapixel evaluation uses 1,000 held-out scenarios, and students are adapted to real NAVSIM camera images using paired simulated-real observations.The evaluation also covers closed-loop HUGSIM scenes and pseudo-closed-loop NAVSIM-v2 navhard; surrounding actors in Gigapixel are log-replayed.Source: §4 · pp. 7–8

Results

  • Self-play DAgger surpasses a Gigapixel Driving Score of 60 in roughly 3000× fewer steps than direct pixel-based self-play RL.Figure 3 plots held-out Gigapixel Driving Score against agent steps for the two pixel-based methods, with vectorized teacher performance marked at 25B steps.Source: Figure 3 · p. 7, §4 · p. 8
  • Gigapixel-DrivoR achieves HUGSIM state-of-the-art average RC (50.1) and HD-Score (38.5); its Extreme-tier HD-Score trails DrivoR, 21.6 vs. 32.5.Scoring-based Gigapixel-DrivoR improves over DrivoR by 2.8 HD-Score points (38.5 vs. 35.7); Gigapixel-DrivoR-Reg improves over DrivoR-Reg by 12.5 points (33.2 vs. 20.7).Source: §4 · p. 8
  • On NAVSIM-v2, Gigapixel-DrivoR and -Reg improve EPDMS over BC (50.1 vs. 48.3; 29.5 vs. 25.5), while SimScale leads overall at 54.7.On Stage 2, Gigapixel-DrivoR-Reg scores 45.5 vs. 38.4 for DrivoR-Reg, and Gigapixel-DrivoR scores 63.5 vs. 59.4 for DrivoR.Source: §4 · p. 9
  • Beyond 10M steps, self-play DAgger outperforms BC and single-agent DAgger; BC plateaus around 100M as students do not see their own rollout states.After adaptation, self-play DAgger gains 3.1 HD-Score over single-agent DAgger and 14.6 over BC; removing perceptual loss drops HD-Score from 33.2 to 18.5.Source: Figure 4 · p. 8, Table 3 · p. 8, §4 · p. 9
  • Near a stopped lead vehicle, self-play DrivoR slows and passes safely, while BC DrivoR maintains a straight-ahead plan and rear-ends it.The authors attribute the BC failure to rare safety-critical stop states being underrepresented in training logs.Source: §4 · p. 9, Figure 5 · p. 9
  • From a road-edge recovery state, self-play DrivoR corrects toward the lane, whereas BC DrivoR drifts off-road.Corrective states are frequently visited during self-play but rarely appear in offline expert logs.Source: §4 · p. 10, Figure 6 · p. 10

5 Conclusion, Limitations, and Future Work

The paper presents Gigapixel, self-play DAgger, and lightweight sim-to-real adaptation as a method for training end-to-end driving policies, reporting state-of-the-art HUGSIM and strong NAVSIM-v2 performance. It identifies limitations in scene representation, teacher–student observations, paired-data requirements, and scenario initialization.

Conclusion

  • The approach combines high-throughput pixel simulation, self-play DAgger distillation from a privileged RL teacher, and lightweight sim-to-real adaptation, achieving state-of-the-art HUGSIM and strong NAVSIM-v2 performance.The authors present self-play as a viable alternative to large-scale behavior cloning for end-to-end driving.Source: §5 · p. 10

Limitations

  • Gigapixel’s abstract boxes, lane polylines, and traffic-light states omit cues such as debris, unusual obstacles, weather, and lighting.These omitted scene details bound what the simulator represents.Source: §5 · p. 10
  • The privileged vectorized teacher may observe information that the pixel-based student cannot recover under occlusion or limited visibility.This teacher–student asymmetry is identified as a limitation of distillation.Source: §5 · p. 10
  • Sim-to-real adaptation requires paired data, which the authors describe as practical on NAVSIM but less direct when only raw sensor logs are available.Source: §5 · p. 10

Future Work

  • The current scenario initialization from nuPlan logs constrains the distribution of initial scenes; generative approaches are proposed to synthesize rare or informative interactions.Source: §5 · p. 10
  • The authors suggest combining human data with self-play, including through KL regularization or post-training toward human-like behavior.They forgo human trajectory supervision in this work to demonstrate self-play alone.Source: §5 · p. 10
  • Direct pixel-based RL in Gigapixel with native trajectory outputs remains an open challenge that could eliminate the distillation approach’s teacher–student asymmetry.Source: §5 · p. 10

A Gigapixel Simulator

Gigapixel extends a batched simulator with GPU perspective rendering, enabling pixel observations across many parallel worlds. Its simplified scene, configurable rendering, observation inputs, controls, and rewards support scalable driving self-play.

High-Throughput Batched Rendering

  • Thousands of independent worlds render together in a single batched GPU launch, with scene elements represented through an entity-component-system abstraction.Gigapixel integrates Madrona rendering with PufferDrive and supports rasterized and ray-traced rendering.Source: A Gigapixel Simulator · p. 19

Scene Representation

  • The simulator uses cuboids for vehicles and static objects, planes for roads, and spheres for traffic lights, with minimal shading and per-instance color.Each ego agent receives an ego-centric rendered view from its own world.Source: A Gigapixel Simulator · p. 19

Rendering

  • Ray tracing trades throughput for higher visual fidelity, while rasterization is faster; images remain GPU tensors, and rendering scales near-linearly across GPUs.Rendering resolution and camera configuration can be adjusted to trade visual fidelity against rendering budget.Source: A Gigapixel Simulator · p. 19, Figure 7 · p. 20

Observations

  • Each controlled agent receives ego-centric ego state and a navigation command, together with either rendered pixels or concatenated vectorized features.Continuous ego quantities are normalized to [−1, 1].Source: A Gigapixel Simulator · p. 19
  • The navigation command is derived from the logged path using a 20 m look-ahead and lateral-offset thresholds, without exposing the dense expert path to the policy.Commands are LEFT, STRAIGHT, RIGHT, or UNKNOWN and are available in self-play and deployment.Source: A Gigapixel Simulator · p. 19
  • Vectorized observations encode nearby agents, road segments, and traffic controls; when enabled, reward weights are appended to the ego state.The nearest K agents within 100 m are included in the agent features.Source: A Gigapixel Simulator · p. 21

Action Space and Dynamics

  • Policies can emit discrete jerk actions for a kinematic bicycle model or predict future trajectories tracked by decoupled LQR controllers in a receding-horizon manner.Trajectory waypoints are interpolated to the simulation step before tracking; episodes run up to 20 seconds in scenarios extracted from nuPlan logs.Source: A Gigapixel Simulator · p. 21

Reward Functions and Reward Conditioning

  • The per-step reward combines safety, rule compliance, progress, and comfort terms, including collision and off-road penalties, goal progress, and penalties for harsh acceleration and jerk.Reward coefficients can be randomized per agent and exposed as conditioning, with a fixed careful-driver setting used at inference.Source: A Gigapixel Simulator · p. 21

B.1 Vectorized Teacher Training

The vectorized teacher is trained through decentralized PPO self-play with randomized reward conditioning and a high-level navigation command. Its compact policy architecture aggregates object sets into a permutation-invariant scene summary, while advantage prioritization emphasizes informative transitions.

Teacher Training

  • The teacher is trained with decentralized PPO self-play, randomized per-agent reward weights, and LEFT/STRAIGHT/RIGHT/UNKNOWN navigation commands; training takes 24 hours on eight NVIDIA H200s.Source: §B.1 · p. 22

Policy Architecture

  • A compact approximately 2.7M-parameter policy embeds ego state and variable-length agent, road-graph, and traffic-light sets separately, then max-pools object sets before shared actor and value heads.The actor outputs factorized longitudinal and lateral jerk actions.Source: §B.1 · p. 22

Advantage Prioritization

  • Minibatch segments are sampled with probability proportional to estimated advantage magnitude raised to α, upsampling high-advantage events over uniform sampling.The paper characterizes this as a soft variant of advantage filtering.Source: §B.1 · pp. 22–23

B.2 Pixel-Based Student Training

The pixel-based student learns through self-play DAgger, receiving on-policy teacher supervision on states it visits while observing rendered images, minimal ego state, and navigation commands. Training uses mixed teacher–student rollouts, curated supervision, and staged image resolutions.

Student Distillation

  • A DrivoR pixel-based student is distilled from the vectorized teacher in self-play DAgger, observing rendered images plus minimal ego state and navigation command while the teacher supplies supervision from student-visited states.The full multimodal variant uses winner-takes-all trajectory loss, while DrivoR-Reg uses L1 trajectory regression.Source: §B.2 · p. 23

Trajectory Execution and Supervision

  • Teacher trajectory targets are rolled out over four seconds at 10 Hz; the LQR controller executes trajectories only for the self-driving car, while other agents use the student’s parallel jerk-action head.The controller is limited to the self-driving car because its kinematic-bicycle parameters are calibrated to that vehicle’s geometry.Source: §B.2 · p. 23

Mixed-Policy Rollouts

  • The executed action is sampled from the teacher with probability β and from the student otherwise; β anneals from 1 to 0 over the first 125M steps.The schedule then holds β at zero, shifting rollouts to the student’s own state distribution.Source: §B.2 · p. 24

Data Curation

  • Supervision upweights left- and right-turn commands 5× relative to straight driving and filters rollouts containing infractions, prior-infracting agents, or truncated waypoint targets.Source: §B.2 · p. 24

Rendering and Scoring

  • Ray-traced training uses front, front-left, front-right, and rear views, increasing resolution from 210 × 126 for the first 125M steps to 434 × 252 for the final 25M.The scoring head is trained separately in open loop on navtrain, with the other modules frozen, because it requires nuPlan PDMS targets.Source: §B.2 · p. 25

B.3 Sim-to-real Adaptation Training

The pixel student is adapted from synthetic Gigapixel images to real camera frames by fine-tuning its perception stack under distillation, using the Gigapixel-trained student as supervision. The section also specifies mode-aligned trajectory supervision and the loss components used for adaptation.

B.3 Sim-to-real Adaptation Training

  • Real-frame adaptation fine-tunes the perception stack under distillation, with the Gigapixel-trained student supplying supervision targets while preserving self-play behavior.NAVSIM-v2 uses 10 epochs on 85k navtrain samples; HUGSIM uses 20 epochs on 103k navtrain plus navval samples.Source: §B.3 · p. 25
  • Mode-aligned distillation regresses each student trajectory proposal toward the teacher proposal with the same index, rather than using winner-takes-all assignment.The teacher and student each produce P = 64 proposals; the usual DrivoR loss instead selects the proposal closest to a single ground-truth target.Source: §B.3 · p. 25
  • The full distillation loss combines decoder-register-token feature matching with planning and scoring losses.For DrivoR-Reg, P = 1 and no scoring loss is applied; its planning loss becomes single-mode L1 regression toward the teacher trajectory.Source: §B.3 · p. 25

C Benchmark Details

The evaluation spans abstract rendered closed-loop driving, photorealistic closed-loop driving in reconstructed scenes, and pseudo-closed-loop planning on real logs. Each benchmark uses a distinct evaluation setting and score.

C Benchmark Details

  • The three benchmarks cover closed-loop driving on Gigapixel pixels, closed-loop driving on reconstructed real scenes, and pseudo-closed-loop planning on real logs.Gigapixel uses held-out scenarios with surrounding actors replaying logged trajectories, isolating performance on abstract pixel observations from sim-to-real adaptation.Source: C Benchmark Details · p. 25
  • HUGSIM evaluates closed-loop driving in a 3D Gaussian-splatting reconstruction of real-world logs and reports HD-Score across four difficulty tiers.The benchmark aggregates nuScenes, Waymo, PandaSet, and KITTI, and the evaluation uses 345 scenarios.Source: C Benchmark Details · p. 26
  • NAVSIM-v2 evaluates planned trajectories non-reactively against rule-based traffic and reports EPDMS on the official navhard split.Its two score stages use real logged poses and synthetically perturbed poses, respectively.Source: C Benchmark Details · p. 26

C.1 Driving-Score Metrics

HUGSIM HD-Score combines per-step safety and driving-quality subscores with route completion, while NAVSIM-v2 EPDMS combines scores from logged and perturbed poses. Both metrics use multiplicative penalty terms that can zero scores when violations occur.

C.1 Driving-Score Metrics

  • The section provides high-level score descriptions and directs readers to the benchmark papers for full subscore definitions, weights, and formulas.HUGSIM and NAVSIM-v2 both combine gating penalties with driving-quality terms, but EPDMS additionally combines its two evaluation-stage scores multiplicatively.Source: §C.1 · p. 26
  • HUGSIM HD-Score multiplies per-step no-collision and drivable-area compliance by a weighted average of time-to-collision and comfort, then scales the episode average by route completion.A collision or off-road event drives that step's score to zero.Source: §C.1 · p. 26
  • NAVSIM-v2 EPDMS combines Stage 1 evaluation on real logged poses with Stage 2 evaluation on synthetically perturbed poses to measure recovery robustness.Penalty terms include no-collision, drivable-area, driving-direction, and traffic-light compliance; any penalty violation zeroes the score.Source: §C.1 · p. 26
Loading 2606.19641v2…