PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies

Sajad Marvi, Julian Schmidt, Alessandro Canevaro, Julian Jordan, Abhinav Valada

arXiv:2610.19714v1cs.ROcs.AIcs.MA

TL;DR

Robust driving policies must generalize beyond the scenarios seen during training, while conditioned self-play can trade driving performance for behavioral diversity. UncArms uses epistemic uncertainty to guide probing agents toward neighbors’ weaknesses while neutral agents adapt. It reports stronger results than standard self-play on zero-shot transfer and Waymo validation, with broader hidden-state coverage.

  • Problem

    Driving policies may not generalize to novel domains because they reflect only scenarios seen during training, while conditioned self-play can degrade normal-mode driving performance when promoting behavioral diversity.

    Source: §I · p. 1

  • Method

    UncArms trains probing agents to increase neighbors’ epistemic uncertainty while neutral agents learn under that pressure, using uncertainty change as the probing reward.

    Source: §I · p. 1

  • Results

    UncArms outperforms standard self-play on zero-shot nuplan Val14 transfer (99.12% vs. 97.97%) and Argoverse 2 transfer (82.39% vs. 78.03%) without fine-tuning, and reduces Waymo validation failure rate by 17%.

    Source: §I · p. 2

  • Takeaways & Limitations

    Epistemic uncertainty provides an automatic signal for identifying and prioritizing challenging scenarios without ground-truth labels.

    Source: §V · p. 8

Abstract

from arXiv · show

Learning robust driving policies that generalize beyond their training domain remains a central challenge in motion planning. We propose an uncertainty-driven self-play scheme, in which half of the agents are rewarded for increasing their neighbors' epistemic uncertainty, while the other half learn to drive safely as the exploring agents continuously seek out their weaknesses. Epistemic uncertainty indicates where the policy is least confident and serves as an automatic proxy for scenario difficulty. Exploring agents use this signal to seek out hard cases, generating a challenging curriculum without manual scenario design. The resulting policy outperforms standard self-play, achieving best-in-class zero-shot transfer to nuplan Val14 (99.12% vs. 97.97%) and Argoverse 2 (82.39% vs. 78.03%) without fine-tuning, while also improving performance over standard self-play on Waymo. Our diversity-driven approach achieves 85% higher LSTM hidden-state coverage than the baseline, indicating a much richer set of behaviors. Further analysis shows that epistemic uncertainty serves as a calibrated difficulty detector, with highest-uncertain agents exhibiting 14.8 times the combined failure rate across collision, off-road, and DNF, of lowest-uncertain agents, enabling automatic prioritization of challenging scenarios without ground-truth labels.

I. INTRODUCTION

The paper asks whether conditioned self-play can produce diverse behavior without sacrificing normal-mode driving performance. UncArms uses epistemic uncertainty to generate challenging interactions, and reports stronger self-play results and zero-shot transfer than standard self-play.

I. INTRODUCTION

  • Conditioned self-play methods face a diversity tax: meaningful behavioral diversity typically degrades driving performance, even when agents are conditioned to drive normally.The paper asks whether a conditioned policy in normal mode can match or exceed an unconditional self-play baseline.Source: §I · p. 1
  • UncArms rewards probing agents for increasing neighbors’ epistemic uncertainty while neutral agents learn to drive safely under that pressure.The method uses uncertainty change as the probing reward and evaluates all agents in the neutral role.Source: §I · p. 1
  • Epistemic uncertainty serves as a calibrated difficulty detector: highest-uncertainty scenarios have 14.8× the combined collision, off-road, and DNF failure rate of lowest-uncertainty scenarios.The reported calibration error is ECE=0.014.Source: §I · p. 1
  • The study evaluates across 10,000 Waymo scenarios and transfers to nuplan Val14 and Argoverse 2, alongside comparisons with DIAYN, entropy-conditioned, and GIGAFLOW-style baselines.Ablations identify neighborhood size K=3, warmup scheduling, FiLM conditioning, and a 50% probing fraction as design choices; warmup improves score by 0.57%.Source: §I · p. 2
  • 99.12% vs. 97.97% on nuplan Val14 and 82.39% vs. 78.03% on Argoverse 2 show stronger zero-shot transfer than standard self-play without fine-tuning.The policy also reduces failure rate by 17% on Waymo validation scenarios compared to the baseline.Source: §I · p. 2

II. RELATED WORK

Prior approaches pursue traffic diversity, robustness, or uncertainty estimation through distinct mechanisms. UncArms combines a shared-role policy with uncertainty-based probing, using MC-Dropout estimates during training to drive an adaptive curriculum.

II. RELATED WORK

  • UncArms uses the learner’s own epistemic uncertainty in a shared policy to target uncertain situations, rather than relying on separate adversarial optimization or undirected diversity.Related approaches include adversarial training, domain randomization, RARL, SaFeR [20], IntSim [21], DIAYN, and maximum entropy RL.Source: §II · p. 2

III. METHOD

  • UncArms assigns agents probing or neutral roles in a shared FiLM-conditioned policy, with neutral agents adapting as probing agents escalate their challenge.The paper describes this co-evolution as an arms race, not a zero-sum conflict.Source: §III · p. 2

A. Problem Formulation

  • The driving task is formulated as a partially observable Markov game with 32 agents, partial observations, and 91 discrete acceleration-steering actions per agent.Observations include ego, nearby-partner, and road features, as well as the conditioning weight; the discount factor is γ=0.99.Source: A. Problem Formulation · p. 2

B. Epistemic Uncertainty via MC-Dropout

  • MC-Dropout estimates epistemic uncertainty from five stochastic forward passes; critic uncertainty uses value-estimate spread, while actor uncertainty uses mutual information.The actor term reflects uncertainty about which action to take, while the critic term reflects uncertainty about situation value; uncertainty is used during training rollouts, not evaluation.Source: B. Epistemic Uncertainty via MC-Dropout · pp. 2–3

C. FiLM-Conditioned Architecture

The architecture encodes ego, partner, and road observations before recurrent processing, then conditions the shared representation on the agent’s role. An auxiliary prediction objective encourages the representation to retain role information.

C. FiLM-Conditioned Architecture

  • Ego, partner, and road encoders produce a concatenated representation that passes through a shared embedding layer and LSTM before FiLM modulation.Max-pooling handles variable numbers of visible partners and road segments.Source: C. FiLM-Conditioned Architecture · p. 3
  • A linear projection of the scalar role signal produces FiLM parameters that modulate the LSTM representation.The model has about 780K parameters.Source: C. FiLM-Conditioned Architecture · p. 3
  • An auxiliary head predicts the conditioning input from the shared representation, with αMI=0.1 weighting the loss against the PPO objective.This maximizes a lower bound on I(wrole; h), encouraging the representation to distinguish probing and neutral roles.Source: C. FiLM-Conditioned Architecture · p. 3

D. Uncertainty-Driven Self-Play via Uncertainty Change

UncArms assigns agents probing or neutral roles: probing agents seek increases in neighbors’ epistemic uncertainty, while neutral agents optimize task reward. The uncertainty-change reward and staged training schedule create an automatic curriculum of increasingly challenging interactions.

D. Uncertainty-Driven Self-Play via Uncertainty Change

  • Probing agents receive an additional reward for increasing uncertainty in their K=3 nearest neighbors, while neutral agents receive only the standard task reward.At each episode, half the agents are assigned each role; neutral agents are evaluated at neutral conditioning.Source: D. Uncertainty-Driven Self-Play via… · pp. 3–4
  • As neutral agents adapt, probing agents must discover increasingly novel situations that raise neighbors’ uncertainty, producing a diversity-driven curriculum without hand-crafted scenarios.The reward targets temporal uncertainty change rather than absolute uncertainty, so probing agents must create new confusion instead of merely approaching already-confused agents.Source: D. Uncertainty-Driven Self-Play via… · p. 4
  • Training uses a three-phase schedule: driving-only warmup through 750M steps, a linear probing-reward ramp from 750M to 1B, then full probing intensity through 1.5B steps.Source: D. Uncertainty-Driven Self-Play via… · p. 4

IV. EXPERIMENTAL EVALUATION

The evaluation covers WOMD and zero-shot transfer to nuPlan and Argoverse 2, with results averaged across four seeds.

  • All methods are evaluated on WOMD, with zero-shot transfer tested on nuPlan and Argoverse 2; results report inter-seed mean±std across four seeds.Source: §IV · p. 4

A. Setup

Experiments train policies in PufferDrive on WOMD and evaluate them on held-out maps and transfer settings. The comparisons include an unconditional self-play baseline and several conditioned or diversity-oriented variants.

A. Setup

  • Training uses PufferDrive with 32 agents across 300 parallel WOMD scenarios, at more than 1M simulation steps per second.Source: A. Setup · p. 4
  • All methods train for 1.5B PPO steps; WOMD evaluation uses 10,000 held-out maps with all 32 agents policy-controlled, while transfer uses SDC-only control and neutral conditioning.Transfer background agents replay logged trajectories.Source: A. Setup · p. 4

B. Baselines and Variants

C. Main Results

UncArms beats the unconditional self-play baseline on WOMD and transfers more strongly to nuPlan and Argoverse 2. Its results also distinguish goal-directed performance from raw collision avoidance and compare favorably with diversity-oriented alternatives.

C. Main Results

  • 82.39% vs. 78.03% score: UncArms leads the baseline on Argoverse 2 zero-shot transfer without fine-tuning.The evaluation covers 24,988 validation scenarios in SDC-only control.Source: C. Main Results · p. 4, Table III · p. 5
  • 99.41% vs. 99.29% goal score: UncArms is the only conditioned method evaluated to surpass the unconditional baseline on WOMD.UncArms also has the lowest DNF rate at 0.20%; score requires reaching the goal without collision.Source: C. Main Results · p. 4, Table I · p. 5

1) nuPlan:

  • A 4.59% score gap versus the baseline marks the GIGAFLOW-style task-weight conditioning diversity tax, while UncArms exceeds DIAYN, entropy-conditioned PPO, and that variant on WOMD.The paper argues that variable task weights force representation of incompatible objectives, degrading driving competence.Source: C. Main Results · p. 4, 1) nuPlan · p. 5
  • 99.12% vs. 97.97% score: UncArms exceeds the baseline on zero-shot nuPlan Val14 transfer and is the only method to outperform it across all five reported metrics.In SDC-only mode, collision rates are elevated across methods because background agents replay fixed logged trajectories.Source: 1) nuPlan · pp. 4–5, Table II · p. 5
  • Entropy conditioning diversifies broadly and DIAYN optimizes discriminability, whereas uncertainty-seeking focuses probing on low-confidence interactions that may better align scenario diversity with policy gaps.Source: 1) nuPlan · p. 5

2) Argoverse 2:

On Argoverse 2, UncArms leads the baseline on score and goal rate, despite a caveat in off-road measurement. The paper also reports epistemic uncertainty as a calibrated predictor of driving failure.

2) Argoverse 2:

  • Argoverse 2 off-road rates are inflated because drivable-area polygon boundaries can flag legal turns as off-road.The dataset lacks explicit road-edge annotations; reported rates are 26–30%, versus <2% on nuPlan.Source: 2) Argoverse 2 · p. 5

E. Uncertainty as Difficulty Detector

  • 14.8× higher combined failure rate separates the highest- from lowest-uncertainty deciles (14.96% vs. 1.01%).The analysis bins 113K agent episodes by MC-Dropout uncertainty; collision, off-road, and DNF failures all scale with uncertainty.Source: Figure 3 · p. 5, E. Uncertainty as Difficulty Detector · p. 6

2) Argoverse 2:

  • 82.39% score and 93.05% goal rate outperform the baseline by +4.36% and +3.72%, respectively.UncArms is the only method reported to outperform the unconditional baseline on WOMD, nuPlan, and Argoverse 2.Source: 2) Argoverse 2 · p. 5

E. Uncertainty as Difficulty Detector

  • ECE=0.014 indicates that a logistic mapping from uncertainty to failure probability is well calibrated on held-out validation maps.The mapping is fitted on a random 1,000-map training subsample and evaluated on 10K held-out maps.Source: E. Uncertainty as Difficulty Detector · p. 6

F. Behavioral Diversity via LSTM Hidden States

UncArms uses a broader range of LSTM hidden states than the baseline, with similar total variance distributed more uniformly across its representation.

F. Behavioral Diversity via LSTM Hidden States

  • 35.7% hidden-state grid coverage versus 19.3% for the Baseline is an 85% increase.The grid is formed by projecting hidden states onto three principal components and dividing the space into 8,000 cells.Source: F. Behavioral Diversity via LSTM… · p. 6
  • UncArms uses 110 effective dimensions versus 94 for the Baseline, indicating a more uniform representation spread.Effective dimensionality counts the PCA components needed to explain 95% of variance.Source: F. Behavioral Diversity via LSTM… · p. 6
  • 33.25 vs. 34.33 spread shows similar total variance for the Baseline and UncArms, but UncArms distributes it more uniformly.Spread is the trace of the covariance matrix; the Baseline concentrates variance in dense clusters with empty gaps.Source: F. Behavioral Diversity via LSTM… · p. 6
  • The authors attribute superior zero-shot transfer to the richer internal set of behavioral modes developed through the diversity-driven curriculum.They relate this to exposure to more situations during training.Source: F. Behavioral Diversity via LSTM… · p. 6

1) MC-Dropout vs. Adversarial Training:

Ablations indicate that gains depend on adversarial training and on specific choices for neighborhood size, warmup, role conditioning, and probing fraction.

1) MC-Dropout vs. Adversarial Training:

  • A +0.91% score swing separates UncArms from the baseline with MC-Dropout alone, which reduces score by 0.79%.UncArms surpasses the clean baseline by 0.12%; the authors attribute the net swing to the arms-race mechanism, while MC-Dropout supplies its uncertainty signal.Source: 1) MC-Dropout vs. Adversarial Training · p. 6

2) Neighborhood Size and Warmup Schedule:

  • K=3 achieves the best score, while K=1 and K=10 degrade performance; removing the 750M-step warmup lowers score by 0.57%.The authors describe K=3 as balancing local interaction density and signal stability; warmup establishes a competent base policy before probing pressure begins.Source: 2) Neighborhood Size and Warmup Schedule · pp. 6–7

4) Probing Fraction:

  • 99.41% vs. 99.40% scores for the 50/50 and 75% neutral splits show that a minority of probing agents can suffice.Reducing neutral agents to 25% degrades performance by −0.37% versus UncArms, with variance ±0.53; the authors attribute this to a weakened task-reward anchor and destabilized training.Source: Table VIII · p. 7, 4) Probing Fraction · p. 7

3) Conditioning Mechanism:

  • Role conditioning is load-bearing: without role assignment score falls 0.11% below baseline, while concatenation and FiLM exceed it by +0.06% and +0.12%.FiLM adds a further +0.06% over concatenation, which the authors associate with feature-wise gating.Source: 3) Conditioning Mechanism · p. 7

H. Qualitative Results

UncArms’ uncertainty-driven self-play improves benchmark results and produces broader behaviors. Its qualitative results show both neutral and probing roles, while epistemic uncertainty also tracks scenario difficulty.

H. Qualitative Results

  • Across three WOMD scenes, neutral-evaluated UncArms negotiates a crossing, avoids a collision, and completes a route where the baseline fails.In the dense crossing, it yields and waits for other agents before proceeding to its goal.Source: H. Qualitative Results · p. 7
  • From one checkpoint, neutral conditioning yields conservative, goal-directed behavior while probing seeks uncertain interactions; neutral trajectories may still violate lane-keeping or right-of-way rules.The rule-compliance limitation follows from those constraints being absent from the reward definition.Source: H. Qualitative Results · p. 7, Figure 5 · p. 8

V. CONCLUSION

  • 99.41% vs. 99.29% on WOMD, 99.12% vs. 97.97% on zero-shot nuplan Val14, and 82.39% vs. 78.03% on Argoverse 2; LSTM hidden-state coverage rises 85%.The transfer results require no fine-tuning; the authors link broader hidden-state coverage to the transfer advantage.Source: §V · pp. 7–8
  • Epistemic uncertainty acts as a calibrated difficulty detector: highest-uncertainty scenarios have 14.8× the failure rate of the lowest decile (ECE=0.014), helping prioritize hard cases without labels.Source: §V · p. 8

Related papers

Loading 2610.19714v1…