PaperlayerView exact revision on arXiv

Source-linked AI summary

Fail2Drive: Benchmarking Closed-Loop Driving Generalization

Simon Gerstenecker, Andreas Geiger, Katrin Renz

arXiv:2604.08535v1cs.ROcs.CV

TL;DR

Closed-loop driving benchmarks often reuse training scenarios, leaving generalization to rare unseen situations insufficiently tested. Fail2Drive introduces paired shifted and in-distribution CARLA routes plus an extensible toolbox, and evaluations show consistent degradation and brittle failure modes across current models.

  • Problem

    Existing benchmarks rarely measure true closed-loop generalization because they often reuse training scenarios at test time.

  • Method

    Fail2Drive uses paired in-distribution and generalization routes with fixed context, novel scenarios, and an extensible CARLA toolbox to quantify sensitivity to distribution shifts.

  • Results

    Evaluations of seven recent models reveal consistent degradation under controlled shifts and failure modes including disregarded LiDAR information, obstacle collisions, and missing fallback behavior.

  • Takeaways & Limitations

    Fail2Drive provides a reproducible basis for exposing brittle driving behavior and studying closed-loop generalization under controlled distribution shifts.

  • Takeaways & Limitations

    Results are limited to controlled CARLA simulation and relative robustness under structural shifts, so robustness in CARLA does not establish real-world robustness.

Abstract

from arXiv · show

Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe and scalable testing, existing benchmarks rarely measure true generalization: they typically reuse training scenarios at test time. Success can therefore reflect memorization rather than robust driving behavior. We introduce Fail2Drive, the first paired-route benchmark for closed-loop generalization in CARLA, with 200 routes and 17 new scenario classes spanning appearance, layout, behavioral, and robustness shifts. Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into quantitative diagnostics. Evaluating multiple state-of-the-art models reveals consistent degradation, with an average success-rate drop of 22.8\%. Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the LiDAR and failing to learn the fundamental concepts of free and occupied space. To accelerate follow-up work, Fail2Drive includes an open-source toolbox for creating new scenarios and validating solvability via a privileged expert policy. Together, these components establish a reproducible foundation for benchmarking and improving closed-loop driving generalization. We open-source all code, data, and tools at https://github.com/autonomousvision/fail2drive .

1. Introduction

Existing closed-loop driving benchmarks do not adequately test generalization to rare unseen situations. Fail2Drive addresses this gap with paired shifted and in-distribution routes, a scenario-generation toolbox, and analysis revealing consistent brittle failures in current models.

  • Current driving models increasingly claim robustness, but whether they generalize to rare unseen situations remains unresolved.
  • CARLA enables safe, standardized testing, yet prominent benchmarks reuse limited scenarios and need stronger out-of-distribution evaluation.
  • Fail2Drive introduces paired routes that hold location and traffic conditions constant while varying only the targeted distribution shift.This design measures generalization drops and supports controlled analysis across perceptual and behavioral capabilities.
  • The benchmark adds a toolbox for designing new scenarios on top of CARLA with lower engineering overhead.
  • Analysis of seven recent models exposes consistent brittle behaviors, including disregarded LiDAR cues, obstacle collisions, and missing fallback behavior.The models also fail to learn a generalizable internal representation of obstacles.

2. Related Work

Related benchmarks cover perception, planning, or selected safety-critical interactions, but existing approaches leave important gaps in closed-loop, sensor-based evaluation under diverse long-tail distribution shifts.

  • nuScenes provides standardized perception benchmarks, while nuPlan evaluates closed-loop planning but lacks diverse long-tail events.
  • interPlan adds safety-critical interactions but evaluates planner outputs without sensor input.
  • Prior out-of-distribution driving studies examine appearance, occlusion, adversarial perturbations, or safety-critical scenarios.
  • Many existing generated scenarios remain simplistic, focus heavily on other actors, or overlap with basic CARLA scenarios.

3. Fail2Drive - Generalization Benchmark

Fail2Drive is a paired-route CARLA benchmark and extensible toolbox for measuring robustness to controlled visual, geometric, and behavioral shifts. It uses diverse held-out routes, 17 novel scenario classes, and route-pair comparisons to quantify generalization while preserving compatibility with existing driving stacks.

  • Benchmark principles: Fail2Drive extends CARLA 0.9.15 with visual, geometric, and behavioral shifts while remaining compatible with existing driving stacks.
  • Paired evaluation: Each generalization route is paired with an equivalent in-distribution route at the same location and traffic configuration, isolating the targeted shift.
  • Route design: The benchmark contains 200 short routes across Town13, spanning varied road widths, curvature, speed limits, and environments with minimal overlap with official validation routes.The routes average 219 meters, supporting attribution of failures to individual scenarios.
  • Generalization scenarios: Fail2Drive introduces 17 novel scenario classes that test visual appearance, object layouts, behavioral adaptation, and resistance to shortcut associations.
  • Metrics: Generalization gaps are computed from paired-route differences using Driving Score, Success Rate, and a harmonic mean of the two.Route Completion is omitted because the intentionally short routes are nearly always finishable.
  • Evaluation rules: The benchmark forbids training on its scenarios, routes, and assets while allowing external pretraining as part of the model prior.
  • Scope and sim-to-real interpretation: Fail2Drive supports controlled simulation analysis, but CARLA's pseudorealism limits direct transfer of its robustness findings to the real world.

4. Analysis of State-of-the-Art Models

Across seven closed-loop driving models, controlled distribution shifts expose substantial and recurring generalization failures. Models often rely on CARLA-specific visual, geometric, and behavioral cues rather than general concepts of obstacles, drivable space, and high-level driving behavior.

  • Generalization gap: Seven representative models show consistent degradation under controlled distribution shifts, with an average HM drop of 16.3%.The evaluated models include camera-based, multimodal, vision-language-action, and privileged planners.
  • Failure patterns: Across failures, models default to familiar CARLA patterns: TransFuser++ misses obstacles visible in LiDAR, while PlanT and SimLingo overreact to non-blocking construction assets or mailboxes.These behaviors support the conclusion that strong models still lack a generalizable notion of high-level driving behavior.
  • Behavior: Behavioral scenarios produce the largest category-level gap, averaging a -53.6% HM drop as models default to memorized lane-following.PlanT drops by -69.5% HM on fully blocked roads despite receiving ground-truth object positions, while TransFuser++ declines by -26.7%.
  • Behavior: SimLingo drops from 98.50 to 19.68 HM on pedestrians walking in the ego lane, with collisions in 87% of cases.Its language-action module often hallucinates a nonexistent car or cyclist to follow, while models commonly approach pedestrians too closely.
  • Visual - Lateral: Visual-lateral shifts cause an average -33.58% HM degradation, and models fail to avoid large unseen obstacles even when they are visible in RGB and LiDAR.CustomObstacles performance falls to 11.44 HM for HiP-AD, 11.73 HM for PlanT, 11.90 HM for SimLingo, and 22.78 HM for TransFuser++.
  • Visual - Longitudinal: Visual-longitudinal shifts produce a moderate average HM drop of 7.2%, indicating that models transfer some cues that an object ahead requires slowing.This category performs better than behavioral and visual-lateral shifts, although animals with compact or non-upright shapes are frequently misclassified or ignored.

5. Fail2Drive Toolbox

Fail2Drive extends CARLA with tools for constructing controlled distribution shifts, validating scenario solvability, and supporting privileged demonstrations.

  • The toolbox enables controlled scenario extension, new benchmarks, and construction of a diverse large-scale dataset.
  • The modified CARLA build adds new assets, including animated animals and visual obstacle families, to test generalization beyond standard CARLA appearances.
  • Parameterized scenarios let users vary one factor at a time while keeping route context fixed, enabling controlled counterfactuals.
  • PDMLite-F2D checks whether newly designed scenarios are solvable and provides privileged demonstrations and a reference policy for debugging.

6. Conclusion and Limitations

Fail2Drive exposes widespread overfitting and evaluation gaps under distribution shifts, while its conclusions remain bounded by simulation realism and the limits of testing long-tail scenarios.

  • Fail2Drive reveals widespread overfitting, shortcut learning, unexpected failure modes, and systematic gaps in current robustness evaluations.
  • The benchmark is limited to controlled closed-loop analysis in pseudorealistic CARLA simulations, so robustness there is not sufficient for real-world robustness.
  • Its paired design primarily measures relative robustness under controlled structural shifts rather than absolute realism.
  • Testing cannot fully resolve the long-tail scenario problem because rare scenarios can never be exhaustively covered.

A. Scenario description

The scenario description presents paired examples for each new scenario class, contrasting an in-distribution case with a new generalization sample.

  • Each new scenario class is illustrated with an in-distribution/generalization pair and a detailed description.
  • The top image is the in-distribution example, while the bottom image is the new generalization sample.

1. BadParking

The BadParking scenario tests spatial understanding by varying a parked vehicle’s orientation, location, and asset while retaining the blocked-lane context.

  • BadParking partially occludes the ego lane with a parked vehicle whose orientation, location, and asset can vary.
  • The standard ParkedObstacle scenario provides the in-distribution sample for comparison.

2. ConstructionPermutations

ConstructionPermutations probes whether driving models generalize beyond familiar assets, placements, appearances, and scenario structures. The scenarios separately test reacting to relevant hazards, ignoring irrelevant cues, and handling situations requiring waiting or changed behavior.

  • Unseen obstacles: Customizable obstacles test generalization to unseen objects and structures at arbitrary road locations and orientations.ParkedObstacle or ConstructionObstacle scenarios provide the in-distribution comparison.
  • Missing visual cues: Occluded stop signs and disabled brake lights test whether models rely on standard visual cues for traffic-sign detection, distance judgment, and deceleration.The corresponding unoccluded scenarios serve as in-distribution samples.
  • Actor variation: Animals, nonstandard yielding vehicles, and pedestrians emerging from behind objects test generalization across actor appearance, shape, identity, and visibility.The benchmark contrasts these cases with default pedestrian, emergency-vehicle, and CARLA-asset scenarios.
  • Irrelevant cues: Off-road and opposite-lane obstacles, deceptive images, and irrelevant roadside objects test whether models distinguish driving-relevant hazards from familiar but non-actionable cues.These scenarios require maintaining the route without reacting unnecessarily.
  • Fallback behavior: Pedestrian crowds, crossing pedestrians, road-blocking objects, and deceptive walls test fallback behavior, including stopping, waiting, evasive action, and resuming after 60 seconds.Several in-distribution routes omit the scenario, isolating the added behavioral demand.

B. Full results

Full results report harmonic scores for all evaluated models across Fail2Drive’s generalization categories. The accompanying analysis indicates that degradation and failure patterns can be examined category by category.

  • Table 2: Numerical results are provided for all models separately by generalization category in Table 2.The table reports harmonic scores across the categories.
Loading 2604.08535v1…