PaperlayerView exact revision on arXiv

Source-linked AI summary

Driving in Corner Case: A Real-World Adversarial Closed-Loop Evaluation Platform for End-to-End Autonomous Driving

Jiaheng Geng, Jiatong Du, Xinyu Zhang, Ye Li, Panqu Wang, Yanjun Huang

arXiv:2512.16055v2cs.CVcs.RO

TL;DR

Safety-critical corner cases are difficult to collect in the real world, and existing adversarial evaluation has been little explored for real-world end-to-end driving. The paper combines flow-matching image generation with adversarial traffic in a closed-loop platform; tests report lower scores and completion rates under adversarial traffic.

  • Problem

    Real-world safety-critical corner cases are difficult to collect, while existing adversarial evaluation methods are built for simplified simulations.

    Source: §I · p. 1

  • Method

    The platform combines a flow-matching real-world image generator with adversarial traffic to evaluate end-to-end models in closed-loop corner cases.

    Source: §I · p. 2

  • Results

    With adversarial surrounding vehicles, PDMS and DS dropped 30%–50%, while SC and RC also decreased.

    Source: C. Adversarial Closed-loop Experiments · p. 6

  • Takeaways & Limitations

    The platform provides a tool for evaluating end-to-end autonomous driving in corner cases.

    Source: §IV · p. 8

Abstract

from arXiv · show

Safety-critical corner cases, difficult to collect in the real world, are crucial for evaluating end-to-end autonomous driving. Adversarial interaction is an effective method to generate such safety-critical corner cases. While existing adversarial evaluation methods are built for models operating in simplified simulation environments, adversarial evaluation for real-world end-to-end autonomous driving has been little explored. To address this challenge, we propose a closed-loop evaluation platform for end-to-end autonomous driving, which can generate adversarial interactions in real-world scenes. In our platform, the real-world image generator cooperates with an adversarial traffic policy to evaluate various end-to-end models trained on real-world data. The generator, based on flow matching, efficiently and stably generates real-world images according to the traffic environment information. The efficient adversarial surrounding vehicle policy is designed to model challenging interactions and create corner cases that current autonomous driving systems struggle to handle. Experimental results demonstrate that the platform can generate realistic driving images efficiently. Through evaluating the end-to-end models such as UniAD and VAD, we demonstrate that based on the adversarial policy, our platform evaluates the performance degradation of the tested model in corner cases. This result indicates that this platform can effectively detect the model's potential issues, which will facilitate the safety and robustness of end-to-end autonomous driving.

I. INTRODUCTION

Safety-critical corner cases matter for evaluating autonomous-driving models but are difficult to collect, and simulator-generated cases can have a visual sim-to-real gap. The paper proposes a real-world adversarial closed-loop platform combining traffic interaction with flow-matching image generation.

I. INTRODUCTION

  • Safety-critical corner cases are important for evaluating model capability but are difficult to collect, and simulator-generated cases can suffer from a visual sim-to-real gap.The paper identifies a need for adversarial closed-loop evaluation in real-world scenes.Source: §I · p. 1
  • The platform’s experiments show that adversarial traffic sharply reduces tested models’ scores and completion rates.The introduction reports this outcome for end-to-end models evaluated in adversarial closed-loop scenarios.Source: §I · p. 2
  • The platform evaluates various end-to-end models in real-world adversarial scenarios, with a flow-matching image generator and adversarial traffic flow.The generator produces high-quality real-world images in a few denoising steps, while adversarial scores select surrounding-vehicle trajectories.Source: §I · p. 2
  • The adversarial traffic flow selects challenging surrounding-vehicle trajectories using adversarial scores, and the selection mechanism applies across tested models and traffic scenarios.The contribution description says the method constructs safety-critical interactions from multimodal predicted trajectories.Source: §I · p. 2

II. METHODOLOGY

The platform closes a loop between adversarial traffic, generated real-world images, and an end-to-end model. Traffic conditions guide image generation, and the model’s planned trajectory returns to the traffic simulator.

II. METHODOLOGY

  • The methodology presents a closed-loop platform for evaluating end-to-end driving in real-world adversarial scenarios.Its described components are adversarial traffic flow, a real-world image generator, and the tested model.Source: §II · p. 2, A. Overall Architecture · p. 3

A. Overall Architecture

  • The simulator supplies traffic conditions to the flow-matching generator, which creates images conditioned on the ego’s current traffic information.The generated images are formatted for the tested model’s input.Source: A. Overall Architecture · p. 3
  • The tested model plans an ego trajectory from generated images, and the traffic flow executes that trajectory to complete the evaluation loop.Source: A. Overall Architecture · p. 3

B. Adversarial Traffic Flow

Adversarial traffic uses a two-episode design to adapt surrounding-vehicle trajectories to each tested model. A multiplicative score balances predictive plausibility, collision likelihood, and motion smoothness.

1) Two Episodes Architecture:

  • The two-episode design records the tested model’s ego trajectory during replay, then executes a selected adversarial surrounding-vehicle trajectory on the same traffic segment.The trajectory is selected from multimodal predictions using the recorded ego trajectory, allowing it to adapt to different model preferences.Source: 1) Two Episodes Architecture · p. 3

2) Adversarial Score:

  • The multiplicative adversarial score jointly accounts for predictive plausibility, likelihood of collision with the ego, and motion smoothness.The score uses the prediction model’s prior, favors earlier collisions, and penalizes normalized jerk.Source: 2) Adversarial Score · p. 3
  • The selected trajectory is adversarial while avoiding unrealistic or unstable motion patterns.Source: 2) Adversarial Score · p. 3

C. Real-World Image Generator

The generator uses ODE-based flow matching with diffusion priors and Euler denoising for efficient, high-quality image generation. Its UNet combines camera-view controls and other conditions through distinct mechanisms.

C. Real-World Image Generator

1) Flow Matching:

  • Linear interpolation transfers pretrained diffusion priors into flow matching, avoiding retraining a nonlinear mapping and reducing training complexity.The approach combines pretrained priors with flow matching’s efficient inference.Source: 1) Flow Matching · pp. 3–4

C. Real-World Image Generator

  • A UNet backbone uses ControlNet for camera-projected information and attention mechanisms for other conditions.The figure describes the UNet as leveraging diffusion priors through linear transformation.Source: Figure 3 · p. 4

1) Flow Matching:

  • The model uses the diffusion output vθ and linearly maps between its discrete timestep and continuous flow-matching time.The paper identifies noise, x0, and vθ as equivalent diffusion-model output forms and chooses vθ.Source: 1) Flow Matching · p. 4

2) Condition Control:

The generator conditions images on projected scene elements and additional information to control content and style. The reported comparison indicates an image-quality advantage with few denoising steps and no increased computational cost.

2) Condition Control:

  • Projected 3D object boxes and lane markings provide view-specific controls, while a reference frame supports temporal consistency and control over weather and street scenes.Object IDs maintain cross-view consistency; additional conditions include camera parameters and object-box coordinates.Source: 2) Condition Control · p. 4
  • Cross-attention injects reference-image features, prompts, and low-dimensional information, while trainable ControlNet injects projected maps and surrounding objects.Prompts for different times and weather support images in different styles.Source: 2) Condition Control · p. 4
  • The generator demonstrates an image-quality advantage with few denoising steps without increasing computational cost.Table I compares FID, FVD, LPIPS, and SPI.Source: Table I · p. 5

D. Evaluation Metrics

The platform combines single-frame driving metrics into PDMS and derives a closed-loop driving score by multiplying segment-average PDMS by route completion.

D. Evaluation Metrics

  • Single-frame measures cover collisions, drivable-area compliance, ego progress, time-to-collision, and comfort.The metrics are NC, DAC, EP, TTC, and C.Source: D. Evaluation Metrics · p. 5
  • PDMS integrates the single-frame metrics, and the platform computes its average across the entire segment.The paper follows the PDMS definition proposed by NAVSIM.Source: D. Evaluation Metrics · p. 5
  • The closed-loop driving score is the average segment PDMS multiplied by route completion.Route completion uses the ground-truth ego trajectory as a reference; segment completion is calculated after recording episode termination reasons and collisions.Source: D. Evaluation Metrics · p. 5

III. EXPERIMENTS

The experiments cover setup, generated-image realism, and results for tested end-to-end models on the platform.

III. EXPERIMENTS

  • The experiments section introduces the study’s experimental setups.The section also covers generated-image realism and results for tested end-to-end models.Source: §III · p. 5

1) Real-World Image Generator:

The image generator uses Stable Diffusion 1.5 with flow matching and a 10-step denoising scheme, while the closed-loop setup replays Waymo traffic and evaluates UniAD and VAD.

1) Real-World Image Generator:

  • The generator uses Stable Diffusion 1.5 with a UNet, conditioning on prompts and reference images encoded by frozen pretrained CLIP components.The remaining components are trained on nuScenes scenes.Source: 1) Real-World Image Generator · p. 5
  • The generator is trained and tested on nuScenes, with 850 scenes for training and validation and 150 for testing.Source: 1) Real-World Image Generator · p. 5
  • A 10-step denoising scheme is used for image generation, compared with the 50–100 steps typical of fast DDIM models.The condition guidance scale is 2.0.Source: 1) Real-World Image Generator · p. 5

2) Closed-loop Setups:

  • The closed-loop setup replays Waymo Open Motion Dataset traffic in MetaDrive and uses DenseTNT trajectories for adversarial vehicles.DenseTNT outputs 32 candidate trajectories with prior probabilities.Source: 2) Closed-loop Setups · p. 5
  • The evaluation tests UniAD and VAD, with the closed loop running at 2Hz and Adversarial Traffic Flow at 10Hz.Source: 2) Closed-loop Setups · p. 5

B. Image Generation Validation

The image generator is evaluated for visual quality and traffic-element fidelity, then tested in adversarial closed-loop scenarios with UniAD and VAD. Adversarial traffic substantially reduces model scores and completion rates, and case studies show collisions despite realistic generated scenes.

B. Image Generation Validation

  • Figure 4 shows higher-quality boxed details in panels a and b, while panel c shows consistent rain across front and back views unlike the baseline.Source: Figure 4 · p. 6
  • With 10 denoising steps, the generator outperforms baselines using 20 steps, while low-step FID, FVD, and LPIPS outperform baselines without extra runtime.The evaluation uses approximately 36,000 images from the nuScenes validation set.Source: B. Image Generation Validation · pp. 5–6
  • Generated-image perception results have a minimal gap from original-image results on most metrics, supporting control over key traffic elements such as vehicles and lane markings.UniAD evaluates 3D object detection and bird’s-eye view segmentation; TABLE II compares perception results with original real images.Source: B. Image Generation Validation · p. 6, Table II · p. 6

C. Adversarial Closed-loop Experiments

IV. CONCLUSIONS

The paper presents a real-world adversarial closed-loop platform that combines a flow-matching image generator with adversarial traffic flow. Its experiments report realistic, controllable images and significant performance drops for UniAD and VAD under adversarial traffic, positioning the platform as a tool for corner-case evaluation.

IV. CONCLUSIONS

  • The platform combines a Real-World Image Generator and Adversarial Traffic Flow to test end-to-end models trained on real-world data.A two-episode design selects adversarial surrounding-vehicle trajectories for different tested models.Source: §IV · p. 7
  • Generated-image realism metrics outperform the baseline, and the generator shows strong controllability over key traffic elements.The flow-matching generator maintains image quality under low denoising steps.Source: §IV · p. 7
  • UniAD and VAD show significant score and completion-rate drops in adversarial closed-loop scenarios despite performing well under steady traffic.The authors conclude that the platform provides a tool for evaluating end-to-end autonomous driving in corner cases.Source: §IV · pp. 7–8

Related papers

Loading 2512.16055v2…