PaperlayerView exact revision on arXiv

Source-linked AI summary

WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios

Runsheng Xu, Hubert Lin, Wonseok Jeon, Hao Feng, Yuliang Zou, Liting Sun, John Gorman, Ekaterina Tolstaya, Sarah Tang, Brandyn White, Ben Sapp, Mingxing Tan, Jyh-Jing Hwang, Dragomir Anguelov

arXiv:2510.26125v3cs.CVcs.AI

TL;DR

Existing E2E benchmarks and open-loop metrics inadequately cover rare, multi-modal long-tail driving situations. The paper introduces WOD-E2E and RFS to address these gaps, and reports divergence between ADE and RFS, promise from MLLM-based models, and stronger reinforcement-learning gains when rewards align with RFS.

  • Problem

    Existing benchmarks mainly contain nominal scenarios, while ADE and related metrics inadequately represent multi-modal behavior and long-tail safety-critical cases.

  • Method

    The paper constructs WOD-E2E, a long-tail benchmark with 4,021 segments, and introduces RFS to compare predicted trajectories with expert-annotated preferences.

  • Results

    ADE and RFS diverge across submissions, with WayNet achieving ADE 2.8 but lower RFS and HMVLM attaining near-top RFS despite worse ADE.

  • Takeaways & Limitations

    RFS captures trajectory effectiveness in safety-critical, multi-modal long-tail scenarios beyond ADE alone, while MLLM-based models and RFS-aligned reinforcement learning show promise.

  • Takeaways & Limitations

    WOD-E2E uses open-loop evaluation because realistic sensor simulation has prohibitive computational cost.

Abstract

from arXiv · show

Vision-based end-to-end (E2E) driving has garnered significant interest in the research community due to its scalability and synergy with multimodal large language models (MLLMs). However, current E2E driving benchmarks primarily feature nominal scenarios, failing to adequately test the true potential of these systems. Furthermore, existing open-loop evaluation metrics often fall short in capturing the multi-modal nature of driving or effectively evaluating performance in long-tail scenarios. To address these gaps, we introduce the Waymo Open Dataset for End-to-End Driving (WOD-E2E). WOD-E2E contains 4,021 driving segments (approximately 12 hours), specifically curated for challenging long-tail scenarios that that are rare in daily life with an occurring frequency of less than 0.03%. Concretely, each segment in WOD-E2E includes the high-level routing information, ego states, and 360-degree camera views from 8 surrounding cameras. To evaluate the E2E driving performance on these long-tail situations, we propose a novel open-loop evaluation metric: Rater Feedback Score (RFS). Unlike conventional metrics that measure the distance between predicted way points and the logs, RFS measures how closely the predicted trajectory matches rater-annotated trajectory preference labels. We have released rater preference labels for all WOD-E2E validation set segments, while the held out test set labels have been used for the 2025 WOD-E2E Challenge. Through our work, we aim to foster state of the art research into generalizable, robust, and safe end-to-end autonomous driving agents capable of handling complex real-world situations.

1. Introduction

Vision-based E2E driving offers scalability and potential synergy with MLLMs, but existing benchmarks and metrics inadequately assess rare, multi-modal long-tail situations. WOD-E2E addresses these gaps with a long-tail dataset, the RFS metric, and benchmark studies.

  • Motivation: Vision-based E2E systems directly map raw sensor data to driving actions, reducing system complexity and reliance on intermediate-concept annotations.The paper also identifies potential for leveraging MLLM world knowledge in E2E driving.
  • Motivation: Existing E2E datasets predominantly contain nominal scenarios, limiting evaluation of robustness and generalization on rare real-world situations.The paper states that these scenarios occur with a frequency of less than 0.03% in daily driving.
  • Contributions: WOD-E2E provides 4,021 challenging segments totaling approximately 12 hours, with eight surrounding cameras, routing information, ego history, and future trajectories.The segments focus on scenarios occurring less than 0.03% of the time and combine autonomous and manual driving data.
  • Evaluation Gap: ADE and L2 compare predictions with a single logged trajectory, while PDMS requires annotated agent positions and trajectories that may be impractical for novel or hard-to-detect objects.These limitations are especially relevant when multiple reasonable future trajectories exist or objects are difficult to annotate.
  • Contributions: RFS is introduced as a human-aligned open-loop metric that evaluates predicted trajectories against rater preference labels rather than only logged-trajectory distance.The paper also reports baseline studies and analyses of methods submitted to a public leaderboard.

2. Related Works

Related work spans BEV-based, MLLM-based, and diffusion-based E2E planners, alongside broad autonomous-driving datasets and E2E benchmarks. The paper motivates WOD-E2E by identifying limited long-tail coverage and safety-focused evaluation in existing resources.

  • E2E Planners: BEV-based E2E planners fuse sensor information into BEV representations for perception, forecasting, occupancy prediction, and planning.UniAD propagates BEV queries across downstream tasks, while BEV-Planner learns a planning policy from BEV features.
  • E2E Planners: MLLM-based E2E methods use language models for action explanation, control prediction, chain-of-thought reasoning, or unified processing of driving tasks.Examples include DriveGPT4, DriveVLM, VLP, EMMA, OpenEMMA, and LightEMMA.
  • E2E Planners: Diffusion-based planners generate diverse plausible trajectories and can estimate planning uncertainty through diffusion policies or ensembles.DiffusionDrive targets real-time E2E driving, while EnDfuser combines camera and LiDAR features to produce candidate trajectory distributions.
  • Datasets: Existing autonomous-driving datasets support varied tasks, while prominent open E2E datasets include nuScenes, NAVSIM, WOMD, and CoVLA.The paper distinguishes dataset breadth from suitability for long-tail E2E evaluation.
  • Dataset Gaps: NAVSIM relies on filtered existing data and PDMS, which may underrepresent nuanced long-tail events and inadequately measure safety-critical cases involving amorphous obstacles or emergency maneuvers.WOD-E2E is designed to provide targeted long-tail data and a more safety-focused scoring mechanism.
  • Dataset Gaps: WOMD emphasizes motion prediction and agent interactions, but its lack of full camera images makes comprehensive external E2E research difficult.Only embeddings are provided instead of full camera images.

3.1. Dataset Overview

WOD-E2E consists of real driving segments centered on long-tail scenarios, with multi-camera observations, routing commands, ego status, scenario tags, and rater feedback labels. Its metadata and inputs support trajectory-based E2E benchmarking.

  • Dataset Composition: WOD-E2E contains 4,021 20-second segments, partitioned into 2,037 training, 479 validation, and 1,505 testing segments.The segments are mined from real driving logs and focus on long-tail scenarios.
  • Coordinate System: The dataset uses vehicle coordinates centered at the ego vehicle and sensor-frame coordinates linked through extrinsic transformations.Vehicle axes point forward, left, and upward; camera frames are centered at the lens.
  • Sensor Inputs: Eight cameras provide 360-degree coverage, with one JPEG image per direction plus camera intrinsics and extrinsics.The views cover front, front-left, front-right, side-left, side-right, rear, rear-left, and rear-right directions.
  • Routing Information: Routing input is a high-level command encoded as GO_STRAIGHT, GO_LEFT, or GO_RIGHT and derived by comparing the vehicle’s 10-second future route with its current route position.Commands indicate direction at decision points, not micro-maneuvers or speed profiles.
  • Routing Information: Figure 2 visualizes five-second ground-truth trajectories colored by left, straight, or right routing commands, with distances measured in meters.The routing labels are derived from ten-second futures.
  • Ego and Annotation Data: Each segment includes past trajectory, velocity, acceleration, future trajectory, one of 11 scenario types, and rater feedback labels.Expert labelers score three distinct five-second future trajectories from 0 to 10, where 0 is worst and 10 is best.

3.2. Quantitative Rareness Comparison

The authors compare dataset rarity using LLM-scored front-camera sequences and find that WOD-E2E concentrates more long-tail events than other datasets.

  • Gemini 2.5 Pro scores each dataset’s test scenes from 0–100 using front-camera sequences and a four-tier rarity prompt.The output includes a rarity_score, rare_factors, and a reasoning trace.
  • The comparative curve ranks scenes by rarity score and plots the average score for progressively larger dataset portions.
  • WOD-E2E’s curve remains above those of all other datasets across the plotted percentage tiles.
  • Around 93, WOD-E2E’s average rarity score for the most extreme 10% remains elevated even across the full dataset.

3.3. Long-tail Data Mining

WOD-E2E mines rare driving events from large-scale logs using rule-based categories, automated filtering, and human review, producing a final long-tail share of 0.03%.

  • Millions of miles of driving logs are predominantly nominal, motivating a mining strategy that combines rule-based heuristics with MLLMs.
  • The mining process categorizes logs into 11 long-tail scenario classes, including construction, intersections, pedestrians, cyclists, maneuvers, cut-ins, and foreign-object debris.
  • 6,888 miles, or 0.1% of 6,391,012 miles, met the automated long-tail criteria in a validation case study.
  • Human filtering with a 30% conversion rate reduced the final long-tail proportion to 0.03%.
  • The dataset is distributed across anonymized cities, scenario clusters, and road types, with Intersections, FOD, and Pedestrians comprising the largest clusters.

3.4. Data Labeling

The labeling pipeline selects critical moments, samples diverse trajectories, and uses human raters to rank candidates across safety-relevant dimensions.

  • The data-labeling pipeline has three stages: critical moment selection, trajectory sampling, and trajectory scoring.
  • 3.4.1. Critical Moment Selection: A critical moment is the frame where a critical event emerges and requires an important decision such as slowing, nudging, or giving way.
  • 3.4.1. Critical Moment Selection: Raters scan the video, choose the earliest frame where the event is visually apparent after action begins, and document their reasoning.
  • 3.4.2. Trajectory Sampling: Up to 64 diverse trajectories are generated for each critical moment using a model such as Wayformer and inputs describing agents, perception, and maps.
  • 3.4.2. Trajectory Sampling: Automated bucketing and lateral sampling produce usually fewer than 12 candidates, from which human labelers select three for final ranking.
  • 3.4.3. Trajectory Scoring: Raters require one optimal or appropriate trajectory and two alternative behavioral modes, then score them on safety, legality, and other dimensions.
  • 3.4.3. Trajectory Scoring: Rank 1 scores never fall below 6, while Rank 2 and Rank 3 span wider ranges, with many Rank 3 examples below 6.

3.5. Rater Feedback Score

RFS evaluates predicted trajectories against three human-rated references using speed-scaled trust regions, decayed scores, and the best matching reference.

  • RFS evaluates a predicted trajectory against three human-annotated reference trajectories, each assigned a score srater in [0, 10].
  • Trust regions surround each rater trajectory at t = 3 and 5 seconds using longitudinal and lateral distance thresholds.
  • The base thresholds are τ̄lat = 1.0, τ̄lng = 4.0 at t = 3 and τ̄lat = 1.8, τ̄lng = 7.2 at t = 5, then scale with initial speed v.
  • A prediction receives the rater score inside a trust region or an exponentially decayed score outside it, using longitudinal and lateral errors.
  • The final RFS takes the maximum score across reference trajectories, averages over t = 3, 5, and floors the result with 4.

4. Experimental Results

Experiments evaluate RFS with NaiveEMMA and leaderboard submissions, showing that the metric rewards long-tail capabilities and captures behavior that ADE can miss. Qualitative examples illustrate full, decayed, and floor scores according to agreement with rater-preferred trajectories.

  • Baseline Model: NaiveEMMA uses eight camera views, 3 seconds of ego-status history, and high-level routing input to predict a 5-second future trajectory.It is fine-tuned exclusively on the released WOD-E2E training split and does not use past camera frames.
  • RFS Validation: RFS assigns higher scores to models that use long-tail training data, multi-camera inputs, and test-time scaling on an internal long-tail test split.The experiment controls for these factors because they are expected to improve model quality in long-tail settings.
  • Qualitative Validation: RFS reaches 10.0 when predictions closely follow the best-rated trajectory, including careful following, intersection traversal, and safe nudging around a bus.A prediction aligned with an 8.0-rated preferred path receives an RFS of 8.0.
  • Qualitative Validation: Predictions outside the trust region receive decayed or floor scores when they diverge from rater-specified maneuvers or enter unrated regions.Examples include excessive speed between lane-following and lane-change trajectories, turning right instead of left, and proceeding straight instead of turning right.
  • Leaderboard Results: Leaderboard submissions span MLP-based, diffusion-based, and MLLM-based approaches, with representative RFS values of 7.556, 7.736, and 7.986 for AutoVLA, HMVLM, and Poutine.Swin-Trajectory achieves 7.543, slightly better than the baseline.
  • Discussion of Results: Extra data helps MLLM-based models more clearly than diffusion-based models, while the authors hypothesize that chain-of-thought reasoning helps MLLMs leverage diverse datasets.The diffusion-based models show only minor improvements when combining data sources.
  • Discussion of Results: A better ADE does not guarantee a better RFS: WayNet has ADE 2.8 but low RFS, whereas HMVLM has worse ADE but near-top RFS.The authors report only a mild positive correlation between RFS and ADE across 19 submissions.
  • Discussion of Results: Reinforcement learning improves performance particularly when its reward matches the target metric, with Poutine using RFS and AutoVLA using ADE.The gain is reported as significantly more pronounced for Poutine, whose reward is directly aligned with RFS.

5. Conclusion

The paper introduces WOD-E2E and RFS to evaluate end-to-end driving in rare, challenging scenarios where conventional metrics are insufficient. Results support the dataset’s utility, show divergence between ADE and RFS, and highlight aligned reinforcement learning rewards, while the benchmark remains open-loop because realistic sensor simulation is computationally prohibitive.

  • Contributions: WOD-E2E provides 4,021 driving segments totaling approximately 12 hours and targets scenarios occurring less than 0.03% of the time.The benchmark is curated specifically for challenging long-tail situations.
  • Contributions: RFS evaluates predicted trajectories against expert-annotated preference labels rather than relying only on distance to a single logged trajectory.The metric is introduced for complex, multi-modal long-tail situations.
  • Findings: Benchmark analysis validates the dataset’s utility by demonstrating a clear divergence between ADE and RFS scores.The conclusion presents RFS as essential for capturing performance in safety-critical scenarios.
  • Findings: The benchmark results highlight MLLM-based models and reinforcement learning whose reward is directly aligned with RFS.The conclusion specifically identifies reward alignment with RFS as effective.
  • Limitation: WOD-E2E uses an open-loop setup because realistic sensor simulation has prohibitive computational cost.The authors identify this setup as a limitation while noting possible use of the data for testing high-fidelity simulator generalizability.
Loading 2510.26125v3…