Source-linked AI summary
One Thousand and One Hours: Self-driving Motion Prediction Dataset
John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, Peter Ondruska
TL;DR
Self-driving research lacks broadly available large-scale datasets for motion prediction, planning, and related downstream tasks. This paper releases a route-focused dataset with detailed perception outputs, semantic mapping, aerial context, software, and baselines. The authors report meaningful performance increases for motion forecasting and motion planning as dataset size grows.
Problem
Large-scale datasets for motion prediction and planning are scarce, while planning is difficult to model and evaluate from real data.
Method
The paper releases a dataset of traffic scenes with full perception outputs, a detailed semantic map, aerial imagery, software tools, and motion forecasting and planning baselines.
Results
The dataset is the largest and most detailed available for prediction and planning, and performance increases meaningfully for both tasks with more training data.
Takeaways & Limitations
The release broadens access to self-driving machine-learning development and suggests that still larger datasets may be desirable.
Abstract
from arXiv · showhide
Motivated by the impact of large-scale datasets on ML systems we present the largest self-driving dataset for motion prediction to date, containing over 1,000 hours of data. This was collected by a fleet of 20 autonomous vehicles along a fixed route in Palo Alto, California, over a four-month period. It consists of 170,000 scenes, where each scene is 25 seconds long and captures the perception output of the self-driving system, which encodes the precise positions and motions of nearby vehicles, cyclists, and pedestrians over time. On top of this, the dataset contains a high-definition semantic map with 15,242 labelled elements and a high-definition aerial view over the area. We show that using a dataset of this size dramatically improves performance for key self-driving problems. Combined with the provided software kit, this collection forms the largest and most detailed dataset to date for the development of self-driving machine learning tasks, such as motion forecasting, motion planning and simulation. The full dataset is available at http://level5.lyft.com/.
1 Introduction
Large-scale datasets have advanced self-driving perception, but downstream motion prediction, planning, and simulation require behavioural observations, interactions, and semantic maps that are less broadly available. This work addresses that gap with a large, detailed dataset focused on motion forecasting and planning.
- Motivation: Downstream self-driving tasks require behavioural observations, interactions, and semantic maps in addition to perception outputs.These requirements support data-driven motion prediction, trajectory planning, and simulation.
- Motivation: Broadly available datasets for motion prediction, planning, and simulation are much more limited than perception datasets.Existing downstream datasets are mostly available through large-scale industrial efforts using in-house data.
- Contribution: The dataset targets motion forecasting and planning rather than training perception systems, using output from an already trained perception system.It processes large quantities of new data for motion prediction.
- Dataset: 1,000 hours of traffic scenes capture traffic-participant motions around 20 self-driving vehicles driving over 26,000 km along a suburban route.The collection is presented as the largest dataset to date for motion prediction.
- Dataset: The release includes over 15,000 human-annotated semantic-map elements, a 74 km2 aerial image at 6 cm per pixel, L5Kit, and baseline solutions.These components support access, visualization, motion forecasting, and motion planning development.
2 Related Work
Related work spans perception, prediction, and planning, but open datasets and evaluations remain especially limited for downstream tasks. The paper distinguishes its dataset through route-focused scale, richer perception outputs, detailed mapping, aerial context, and an open planning evaluation.
- Self-driving stack: Self-driving stacks estimate nearby actors, predict their future motion and intent, and use those predictions to plan the vehicle trajectory.Figure 2 presents this sequence from raw sensor input through perception and prediction to planning.
- Perception datasets: Perception datasets commonly provide 3D object labels, while the released dataset instead processes already trained perception output for motion prediction.Its main target is not training perception systems.
- Prediction datasets: Prediction methods require detailed environmental information such as semantic maps, yet representative large-scale training datasets are rare and mostly proprietary.The most relevant existing open dataset, Argoverse Forecasting, provides 300 hours and a lightweight map encoding lane centers.
- Prediction datasets: The dataset differs from Argoverse by providing 1,000 hours along a single route and full perception outputs including bounding boxes and class probabilities.The route focus is motivated by likely early deployment along a few high-demand ride-hailing routes.
- Prediction datasets: Its semantic map contains more than 15,000 human annotations rather than only lane centers, and the release also provides a high-resolution aerial image.The aerial representation may support development of semantic-map-free solutions.
- Planning datasets: Planning has received relatively little community attention because it is difficult to model and evaluate from real data, especially under closed-loop evaluation.Closed-loop evaluation requires collecting new data that are absent from the dataset.
- Planning datasets: An imitation-learning planning baseline with perturbations enables the first open evaluation of this approach and improves with more training data.The approach had previously been demonstrated only on a proprietary dataset with no available open equivalent.
3 Dataset
The released dataset contains 170,000 annotated driving scenes, supported by HD semantic and aerial maps, and collected from a sensor-equipped self-driving fleet on a fixed route.
- 170,000 scenes, each 25 seconds long, capture the self-driving vehicle, nearby traffic participants, and traffic-light states.
- The collection uses seven cameras, three LiDARs, and five radars processed by a perception system to represent traffic participants’ motion.
- The data was collected by a fleet driving along a fixed route and split into train, validation, and test sets using an 83–7–10% ratio.
- The HD semantic map contains 15,242 labelled elements, including 8,505 lane segments, with precise lane geometry and connectivity.
- The aerial map covers 74 km2 at 6 cm per pixel and provides spatial information that can aid motion prediction.
4 Development tools
The software toolkit supports efficient scene access, visualisation, rasterisation, and baseline implementations for motion forecasting and SDV motion planning.
- L5Kit provides data loading, visualisation, and implementations for motion forecasting and SDV motion planning baselines.
- Scenes can be sampled around either the SDV for planning or other traffic participants for predicting their motions.
- BEV rasterisations can combine semantic-map and aerial-map information into images suitable for conventional machine-learning pipelines.
- Motion forecasting baseline: The forecasting baseline predicts traffic participants’ future (x,y)-positions over a 5-second horizon using ResNet-50 and stacked BEV histories.
- Motion forecasting baseline: Adding motion history and increasing training-set size improve open-loop displacement-error performance.
- Motion planning baseline: The planning baseline executes SDV trajectories in closed-loop simulations where other participants follow logged behaviour and the SDV may diverge from the log.
5 Conclusion
The paper presents a large, detailed dataset for prediction and planning, reporting performance gains for both tasks as dataset size increases. It argues that continued scaling and public release can support further self-driving development.
- The dataset is described as the largest and most detailed available for training prediction and planning solutions.
- The authors report meaningful performance increases for both motion forecasting and motion planning relative to existing alternatives.
- Performance continues increasing with training-data size, motivating future datasets with tens of thousands or millions of hours.
- The authors frame publication as a step toward democratising self-driving application development and faster progress toward autonomy.