Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting

Benjamin Wilson; William Qi; Tanmay Agarwal; John Lambert; Jagjeet Singh; Siddhesh Khandelwal; Bowen Pan; Ratnesh Kumar; Andrew Hartnett; Jhony Kaesemodel Pontes; Deva Ramanan; Peter Carr; James Hays

Source-linked AI summary

Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting

Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, James Hays

arXiv:2301.00493v1Jan 2, 2023cs.CVcs.AIcs.LGcs.RO

TL;DR

Argoverse 2 addresses the need for self-driving datasets that support challenging, diverse research without simply increasing sensor-data scale. It introduces three datasets spanning multimodal perception, lidar self-supervision and forecasting, and motion forecasting, with baseline explorations indicating broad research utility.

Problem
Self-driving benchmarks face logistical scale constraints, while motion forecasting also needs challenging long-tailed scenarios and increased task difficulty.
Method
The paper constructs three datasets: a multimodal annotated Sensor Dataset, a large high-frequency Lidar Dataset, and a diverse Motion Forecasting Dataset, with HD-map information and task baselines.
Results
The datasets provide baseline explorations for 3D object detection, point cloud forecasting and motion forecasting, and offer HD maps, rich taxonomies, geographic diversity and interesting scenes.
Takeaways & Limitations
Argoverse 2 provides new opportunities for perception and forecasting research, including point cloud forecasting, self-supervised learning and safety-focused evaluation of tail events.
Takeaways & Limitations
The datasets contain label noise and imperfect tracking, and the motion forecasting dataset omits 3D object bounding boxes.

Abstract

from arXiv · show

We introduce Argoverse 2 (AV2) - a collection of three datasets for perception and forecasting research in the self-driving domain. The annotated Sensor Dataset contains 1,000 sequences of multimodal data, encompassing high-resolution imagery from seven ring cameras, and two stereo cameras in addition to lidar point clouds, and 6-DOF map-aligned pose. Sequences contain 3D cuboid annotations for 26 object categories, all of which are sufficiently-sampled to support training and evaluation of 3D perception models. The Lidar Dataset contains 20,000 sequences of unlabeled lidar point clouds and map-aligned pose. This dataset is the largest ever collection of lidar sensor data and supports self-supervised learning and the emerging task of point cloud forecasting. Finally, the Motion Forecasting Dataset contains 250,000 scenarios mined for interesting and challenging interactions between the autonomous vehicle and other actors in each local scene. Models are tasked with the prediction of future motion for "scored actors" in each scenario and are provided with track histories that capture object location, heading, velocity, and category. In all three datasets, each scenario contains its own HD Map with 3D lane and crosswalk geometry - sourced from data captured in six distinct cities. We believe these datasets will support new and existing machine learning research problems in ways that existing datasets do not. All datasets are released under the CC BY-NC-SA 4.0 license.

1 Introduction

AV2 is designed around scale-aware, diverse, map-rich datasets that target difficult and underrepresented self-driving problems. It contributes three datasets spanning multimodal sensing, lidar self-supervision, and motion forecasting.

Guiding principles: AV2 follows four design principles: manageable scale, difficult scenarios, geographic diversity, and richer HD maps.The datasets match, rather than exceed, the scale of prior sensor benchmarks while mining crowded or unusual scenes across six cities and seasons.
Sensor Dataset: 30 categories define the 1,000-scenario Sensor Dataset, with 26 categories containing at least 6,000 cuboids and stereo imagery included.The taxonomy is intended to support diverse training and testing, including rare object classes.
Lidar Dataset: 20,000 scenarios make the Lidar Dataset the largest dataset for self-supervised lidar learning, with HD maps unavailable in the similar ONCE dataset.Its scale is motivated by learning better representations from large-scale lidar data in dynamic scenes paired with maps.
Motion Forecasting Dataset: 250,000 scenarios give the Motion Forecasting Dataset five dynamic and five static actor types and the largest mapped area among such datasets.The dataset emphasizes rare, diverse, and challenging interactions rather than expected performance on easy driving miles.
Research scope: AV2 is intended to support tasks ranging from 3D detection and tracking to depth estimation, forecasting, map automation, and point cloud forecasting.The authors frame the datasets as supporting both established and emerging self-driving machine learning problems.

2 Related Work

Prior self-driving datasets have driven advances in detection, forecasting, map representations, and broader perception research. AV2 extends this ecosystem with diverse, high-frequency forecasting scenarios and opportunities for self-supervised and map-based methods.

Sensor datasets and 3D detection: Sensor datasets have supported major advances in 3D detection and tracking, including anchor-based, anchor-free, offboard detection, tracking, and simulation methods.Improved detection has enabled isolation of object-specific point clouds for downstream applications.
Motion forecasting: Attention-based motion forecasting methods have introduced vector representations for maps and trajectories, while datasets increasingly provide detailed lane and crosswalk information.Prior releases established lane graphs and added features such as traffic controls, crosswalk polygons, lane markings, speed limits, and stop signs.
Motion forecasting: AV2 provides forecasting data at exactly 10 Hz across more than 2,000 km of unique roadways in six cities.This contrasts with nuScenes at 2 Hz and Lyft’s single 10 km road segment, while complementing Waymo Open Motion Dataset’s similar scenario approach.
Broader perception problems: Large-scale self-driving sensor datasets also enable emerging perception problems such as video semantic segmentation, HD map automation, and map change detection.These datasets broaden research beyond object detection and motion forecasting, particularly for tasks potentially solvable through self-supervision.

3 The Argoverse 2 Datasets

Argoverse 2 comprises richer Sensor, Lidar, and Motion Forecasting datasets designed around diverse, challenging scenarios and detailed per-scenario HD maps. Its scale, taxonomy, temporal sampling, and map representation target perception, forecasting, self-supervision, and long-tail driving research.

3.1 Sensor Dataset: The Sensor Dataset contains 1,000 manually selected 15-second scenes with 30 object categories, 26 of them sufficiently sampled for training and evaluation.Scenes emphasize crowded settings, under-represented objects, noteworthy weather, and unusual behaviors.
3.1 Sensor Dataset: AV2 Sensor scenes contain 23× as many non-vehicle, non-pedestrian cuboids as Argoverse 1, alongside comparisons of annotation range, diversity, density, and dynamism.The dataset was designed to include richer and more challenging scenes than its predecessor.
3.2 Lidar Dataset: The Lidar Dataset provides 20,000 unlabeled 30-second sequences, totaling about 40× as many driving hours as the Sensor Dataset within five times the space budget.It supports self-supervised lidar learning and point cloud forecasting, with 6 million frames sampled at 10 Hz.
3.4 HD Maps: All three datasets provide scenario-specific HD maps, including 3D lane boundaries, vectorized driveable areas, and, for Sensor data, dense ground-surface height.The lane graph uses 3D geometry rather than only centerlines or 2D lane geometry.

4 Experiments

Argoverse 2 experiments establish baselines for 3D detection, point-cloud forecasting, and motion forecasting, illustrating both dataset difficulty and the value of scale, map information, and social context.

Overview: The experiments evaluate 3D object detection, point-cloud forecasting, and motion forecasting using the Sensor, Lidar, and Motion Forecasting datasets.The paper provides baseline explorations for these three learning problems.
3D Object Detection: 3D detection results show poor performance on long-tail categories despite substantial training data.The evaluation includes classes such as strollers, wheelchairs, and dogs.
Point Cloud Forecasting: Increasing training data steadily improves point-cloud forecasting across mean IoU, l1-norm, and Chamfer distance.The same SPF2 model architecture is trained with up to 16,000 sequences.
Motion Forecasting: Argoverse 2 motion forecasting is more kinematically diverse than Argoverse 1 and cannot be solved by constant-velocity assumptions.Constant-velocity models perform particularly poorly in the mined intersection scenarios.
Motion Forecasting: For K=6, WIMP significantly outperforms the other baselines, while graph-based attention methods are nearly twice as accurate as Nearest Neighbor.The results emphasize using map priors, social context, and diverse predictions.

5 Conclusion

Argoverse 2 introduces three datasets with baselines for three forecasting and perception tasks, offering HD maps, rich taxonomies, geographic diversity, and interesting scenes. The datasets retain limitations from human annotation and tracking, including missing 3D bounding boxes in motion forecasting.

Conclusion: Argoverse 2 comprises three datasets and provides baseline explorations for 3D object detection, point-cloud forecasting, and motion forecasting.The datasets also support additional research tasks.
Conclusion: The datasets provide HD maps, rich taxonomies, geographic diversity, and interesting scenes.These are the paper’s stated comparative strengths.
Limitations: The datasets contain label noise, while the motion forecasting dataset lacks 3D bounding boxes and has imperfect tracking.The authors state that label noise is minimized before release and object extent can be reasonably assumed from object type.

6 Appendix

The appendix documents sensor and map representations, scenario-mining criteria, benchmark trends, and additional detection and actor-distribution analyses.

Additional Information About Sensor Suite: The sensor suite uses vehicle, camera, and lidar coordinate systems, while scenarios include pose, calibration, multimodal video, and lidar returns.Each Sensor Dataset scenario lasts 15 seconds and includes seven ring cameras, two stereo cameras, and two lidars.
Scenario Mining: Scenario mining favors sharp turns, significant acceleration changes, complex lane-graph traversals, dense actor regions, rare object classes, and ego-route intersections.The criteria target kinematically unusual, map-complex, socially dense, and rare-object interactions.
Benchmark Trends: Argoverse 1.1 MinFDE gains plateaued since early 2020, although the number and diversity of near-SOTA methods continued to grow.Figure 10 also indicates changing leaderboard sorting by Miss Rate in later competitions.
Additional Results: The appendix includes additional 3D detection metrics and distributions of pedestrian speeds, annotated cuboid yaw, and scenario actor counts.The Lidar Dataset follows the same actor-count distribution as the Motion Forecasting Dataset because it is mined using the same criteria.
HD Maps: HD map attributes include intersection status, lane type, neighboring segments, 3D lane boundaries, and left and right lane-mark types.Map attributes are summarized for each Argoverse 2 scenario.

Average Orientation Error (AOE)

Average Orientation Error measures the absolute angular difference between a detected object’s orientation and the ground-truth orientation.

Average Orientation Error (AOE): AOE = |θdet − θgt| measures the absolute difference between detected and ground-truth orientation angles.The metric directly quantifies orientation error.

Composite Detection Score (CDS)

The supplied passages describe AV2 evaluation across detection, transfer learning, and object-type forecasting, highlighting distribution sensitivity and metric limitations.

Transfer Learning: Transfer learning performs best when training and testing use the same Argoverse distribution.On Argoverse 2, training on Argoverse 2 yielded minFDE 2.91 versus 6.82 when trained on Argoverse 1.1.
Transfer Learning: 2.3x worse minFDE occurred when a model trained on Argoverse 1.1 (3s) was tested on Argoverse 2 (6s).The reverse transfer setting was approximately 1.8x worse, indicating asymmetric transfer usefulness.
Transfer Learning: Argoverse 2 transfer results indicate greater challenge and diversity than its predecessor.The passage attributes distribution shift to prediction horizon, cities, mining protocols, and object types.
Different Object Types: Pedestrians have significantly lower displacement errors than other object types in the Nearest Neighbor baseline without map priors.The passage attributes this pattern to pedestrians' significantly slower velocities and argues that lower errors do not make forecasting solved.

7 Datasheet for Argoverse 2

Argoverse 2 was created to advance machine-learning research for self-driving across multiple tasks, with three datasets addressing distinct gaps through richer taxonomies and challenging data.

Purpose: Argoverse 2 supports research spanning 3D perception, motion forecasting, and HD map automation.The datasets were created to improve upon the initial Argoverse datasets and support the global research community.
Purpose: The three proposed datasets address different gaps in existing self-driving research datasets.The manuscript refers readers to comparison charts for a detailed breakdown.
Sensor Dataset: The Sensor Dataset combines a richer taxonomy, stereo imagery, and 1,000 diverse logs containing varied object types and interactions.It is described as the only dataset of similar size to have stereo imagery.
Motion Forecasting Dataset: The Motion Forecasting Dataset uses a richer taxonomy and mines scenarios emphasizing unusual behaviors that are difficult to predict.
Lidar Dataset: The Lidar Dataset is the largest lidar dataset and includes HD maps and high-frame-rate lidar.Only the concurrent ONCE dataset is described as similarly sized for self-supervised learning in lidar space.

COMPOSITION

Argoverse 2 comprises three scenario-based datasets for sensor perception, lidar learning, and motion forecasting, each paired with an HD map. Their contents, labels, sampling strategies, splits, and documented limitations differ by dataset.

Dataset instances: The three datasets contain 1,000 Sensor, 20,000 Lidar, and 250,000 Motion Forecasting scenarios, representing continuous observations of scenes around a self-driving vehicle.Scenario durations are 15, 30, and 11 seconds, respectively.
Sensor Dataset: Sensor scenarios provide synchronized multimodal data from seven ring cameras, two stereo cameras, two lidars, vehicle pose, and 3D tracks for dynamic objects.The Sensor Dataset supports 3D detection and tracking, while its labels are held out for the test set.
Lidar Dataset: Lidar scenarios contain lidar data without imagery or human annotations beyond the HD map, while evolving point clouds support point cloud forecasting.The point cloud itself can serve as ground truth for forecasting.
Motion Forecasting Dataset: Motion Forecasting scenarios contain object tracks sampled at 10 Hz, with scored actors guaranteed observed throughout the scenario for future-motion prediction.Tracks include object category, location, and heading; future motion is withheld in the test set.
Maps and labeling: Every scenario carries an HD map with lane, crosswalk, driveable-area, lane-boundary, paint-type, and ground-height information, while labels can be incomplete under defined observation limits.Sensor objects are labeled only within 5 meters of driveable area, and Sensor and Motion Forecasting tracks may end when objects leave range or become occluded.
Sampling and limitations: Scenarios are sampled through manual review or behavior-based heuristics, split into train, validation, and test sets, and distributed with documented sensor, map, annotation, and calibration noise.The datasets support additional tasks including visual odometry, lane detection, self-supervised learning, and point cloud forecasting; preprocessing software is unavailable.