PaperlayerView exact revision on arXiv

Source-linked AI summary

Argoverse: 3D Tracking and Forecasting with Rich Maps

Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, James Hays

arXiv:1911.02620v1cs.CVcs.RO

TL;DR

Autonomous-driving research lacks large annotated datasets that combine dynamic-object benchmarks with richly structured map context. Argoverse releases tracking and forecasting datasets with synchronized multimodal sensing, semantic vector maps, and diverse trajectories, then evaluates map-aware baselines. The reported baseline experiments find higher accuracy for both 3D tracking and motion forecasting, while the tracking study remains an initial exploration using relatively simple methods.

  • Problem

    Public autonomous-driving datasets rarely include richly mapped attributes, leaving their representation and use for 3D tracking and motion forecasting as an open research question.

  • Method

    Argoverse releases large-scale annotated datasets for 3D tracking and motion forecasting together with semantic vector maps and evaluates map-aware baseline methods.

  • Results

    Map context yields higher accuracy in both 3D tracking and motion forecasting across the reported baseline experiments.

  • Takeaways & Limitations

    Argoverse provides benchmarks for investigating rich-map context in autonomous-driving perception and forecasting.

  • Takeaways & Limitations

    The tracking and forecasting experiments are an initial exploration, and the tracking study uses relatively simple baselines.

Abstract

from arXiv · show

We present Argoverse -- two datasets designed to support autonomous vehicle machine learning tasks such as 3D tracking and motion forecasting. Argoverse was collected by a fleet of autonomous vehicles in Pittsburgh and Miami. The Argoverse 3D Tracking dataset includes 360 degree images from 7 cameras with overlapping fields of view, 3D point clouds from long range LiDAR, 6-DOF pose, and 3D track annotations. Notably, it is the only modern AV dataset that provides forward-facing stereo imagery. The Argoverse Motion Forecasting dataset includes more than 300,000 5-second tracked scenarios with a particular vehicle identified for trajectory forecasting. Argoverse is the first autonomous vehicle dataset to include "HD maps" with 290 km of mapped lanes with geometric and semantic metadata. All data is released under a Creative Commons license at www.argoverse.org. In our baseline experiments, we illustrate how detailed map information such as lane direction, driveable area, and ground height improves the accuracy of 3D object tracking and motion forecasting. Our tracking and forecasting experiments represent only an initial exploration of the use of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

1. Introduction

Argoverse addresses the lack of richly mapped autonomous-driving datasets by releasing annotated benchmarks for 3D tracking and motion forecasting. Its HD maps support map-based methods, which improve accuracy in both tasks.

  • Motivation: Public autonomous-driving datasets rarely include detailed maps, although such maps are critical for real-world autonomous systems.OpenStreetMap can help but has limited detail and accuracy.
  • Research scope: Argoverse investigates how rich map features can support 3D tracking and motion forecasting while enabling new benchmarks with real-world annotated data.The maps expose attributes such as road membership, lane assignment, and intersection proximity.
  • Datasets: The 3D tracking dataset synchronizes LiDAR, 360° cameras, and stereo cameras across two cities and varied conditions, with 360° imagery captured at 30fps.It includes synchronized multimodal sensor data sampled across Pittsburgh and Miami conditions described by the paper context.
  • Datasets: Ground-truth 3D tracks cover 15 object classes and contain five times as many tracked objects as the KITTI tracking benchmark.This expands the scale of annotated 3D tracking data available for benchmarking.
  • Datasets: The motion forecasting dataset contains diverse scenarios such as intersection turns, high-traffic clutter, and lane changes.These scenarios target forecasting conditions beyond routine parked or nearly constant-velocity vehicle motion.
  • Maps: Argoverse provides a semantic vector map of road infrastructure and traffic rules, supporting map-based perception, forecasting, and map automation benchmarks.The map includes HD information and an accompanying API.
  • Experiments: Map context improves accuracy in both 3D tracking and motion forecasting through ground-point removal, lane-orientation snapping, diverse lane-graph predictions, and driveable-area pruning.The experiments examine these map-based choices as initial evidence of rich-map utility.

2. Related Work

Argoverse extends autonomous-driving benchmarks by pairing annotated dynamic-object data with detailed 3D vector maps. Its scene visualization illustrates how map context organizes LiDAR points, ground structure, object annotations, and road geometry.

  • Datasets with map information: Before Argoverse, public autonomous-driving datasets rarely paired annotated data with detailed maps, limiting common benchmarks for map automation.The cost of annotating and constructing such maps helped drive interest in map automation.
  • Datasets with map information: Unlike rasterized region indicators, Argoverse includes driveable area, ground height, and connected vector lane centerlines.The comparison passage contrasts this representation with earlier nuScenes map releases.
  • 3D object tracking: Prior tracking datasets variously provide pedestrian sequences, raw vehicle-sensor data without object annotations, or tasks other than dynamic-object tracking.The related-work discussion distinguishes tracking resources by annotation and task coverage.
  • Trajectory forecasting: Prior trajectory-forecasting methods model social interaction or multimodality, but the cited pedestrian studies do not use static context such as maps.The passage motivates examining spatial context in self-driving forecasting.
  • Argoverse visualization: Figure 2 maps LiDAR points onto a virtual image plane and uses driveable area and ground height to visually distinguish regions and points.Cuboid annotations appear in pink, road centerlines in yellow, and near-ground points in cyan.

3. The Argoverse Dataset

Argoverse combines multimodal vehicle data, extensive annotations, mined forecasting scenarios, and rich map representations across Pittsburgh and Miami. Its maps encode lane geometry and semantics, ground height, and driveable areas for tracking and forecasting research.

  • Dataset overview: Argoverse releases sensor data, maps, annotations, an API, tutorials, and baseline code under Creative Commons and MIT licenses.The statistics and experiments are based on Argoverse v1.1, released in October 2019.
  • Dataset overview: Data were collected in Pittsburgh and Miami across varied climates, architecture, infrastructure, seasons, weather conditions, and times of day.The captured data traverses nearly 300 km of mapped road lanes.
  • Maps: Argoverse maps combine lane-centerline vectors, rasterized ground height, and rasterized driveable-area and region-of-interest maps.Lane centerlines form a localized graph whose attributes include intersection status, traffic controls, turn direction, and predecessor or successor identifiers.
  • Maps: Driveable-area labels use a binary 1-meter grid, while lane centerlines are polylines formed from ordered straight segments.Curved lanes are approximated by connected line segments, and lane segments may overlap physically at intersections.
  • 3D tracking data: The tracking dataset contains 113 log segments with 11,052 human-annotated tracked objects, restricted to objects within 5 m of mapped driveable area.Segments last 15–30 seconds and include annotated dynamic and static objects with 3D bounding cuboids.
  • Motion forecasting data: The forecasting benchmark contains 324,557 five-second sequences mined for intersections, turns, adjacent-lane changes, and dense traffic.Vehicles were selected from 1006 driving hours across Miami and Pittsburgh, with interesting behavior found in 320 hours.

4. 3D Object Tracking

The tracking benchmark evaluates a hybrid 3D LiDAR-and-camera tracker and tests map-based heuristics for vehicle detection, ground removal, and orientation. Map-based ground removal improves longer-range geometry and detection, while lane direction substantially improves orientation.

  • Baseline and evaluation: The baseline tracker operates directly in 3D using LiDAR and ring-camera images, while evaluating the effect of simple map-based heuristics on vehicle tracking.Experiments run on the 24 logs in the Argoverse 3D Tracking test set.
  • Tracking formulation: The tracking formulation estimates object-center locations and orientations over F frames, producing track hypotheses for unique objects in the sequence.The observer vehicle is generally moving, while surrounding tracked vehicles may be static or moving.
  • Baseline tracker: The baseline clusters LiDAR returns, prunes non-vehicle returns with Mask R-CNN, associates clusters over time, estimates transformations with ICP, and uses a constant-velocity Kalman filter.Unmatched objects retain their motion-model pose for up to five frames, supporting short-term identity through occlusion.
  • Map-based heuristics: Driveable-area constraints reduce false positives, while map-based ground-height information removes ground returns effectively in sloping and uneven environments.The ground-height approach is contrasted with local ground-plane estimation.
  • Evaluation: Map-based ground removal yields better 3D IoU and slightly higher MOTA than plane fitting at longer ranges, but slightly worse orientation.Lane direction from the map significantly improves orientation performance.
  • Scope: The authors use relatively simple 3D tracking baselines and position the data as enabling further map-based and multimodal tracking research.The statement frames the experiments as an initial baseline rather than an exhaustive tracking approach.

5. Motion Forecasting

This section defines Argoverse motion forecasting as predicting 3 seconds of future vehicle motion from 2 seconds of observation, while evaluating multimodal, social, and map-based context. Baselines compare constant-velocity, nearest-neighbor, and LSTM models with map priors or pruning, showing that maps and multiple hypotheses improve forecast quality.

  • 5. Motion Forecasting: The task observes 20 frames over 2 seconds and predicts 30 frames over the next 3 seconds, using other-object trajectories and map information as context.The five-second horizon is intended to capture salient vehicle behavior such as crossing an intersection.
  • 5. Motion Forecasting: The evaluation uses minADE, minFDE, Drivable Area Compliance, and Miss Rate to assess best-case accuracy, map compliance, and misses across multiple forecasts.DAC is (n−m)/n when m of n predicted trajectories leave the drivable area; MR counts forecasts whose best final point exceeds the stated threshold.
  • 5. Motion Forecasting: Forecasting is evaluated over 3-second horizons using multimodal predictions, social context, and spatial context from vector maps.The evaluation compares models that vary prediction multiplicity and contextual inputs.
  • 5. Motion Forecasting: The baselines include Constant Velocity, Nearest Neighbor, deterministic LSTM, LSTM+social, and map-based nearest-neighbor or LSTM variants.Map-prior models query trajectories relative to lane centerlines, allowing guesses along multiple centerlines.
  • 5.3. Results: Map-prior baselines consistently improve DAC, while map pruning improves selected minADE and minFDE as the number of predictions increases.With K = 6, NN+map(prior) m-G,n-C outperforms deterministic LSTM+social with K = 1, and multiple velocity profiles along shared centerlines outperform one prediction per centerline.

6. Discussion

Argoverse contributes two large-scale autonomous-driving datasets, examines map-derived context for tracking and forecasting, and provides public access to the data and evaluation infrastructure. The discussion positions map data as improving forecasting accuracy while supporting public benchmarking.

  • 6. Discussion: Argoverse contains large-scale datasets for 3D tracking and motion forecasting with rich map information including lane centerlines, ground height, and driveable area.The work also mines one thousand hours of fleet logs for diverse object trajectories.
  • 6. Discussion: The authors report that map data improves motion-forecasting accuracy and maintain a public leaderboard for both tracking and forecasting.Sensor data, maps, annotations, and code are made available through Argoverse.org.

Appendices

The appendices describe Argoverse’s city-level coordinate system, map coverage and attributes, and software interfaces for using rich maps with sensor data. The mapping design favors interpretable local coordinates and exposes rasterized and geometric representations for perception.

  • Appendices: The map covers 204 linear kilometers of lane centerlines in Miami and 86 linear kilometers in Pittsburgh, plus driveable-area coverage in both cities.Driveable area covers 788,510 m2 in Miami and 286,104 m2 in Pittsburgh.
  • Appendices: Argoverse uses a city-level local tangent plane with a flat-Earth assumption that is approximately correct at city scale.City coordinates can be converted to UTM coordinates by adding the city origin in UTM coordinates.
  • Appendices: The city coordinate system is favored over WGS84 because its pose values are more interpretable for travel within a city.Ground-truth object poses are supplied in the ego-vehicle frame and aligned with maps through one SE(3) transform.
  • Appendices: The vector map is based on lane centerlines with intersection labels and predecessor-successor connectivity, alongside driveable-area and ground-height representations.These attributes are projected onto camera images in the appendix examples.
  • Appendices: The Python API provides bird’s-eye-view map images up to 100 m around the ego vehicle and a dense 1 m ground-surface grid.The grid is intended to support ground classification when globally planar assumptions fail.

B. 3D Tracking Taxonomy Details

This appendix defines the 15 object categories used in Argoverse 3D Tracking and illustrates map structures such as lane centerlines, intersections, and connectivity. The taxonomy distinguishes road users, vehicles, animals, and static or miscellaneous movers.

  • B. 3D Tracking Taxonomy Details: Argoverse 3D Tracking version 1.1 contains 15 object categories with definitions supplied for each class.The categories include vehicles, vulnerable road users, animals, and road obstacles.
  • B. 3D Tracking Taxonomy Details: The map’s lane centerlines represent complex road geometry, distinguish intersection segments, and encode predecessor and successor lane relationships.The figure uses separate colors for centerlines, hallucinated areas, intersection segments, and other lane segments.
  • B. 3D Tracking Taxonomy Details: The taxonomy defines Motorcyclist, Bicyclist, Pedestrian, and Animal as people riding vehicles, people walking, or four-legged animals.Bicycle and Motorcycle denote the corresponding non-motorized and motorized two-wheeled vehicles.
  • B. 3D Tracking Taxonomy Details: Other Mover denotes movable road objects not covered by another category, while On Road Obstacle denotes static obstacles on the driveable surface.These categories cover residual movable and static road objects within the taxonomy.
  • B. 3D Tracking Taxonomy Details: Vehicle classes distinguish standard automobiles, buses, large vehicles, emergency vehicles, mopeds, motorcycles, and trailers by size, propulsion, use, or operational role.Emergency vehicles are defined by active lights and sirens, while trailers are non-motorized vehicles towed behind motorized vehicles.

C.1. Motion Forecasting Data Mining Details

The forecasting dataset is mined from extensive driving data by identifying rare, behaviorally interesting trajectories with vector-map support. Scoring emphasizes intersection activity, turns, lane changes, speed variation, and visibility.

  • Forecasting scenarios are mined from over 1006 driving hours because challenging behaviors are rare but easy to identify with a vector map.
  • Each track in every 5-second sequence receives an interesting score based on its driving context and observed motion.
  • High-interest cases include intersections, turn lanes, lane changes, high median velocity, velocity variance, and longer visibility.
  • Changing lanes and left or right turns receive greater importance because these scenarios are very rare.
  • Map-based baselines transform trajectories into curvilinear centerline coordinates, improving generalization across map locations and orientations.

D.1. Tracker Implementation Details

The baseline tracker combines map-constrained LiDAR processing, image-based pruning, temporal association, point-cloud registration, and Kalman filtering. It uses heuristic object selection and fixed-size vehicle boxes.

  • The baseline tracker is presented as a non-novel implementation that works reasonably well, particularly when map information is available.
  • The tracker processes surrounding vehicles from LiDAR through segmentation, detection, association, geometric registration, and pose estimation.
  • LiDAR points are clustered with DBSCAN, then Mask R-CNN image masks remove clusters whose projections do not fall inside detected objects.
  • The tracker restricts points to the mapped driveable region and removes ground using either a ground-height map or plane fitting.
  • Vehicle candidates are selected with heuristics for typical car shape and volume, with centers estimated by fitting a smallest enclosing circle.
  • Clusters are associated using nearest-neighbor and Hungarian assignment, while ICP measurements feed a Kalman Filter with static or constant-velocity motion models.

D.2. Tracking Evaluation Metrics

Tracking is evaluated with standard multiple-object tracking metrics based primarily on centroid distance. The metrics capture accuracy, precision, identity consistency, track coverage, and detection errors.

  • The MOT evaluation uses centroid distance as its distance measure.
  • MOTA combines false negatives, false positives, and identity switches relative to ground-truth objects and is reported as a percentage.
  • MOTP measures the distance of matched detections, using the number of matches and their distances.
  • IDF1 is an F1 score based on precision and recall, where recall uses true positives over ground-truth labels.
  • MT and ML measure trajectories tracked for more than 80% and less than 20% of their lifetimes, respectively.
  • FP, FN, IDsw, and Frag count false positives, false negatives, identity switches, and transitions from tracked to untracked.

D.3. True Positive Thresholding Discussion

The paper argues that 3D tracking evaluation should account for absolute spatial error rather than relying strictly on scale-invariant IoU. Centroid distance is used because distant LiDAR objects have sparse measurements but can retain estimable centers.

  • IoU is scale invariant, but the authors argue that 3D tracking evaluation should not be strictly scale invariant because absolute error matters.
  • Small objects can receive favorable IoU judgments despite substantial absolute errors, making scale-sensitive evaluation important in 3D.
  • Distant LiDAR objects may have only three returns, making spatial extent difficult to estimate while their centroids remain estimable.
  • The evaluation therefore classifies correct versus incorrect matches using absolute distance between object centroids.
  • Figure 16 contrasts fixed inter-centroid distance and fixed intersection area, showing that the corresponding IoU values vary across matched cuboid shapes.
Loading 1911.02620v1…