PaperlayerView exact revision on arXiv

Source-linked AI summary

nuScenes: A multimodal dataset for autonomous driving

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, Oscar Beijbom

arXiv:1903.11027v5cs.LGcs.CVcs.ROstat.ML

TL;DR

Autonomous-driving research lacks multimodal benchmarks covering the full sensor suite and diverse conditions. nuScenes provides such a dataset with detection and tracking tasks, metrics, and baselines, and its challenge enabled lidar- and camera-based methods to improve over prior state of the art.

  • Problem

    Autonomous-driving perception lacks multimodal benchmark datasets covering full sensor suites and diverse operating conditions needed for detection and tracking research.

  • Method

    nuScenes constructs a large-scale 360° multimodal dataset and introduces AV-focused detection and tracking metrics with lidar- and image-based baselines.

  • Results

    40% and 81% improvements over prior state of the art were achieved by lidar-based and camera-based detection methods enabled by the nuScenes challenge.

  • Takeaways & Limitations

    nuScenes supports object detection, tracking, and behavior-modeling research across varied conditions while encouraging radar and sensor-fusion studies.

  • Takeaways & Limitations

    The dataset assumes that annotations sampled at 2Hz can be robustly interpolated to finer temporal resolutions such as 10Hz or 20Hz.

Abstract

from arXiv · show

Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

1. Introduction

nuScenes addresses the need for multimodal, large-scale autonomous-driving data by combining complementary sensors, diverse conditions, and map information for 3D scene understanding. It contributes a holistic benchmark with new detection and tracking metrics, baselines, and openly available standardization resources.

  • Motivation: Multimodal sensing is essential because cameras, lidar, and radar provide complementary information and have different failure modes in difficult conditions.Cameras support image-plane classification and localization, while lidar provides accurate 3D localization; joint treatment can add redundancy against failures and adverse conditions.
  • Motivation: Existing autonomous-vehicle datasets primarily provide 2D annotations or lack the full sensor suite, limiting 3D research across complex and unseen environments.The introduction identifies a need to study generalization across countries, lighting, driving directions, road markings, vegetation, precipitation, and unseen object types.
  • Dataset contribution: nuScenes provides large-scale 360◦ coverage across the complete vision and range-sensor suite, diverse situations, and map information for autonomous-vehicle scene understanding.It is described as the main contribution and as a holistic benchmark supporting multiple tasks, with nighttime and rainy data, object attributes, and scene descriptions.
  • Methods and evaluation: The work introduces autonomous-driving detection and tracking metrics, baseline 3D detectors and trackers, and a multiple-lidar-sweep approach for enhancing object detection.The paper also presents and analyzes results from the nuScenes object detection and tracking challenges.
  • Resources: The authors release the devkit, evaluation code, taxonomy, annotator instructions, and database schema under a license permitting non-commercial research use.These resources are intended to support industrywide standardization and compatibility across datasets.

2. The nuScenes dataset

The nuScenes dataset is built from diverse urban drives in Boston and Singapore, with 1,000 selected 20-second scenes and expert-validated multimodal annotations. Its 40,000 keyframes cover varied locations, weather, lighting, object classes, distances, and orientations.

  • Scene selection: 1,000 manually selected scenes each last 20s and target dense traffic, rare classes, dangerous situations, maneuvers, and difficult AV scenarios.Selection also encourages diversity in spatial coverage and driving situations.
  • Data annotation: Annotations are sampled at 2Hz for 23 object classes, recording semantic categories, visibility, activity, pose, and 3D cuboids.Cuboids encode x, y, z, width, length, height, and yaw angle; objects are annotated continuously when covered by at least one lidar or radar point.
  • Annotation statistics: 40k keyframes span four locations, with Boston contributing 55%, SG-OneNorth 21.5%, SG-Queenstown 13.5%, and SG-HollandVillage 10%.The keyframes include varied weather and lighting conditions, averaging 7 pedestrians and 20 vehicles per keyframe.
  • Annotation statistics: Car annotations range as far as 80m from the ego-vehicle, with orientations concentrated at vertical and horizontal angles; boxes contain up to 100 lidar points.The dataset’s spatial coverage is concentrated mainly at intersections.

3. Tasks & Metrics

nuScenes defines multimodal detection and tracking tasks with distinct temporal inputs and metrics tailored to 3D object quality. Detection combines center-distance AP with native-unit true-positive errors and a composite detection score, while tracking uses recall-aware and traditional metrics plus novel trajectory measures.

  • Task setup: Detection uses the preceding 0.5 seconds of sensor data, whereas tracking uses all data from scene start through time t.The detection task operates on [t−0.5, t], while tracking operates on [0, t].
  • Detection: The detection task predicts 3D boxes, attributes, and velocities for 10 object classes, a subset of nuScenes’ 23 annotated classes.The task requires detecting 10 classes with 3D bounding boxes, attributes, and velocities.
  • Detection metrics: Average Precision matches predictions using ground-plane center distance rather than IOU, averages thresholds D = {0.5, 1, 2, 4} meters, and removes operating points below 10% recall or precision.This distance-based matching decouples detection from object size and orientation; AP is the normalized precision–recall area over the retained region.
  • Detection metrics: True-positive metrics use 2m center-distance matching and report native-unit errors for translation, scale, orientation, velocity, and attributes.ATE is measured in meters, ASE as 1 − IOU, and AOE in radians; some measurements are omitted where classes lack meaningful definitions.
  • Detection metrics: The nuScenes detection score combines mean Average Precision with five mean True Positive metrics, assigning half the score to detection performance and half to box and attribute quality.Metrics exceeding 1 are bounded between 0 and 1 before inclusion.
  • Tracking: Tracking excludes static classes and uses recall-averaged AMOTA and AMOTP, 40-point interpolated sAMOTA, traditional tracking metrics, and novel TID and LGD measures.The traditional metrics include MOTA, MOTP, false alarms, trajectory coverage, false positives, false negatives, identity switches, and fragmentations.

4. Experiments

The experiments establish detection and tracking baselines across lidar and camera modalities, then analyze dataset scale, matching, sensor importance, temporal aggregation, pretraining, and detection–tracking relationships. Results show that modality rankings depend on object class and matching function, while stronger detections generally support better tracking.

  • Detection and tracking baselines: Baselines cover lidar-only and image-only detection, plus camera- and lidar-based tracking using detections from Megvii, PointPillars, and MonoDIS.PointPillars accumulates lidar sweeps and predicts velocities; tracking uses the approach described in [77].
  • Detection and tracking baselines: MonoDIS was the strongest image-only submission and outperformed the image baseline and some lidar methods, with importance sampling highlighting class imbalance.MonoDIS uses a disentangled 2D and 3D detection loss.
  • Matching function: 2m center-distance matching ranks MonoDIS clearly ahead, whereas IOU matching leaves small-object pedestrian and bicycle performance below 0 AP.The matching function also switches the bicycle ordering to favor MonoDIS over both lidar methods.
  • Multiple lidar sweeps: 10 previous lidar sweeps are available within the 0.5s protocol window, and accumulating multiple sweeps significantly improves PointPillars detection performance.Accumulation moves pointclouds into the keyframe coordinate system and appends them.
  • Sensor importance: PointPillars was stronger for cars at 68.4% vs. 47.8% AP and pedestrians at 59.7% vs. 37.0% AP, while MonoDIS led bicycles at 24.5% vs. 1.1% AP and cones at 48.7% vs. 30.8% AP.The reported differences reflect lidar’s difficulty with thin or small objects and image detection’s advantages for cones.
  • Detection and tracking: Megvii, PointPillars, and MonoDIS achieve validation sAMOTA values of 17.9%, 3.5%, and 4.5%, respectively, with AMOTP values of 1.50m, 1.69m, and 1.79m.The experiments analyze whether better detection performance implies better tracking performance on nuScenes.

5. Conclusion

The paper presents nuScenes as a multimodal autonomous-driving dataset with a full 360° sensor suite, alongside detection and tracking tasks, metrics, baselines, and results. It also introduces a detection metric intended to balance all aspects of detection performance.

  • nuScenes is the first dataset collected from an autonomous vehicle approved for public-road testing and containing lidar, images, and radar with full 360° coverage.
  • nuScenes has the largest collection of 3D bounding-box annotations among previously released datasets.
  • The paper presents nuScenes detection and tracking tasks, metrics, baselines, and results.
  • A new detection metric is introduced to balance all aspects of detection performance and support research on 3D object detection for autonomous vehicles.

nuScenes: A multimodal dataset for autonomous driving Supplementary Material · A. The nuScenes dataset

The supplementary material details nuScenes’ sensor calibration, privacy protection, data representation, class mapping, annotation distributions, and qualitative scene reconstruction. It describes calibration procedures for each sensor, automated redaction across 1.4M images, relational storage of annotations and metadata, and analyses of object and lidar-point distributions.

  • A. The nuScenes dataset: Calibration parameters are updated around twice per week during the six-month collection period to maintain high-quality multi-sensor data.The supplementary material describes intrinsic and extrinsic calibration as necessary for the dataset’s quality.
  • A. The nuScenes dataset: Lidar, camera, and radar extrinsics are calibrated using measured sensor geometry, planar target alignment, and radar range-rate minimization, respectively.Lidar-to-ego location is measured with a laser liner; camera-to-lidar transformation uses a cube-shaped target; radar yaw is optimized on static-object returns.
  • A. The nuScenes dataset: Camera intrinsic and distortion parameters are inferred from a calibration target board with known patterns.This procedure is distinct from the camera extrinsic calibration based on aligning orthogonal target planes.
  • A. The nuScenes dataset: Predicted vehicle boxes from Faster R-CNN with a ResNet-101 backbone trained on Cityscapes are used to blur faces and license plates across 1.4M images.The automated approach addresses the prohibitive cost of manually labeling faces and license plates.
  • A. The nuScenes dataset: Annotations and metadata, including localization, timestamps, and calibration data, are stored in a relational database to avoid redundancy and enable efficient access.The devkit, taxonomy, and annotation instructions are available online.
  • A. The nuScenes dataset: nuScenes annotations cover 23 classes, with cars and adults most frequent and ambulances least frequent; selected attributes show category-specific motion patterns.Construction vehicles are rarely moving, pedestrians are rarely sitting, and buses are commonly moving.
  • A. The nuScenes dataset: Annotation density is generally higher near the ego-vehicle, while pedestrians and bicycles have fewer annotations beyond 100m and cars extend to 200m.The polar density maps are log-scaled and show distributions up to 150m for all categories, with car annotations reaching 200m.
  • A. The nuScenes dataset: Cars contain more lidar points inside their boxes than pedestrians and bicycles across varying distances because they have larger, more reflective surfaces.The supplementary analysis uses log-scaled occurrence bins stratified by category.

B. Implementation details

The baselines use sensor-specific spatial discretizations and training configurations. PointPillars varies pillar capacity and batch size with lidar sweeps, while OFT defines camera-frame voxel ranges and training filters constrain annotations by distance and visibility.

  • PointPillars implementation details: PointPillars uses 0.25-meter pillar xy resolution over an x and y range of [−50, 50] meters.All experiments were trained for 750 epochs with an initial learning rate of 10−3.
  • OFT implementation details: OFT trains each camera on a camera-frame voxel grid spanning lateral [−40, 40] meters, longitudinal [0.1, 50.1] meters, and vertical (−3, 1) meters.
  • Training data and optimization: Training annotations were limited to within 50 meters of the ego-frame origin and filtered to visibility of at least 40%.The network used 60 epochs, a learning rate of 2 × 10−3, and random initialization without ImageNet pretraining.

C. Experiments

Experiments analyze nuScenes detection across challenging conditions and classes, semantic-map filtering, and tracking. Results show strong gains from map-aware evaluation and Mahalanobis-distance matching, alongside substantial variation across object categories and modalities.

  • Detection performance: 45.3% NDS for PointPillars exceeded 38.4% for MonoDIS on the test set.Table 7 reports detailed detection metrics for both methods, including average precision and localization, scale, orientation, velocity, and attribute errors.
  • Per-class performance: 21° pedestrian orientation error exceeded 11° for cars, despite similar translational errors.Bicycles and construction vehicles were the worst-performing categories and among the rarest, while construction vehicles also vary substantially in size and shape.
  • Semantic map filtering: 30% AP on bicycles after semantic-map filtering improved over the PointPillars baseline’s 1% AP.Both predictions and ground truth were restricted to boxes on the semantic map, defined as the union of roads and sidewalks; performance decreased farther from the map prior.
  • Tracking challenge results: +40% sAMOTA enabled Stan’s Mahalanobis-distance matching to significantly outperform the strongest baseline and establish a new nuScenes tracking benchmark state of the art.These results come from the 2019 nuScenes tracking challenge; the cited table reports test-set tracking results for multiple submissions using AB3DMOT.
Loading 1903.11027v5…