PaperlayerView exact revision on arXiv

Source-linked AI summary

Zenseact Open Dataset: A large-scale and diverse multimodal dataset for autonomous driving

Mina Alibeigi, William Ljungbergh, Adam Tonderski, Georg Hess, Adam Lilja, Carl Lindstrom, Daria Motorniuk, Junsheng Fu, Jenny Widahl, Christoffer Petersson

arXiv:2305.02008v2cs.CVcs.LGcs.RO

TL;DR

Existing AD datasets can have limited diversity and long-range capabilities because correlated scenes constrain coverage across conditions. ZOD combines geographically diverse data, high-resolution sensors, detailed annotations, and multiple temporal formats. It covers an area 9× larger than any other AD dataset, while its annotations extend to long-range objects.

  • Problem

    Existing AD datasets often emphasize temporal reasoning and 360° perception, while correlated scenes limit diversity across conditions and geography.

    Source: §1 · p. 2

  • Method

    ZOD organizes its multimodal collection into 100k independent Frames, 1473 twenty-second Sequences, and 29 Drives lasting a few minutes.

    Source: §1 · p. 2

  • Results

    9× larger than any other AD dataset in area, ZOD pairs high-resolution sensors with detailed annotations, including 2D/3D objects extending to 245 meters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ZOD’s data formats and annotations support perception, spatio-temporal learning, prediction, localization, mapping, and planning across multiple operational domains.

    Source: §1 · p. 2

    Keyframe annotations do not provide consistent object tracking over time.

    Source: §6 · p. 9

Abstract

from arXiv · show

Existing datasets for autonomous driving (AD) often lack diversity and long-range capabilities, focusing instead on 360{\deg} perception and temporal reasoning. To address this gap, we introduce Zenseact Open Dataset (ZOD), a large-scale and diverse multimodal dataset collected over two years in various European countries, covering an area 9x that of existing datasets. ZOD boasts the highest range and resolution sensors among comparable datasets, coupled with detailed keyframe annotations for 2D and 3D objects (up to 245m), road instance/semantic segmentation, traffic sign recognition, and road classification. We believe that this unique combination will facilitate breakthroughs in long-range perception and multi-task learning. The dataset is composed of Frames, Sequences, and Drives, designed to encompass both data diversity and support for spatio-temporal learning, sensor fusion, localization, and mapping. Frames consist of 100k curated camera images with two seconds of other supporting sensor data, while the 1473 Sequences and 29 Drives include the entire sensor suite for 20 seconds and a few minutes, respectively. ZOD is the only large-scale AD dataset released under a permissive license, allowing for both research and commercial use. More information, and an extensive devkit, can be found at https://zod.zenseact.com

1. Introduction

Existing AD datasets emphasize temporal and 360° perception, but correlated scenes can limit diversity and generalization across real-world conditions. ZOD addresses this gap with geographically diverse data, high-resolution sensors, detailed annotations, and multiple data formats for perception and longer-term tasks.

1. Introduction

  • Correlated scenes in existing AD datasets can limit diversity across weather, lighting, driving situations, and geography, potentially restricting generalization across the operational design domain.The paper identifies this as a limitation of datasets emphasizing temporal reasoning and 360° perception.Source: §1 · p. 2
  • ZOD comprises 100k independent Frames, 1473 twenty-second Sequences, and 29 Drives lasting a few minutes, spanning an area 9× larger than any other AD dataset.Frames target non-temporal perception, Sequences spatio-temporal learning and prediction, and Drives longer-term localization, mapping, and planning.Source: §1 · p. 2
  • ZOD combines an 8MP camera, rooftop LiDAR with 254k points per scan, and high-precision GNSS/IMU with 0.01m position accuracy with object annotations extending to 245 meters.Annotations also include road and lane semantic and instance segmentation, road-condition labels, and a 156-class traffic-sign taxonomy.Source: §1 · p. 2
  • ZOD is released under the permissive CC BY-SA 4.0 license, allowing research and commercial use subject to its terms.The dataset also includes a development kit with tutorials and examples.Source: §1 · p. 2
  • High-resolution sensors and keyframe annotations cover 2D/3D objects, lane instances, and road segmentation, with objects annotated farther away than in comparable AD datasets.The contribution is framed as supporting long-range perception.Source: §1 · p. 2
  • ZOD’s traffic-sign annotations contain more than twice as many unique instances as the largest existing traffic-sign dataset.Source: §1 · p. 2

2. Related work

Prior multimodal datasets emphasize surround-view perception and temporal reasoning, with different trade-offs in diversity, annotation, and sensing. ZOD is presented as complementary through broader geographic coverage, long-range annotations, multimodal sequences and drives, and research and commercial use.

2. Related work

  • Earlier datasets illustrate varied trade-offs: KITTI offers 22 multimodal sequences, while ApolloScape’s temporally connected frames limit training-data diversity and A2D2 has sparse point clouds.A2D2 and PandaSet are also described as having low annotation counts.Source: §2 · p. 3
  • nuScenes provides six cameras, LiDAR, five radars, maps, and GNSS in 20-second sequences with 3D-object annotations at 2 Hz.The passage describes it as an early large-scale public dataset supporting temporal annotations.Source: §2 · p. 3
  • Waymo Open and Argoverse 2 offer broader driving scenarios and higher-resolution sensors than nuScenes, but these datasets trade off against high sample correlation.Waymo is noted for geographical diversity, while Argoverse 2 stands out for long-range annotations, maps, and object taxonomy.Source: §2 · p. 3
  • ZOD’s Frames cover nine times more geography than Waymo Open with one quarter of its frames, complementing existing datasets for long-range perception and multitask learning.The authors also aim to support benchmarking for research and commercial use.Source: §2 · p. 3
  • ZOD’s mostly unlabeled Sequences and Drives include camera, LiDAR, and GNSS/IMU data, addressing sensor and localization gaps in ONCE and Argoverse 2 Lidar.ONCE lacks localization information, while Argoverse 2 Lidar lacks camera data.Source: §2 · p. 4
  • ZOD has 446k labeled traffic-sign instances across 156 classes, compared with MTSD’s 206k instances across 313 classes.The passage describes MTSD as one of the largest and most diverse sign datasets.Source: §2 · p. 4

3. Zenseact Open Dataset

ZOD combines diverse European driving data with high-resolution sensors, multiple annotation types, and three data categories for different temporal and localization tasks. Its Frames have an object-distance distribution similar to Argoverse 2, with less temporal correlation among annotated objects.

3. Zenseact Open Dataset

  • ZOD contains European real-world scenes across highways, urban areas, and country roads, collected in varied weather and lighting conditions.The dataset also provides fine-grained labels for road segmentation, 2D/3D objects, traffic signs, and road classification.Source: §3 · p. 4

3.1. Sensor suite

  • The sensor suite includes high-resolution 8MP wide-angle fish-eye camera imagery, LiDAR point clouds containing around 254k points on average, high-precision GNSS/IMU, and vehicle data released for Sequences and Drives.RGB camera images are captured at 10 Hz; high-precision GNSS/IMU is logged at 100 Hz.Source: §3.1 · p. 5

3.2. Calibration and coordinate systems

  • Sensors are synchronized and calibrated to an ISO-8855 reference frame, with per-datapoint calibrations and devkit functions for transforming and projecting between sensors.The reference axes are X-forward, Y-left, and Z-up.Source: §3.2 · p. 5

3.3. Privacy protection

  • Each Frame includes blurred and DNAT-anonymized images; the two versions support study of anonymization effects, while DNAT preserves information such as pedestrians’ line of sight for compliant human-intent use cases.The authors describe DNAT as generative-AI-based with minimal pixel impact.Source: §3.3 · pp. 5–6

3.4. Data categories

  • Frames provide curated keyframes and nearby sensor data, Sequences provide 20 seconds for temporal tasks, and Drives provide longer scenes for localization and mapping.There are 100k Frames, 1,473 Sequences, and 29 Drives; Drive scenes include varied road structures and traffic scenarios.Source: §3.4 · p. 6

3.5. Annotations

  • Keyframes receive manually created, quality-checked labels for lane and road segmentation, 2D/3D object boxes, and road-surface classification.Object labels use hierarchical taxonomies, including 156 granular traffic-sign classes.Source: §3.5 · pp. 6–7
  • For distances from the ego-vehicle across all annotated objects, ZOD Frames has a distribution on par with Argoverse 2, and its annotated objects are less temporally correlated.Source: Figure 6 · p. 7

4. Dataset analysis

Section 4 examines ZOD’s geographic and driving-condition diversity, long-range perception, long-tailed detection and traffic-sign recognition, and image anonymization. It reports broad dataset coverage alongside baseline weaknesses at extended distances and on rare classes, while anonymized-image training shows no statistically significant detection degradation.

4.1. Diversity

  • 705km2 for ZOD Frames’ diversity area, compared with 5km2 for nuScenes, 17km2 for Argoverse 2, and 76km2 for Waymo Open.The paper defines this metric as the union of all 75m-radius diluted ego-poses.Source: §4.1 · p. 7
  • ZOD Frames cover varied weather, road types, and times of day, including almost 20k nighttime scenes.Source: §4.1 · p. 8

4.2. Long-range perception

  • ZOD annotates 3D objects up to 245 meters; its distance distribution resembles Argoverse 2, except for a longer vehicle tail and fewer distant vulnerable vehicles.Source: §4.2 · p. 8
  • CenterPoint struggles significantly at extended ranges in the range-bin mAP and CDS evaluation, highlighting a need for improved detectors or more relevant metrics.Source: §4.2 · p. 8, Table 4 · p. 8

4.3. Long-tail perception

  • Nearly half of the analyzed classes have under 10% AP across the full 250m distance, despite consistently detected cars.Source: §4.3 · pp. 8–9, Figure 7 · p. 9
  • Traffic-sign common classes perform well by F1-micro, while class-balanced F1-macro is significantly lower, highlighting difficulty with rare classes.Source: Table 5 · p. 8, §4.3 · p. 9

4.4. Anonymization

  • Training Faster-RCNN and YOLOv7 on blurred or DNAT images shows no statistically significant detection-performance degradation when evaluated on original images.Source: Table 3 · p. 8, §4.4 · p. 9

5. Conclusion

The conclusion presents ZOD as a diverse, high-resolution multimodal dataset with dense annotations and broad European coverage. It also identifies planned annotation and dataset expansions.

  • ZOD combines data from 14 European countries with dense road and object annotations, including 3D object labels up to 245 meters.Source: §5 · p. 9
  • The dataset includes a comprehensive traffic-sign taxonomy with more unique instances than comparable datasets.Source: §5 · p. 9
  • Planned additions include temporally consistent Sequence annotations, two seconds of supporting camera frames for every Frame, and more Sequences and Drives with high-definition maps.Source: §5 · p. 9

6. Limitations

The limitations section identifies gaps in temporal tracking, evaluation of anonymization beyond 2D detection, long-range 3D annotation quality, and annotation availability.

  • Keyframe annotations do not support consistent object tracking over time; full-sequence annotations are planned.Source: §6 · p. 9
  • The anonymization impact has been analyzed only for 2D object detection, leaving other tasks unexplored.Source: §6 · p. 9
  • 3D annotation quality and recall wane near the 250-meter limit, while 2D annotations extend much farther.Source: §6 · p. 9
  • Traffic-sign and ego-road annotations are unavailable for some Frames.Source: §6 · p. 9

8. Supplementary material

The supplementary material details Drive coverage, annotation formats and counts, additional anonymization and compression experiments, and a panoptic-segmentation benchmark.

8.1. Drives

8.2. Annotation details

8.3. Additional dataset statistics

8.4. Additional anonymization results

  • The supplementary anonymization results cover Faster-RCNN training and evaluation across all possible combinations of anonymization methods.Source: §8.4 · p. 13, Table 6 · p. 13

8.5. Compression results

  • Faster-RCNN trained on JPG images shows no significant performance degradation versus PNG-trained models when both are evaluated on PNG images.Source: §8.5 · p. 13, Table 7 · p. 13

8.6. Panoptic segmentation benchmark

  • The Panoptic-DeepLab benchmark reports PQ, SQ, and RQ, with lanes treated as things and ego-road and crosswalks as stuff.Source: §8.6 · p. 13, Table 8 · p. 13

Related papers

Loading 2305.02008v2…