Source-linked AI summary
One Million Scenes for Autonomous Driving: ONCE Dataset
Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Jie Yu, Hang Xu, Chunjing Xu
TL;DR
Autonomous-driving perception needs more diverse scene data and better ways to exploit the large volume of unlabeled data collected in practice. ONCE introduces a million-scene multimodal dataset and benchmarks self-/semi-supervised 3D detection methods, showing benefits from scale and dataset quality.
Problem
Autonomous-driving research lacks sufficient diverse scene data and effective ways to exploit the large unlabeled datasets collected in practice.
Method
The paper introduces the ONCE dataset and benchmarks self-supervised, semi-supervised, and unsupervised domain-adaptation methods for 3D object detection.
Results
Self-supervised detection improves from 50.75% mAP with 100k unlabeled data to 52.99% with one million, a 1.09% gain over baseline.
Takeaways & Limitations
ONCE provides large-scale, diverse data and benchmark evidence for studying unlabeled-data learning in autonomous-driving 3D detection.
Abstract
from arXiv · showhide
Current perception models in autonomous driving have become notorious for greatly relying on a mass of annotated data to cover unseen cases and address the long-tail problem. On the other hand, learning from unlabeled large-scale collected data and incrementally self-training powerful recognition models have received increasing attention and may become the solutions of next-generation industry-level powerful and robust perception models in autonomous driving. However, the research community generally suffered from data inadequacy of those essential real-world scene data, which hampers the future exploration of fully/semi/self-supervised methods for 3D perception. In this paper, we introduce the ONCE (One millioN sCenEs) dataset for 3D object detection in the autonomous driving scenario. The ONCE dataset consists of 1 million LiDAR scenes and 7 million corresponding camera images. The data is selected from 144 driving hours, which is 20x longer than the largest 3D autonomous driving dataset available (e.g. nuScenes and Waymo), and it is collected across a range of different areas, periods and weather conditions. To facilitate future research on exploiting unlabeled data for 3D detection, we additionally provide a benchmark in which we reproduce and evaluate a variety of self-supervised and semi-supervised methods on the ONCE dataset. We conduct extensive analyses on those methods and provide valuable observations on their performance related to the scale of used data. Data, code, and more information are available at https://once-for-auto-driving.github.io/index.html.
1 Introduction
The introduction identifies scarce, insufficiently diverse autonomous-driving scene data and the challenge of exploiting abundant unlabeled data. ONCE addresses both gaps with a large, diverse dataset and a benchmark for self-/semi-supervised 3D detection.
- 3D object detection identifies and localizes surrounding vehicles and humans from LiDAR point clouds and camera images.
- Autonomous-driving data is scarce and lacks diversity because collection requires expensive vehicle-mounted sensors and regulatory-compliant road driving.
- Unlabeled data accumulates rapidly because vehicles can collect more than 200k point-cloud frames in eight hours, while workers annotate only 100–200 frames per day.
- ONCE contains 1 million 3D scenes and 7 million images from 144 driving hours, with varied weather, traffic conditions, time periods, and areas.
- The dataset annotates 16k scenes with 3D boxes across five categories, while the paper also contributes a self-/semi-supervised 3D detection benchmark.
- Table 1 reports that ONCE has 4x scenes, 7x images, and 20x driving hours compared with the largest dataset.
2 Related Work
Related work covers autonomous-driving datasets, 3D detection methods, and learning from unlabeled data. ONCE extends this landscape with larger, more diverse sensing data and a benchmark comparing self-/semi-supervised approaches.
- Existing autonomous-driving datasets collect multimodal road data for perception tasks such as detection and tracking, with examples spanning LiDAR, cameras, thermal imaging, and maps.
- nuScenes records 5.5 driving hours and 400k 3D scenes, while Waymo Open offers 200k scenes from 6.4 driving hours.
- ONCE is larger and more diverse than nuScenes and Waymo, with 1M scenes and 144 driving hours covering all time periods and most weather conditions.
- Driving 3D detectors include single-modality point-cloud methods and multimodal approaches that combine point clouds with camera images.
- Figure 2 depicts a vehicle equipped with 1 LiDAR and 7 cameras providing a 360◦ field of view.
- Semi-supervised methods use pseudo-labeling, teacher models, or consistency constraints, whereas self-supervised methods use auxiliary tasks, clustering, or contrastive learning.
3 ONCE Dataset
ONCE combines large-scale, diverse multimodal driving data with explicit metadata, annotations, and benchmark infrastructure for 3D detection research. Its design supports evaluating data quality, diversity, and methods that exploit unlabeled scenes.
- Data collection: The acquisition system uses one 40-beam LiDAR and seven synchronized cameras, both covering a 360° horizontal field of view.The sensors are mounted on a car and enable cross-modality alignment.
- Data format: The dataset provides LiDAR point clouds, seven million JPEG camera images, and 3D boxes for five object categories.Each point is represented by (x, y, z, r), while each 3D box uses center, dimensions, and yaw parameters.
- Metadata: Weather, time period, timestamp, pose, calibration, and annotations are packaged per scene, with labels covering three weather conditions and four time periods.The metadata includes sunny, cloudy, and rainy conditions, plus morning, noon, afternoon, and night.
- Dataset splits: The labeled splits contain 5k training, 3k validation, and 8k testing scenes across sequences covering downtown and suburban areas.Validation and testing sequences include sunny and rainy conditions during daytime and nighttime.
- Unlabeled data: The remaining 560 sequences form unlabeled subsets of 100k, 500k, and about 1M scenes for self-supervised and semi-supervised learning.Usmall ⊂ Umedium ⊂ Ularge, and the smaller subsets are selected in time order from particular roads to reflect incremental data updates.
- Dataset analysis: Quality analysis compares identical SECOND backbones pretrained with DeepCluster on nuScenes, Waymo, and ONCE, while diversity analysis compares weather, time, area, and road-type coverage.ONCE contains 20% rainy scenes and 6% nighttime scenes, compared with 10% and 12% rainy scenes in Waymo and nuScenes, respectively; its pretrained model shows superior downstream performance.
- Benchmark: The benchmark reproduces six detection, four self-supervised, five semi-supervised, and two unsupervised domain adaptation methods for 3D detection.It evaluates these methods and analyzes their results.
4 Benchmark for 3D Object Detection
The benchmark evaluates standard 3D detectors and learning methods for exploiting unlabeled data on ONCE, revealing benefits from richer representations and larger unlabeled sets.
- Benchmark scope: The benchmark reproduces detection, self-supervised, semi-supervised, and unsupervised domain-adaptation methods under a unified evaluation standard.It includes widely used models and methods adapted to autonomous-driving 3D detection.
- 3D object detection: 53.85% mAP is achieved by PV-RCNN, outperforming voxel-based SECOND at 51.90% and pillar-based PointPillars at 45.47%.The results associate finer geometric information in voxels and combined point-voxel representations with stronger detection performance.
- 3D object detection: CenterPoints performs better on pedestrians and cyclists, whereas SECOND performs better on vehicles.The reported category results are 51.80% versus 26.09% for pedestrians, 65.57% versus 59.92% for cyclists, and 69.71% versus 66.35% for vehicles.
- 3D object detection: PointPainting reduces performance from 61.24% to 59.78% because its effectiveness depends on accurate image-based segmentation scores.The dataset lacks explicit segmentation labels, limiting the quality of semantic maps used by PointPainting.
- Self-supervised learning: 54.28% and 54.27% mAP are achieved by SwAV and DeepCluster, exceeding BYOL at 52.10% and PointContrast at 52.99% on Ularge.The analysis attributes the advantage of clustering methods partly to the difficulty of constructing representative contrastive views in driving scenes.
- Semi-supervised learning: 59.99% mAP is reached by Mean Teacher on Ularge, while SESS gains increase from 1.93% with 100k scenes to 6.88% with one million scenes.Most semi-supervised methods improve detection, and performance generally increases with more unlabeled data.
- Cross-method analysis: Semi-supervised methods generally outperform self-supervised methods, while domain adaptation improves over Source Only but remains below Oracle results.Mean Teacher reaches 59.99% mAP versus 54.28% for the best self-supervised method; ST3D gains 15.46% AP3D on Waymo →ONCE but performs poorly on nuScenes →ONCE.
5 Conclusion
ONCE provides a large autonomous-driving dataset and benchmarks self-supervised, semi-supervised, and unsupervised domain-adaptation methods for 3D object detection.
- Conclusion: ONCE is introduced as an autonomous-driving dataset together with benchmarks for detection, self-supervised learning, semi-supervised learning, and unsupervised domain adaptation.The authors plan to support additional tasks including 2D detection, 3D semantic segmentation, and planning.
A The ONCE dataset
The dataset is distributed with documentation, download and benchmark resources, diverse scene statistics, and a stated limitation in annotation coverage.
- Resources: The dataset, documentation, download links, benchmark results, data format, and annotation instructions are published through dedicated project resources.Data are hosted on Google Drive and BaiduYunPan, with ongoing maintenance planned.
- License: The ONCE dataset is released under CC BY-NC-SA 4.0 for non-commercial research use.The paper directs users to separate terms-of-use, privacy, and license information.
- Dataset statistics: Vehicle counts range from less than 1 to more than 50 across annotated scenes, while pedestrian and cyclist counts range from less than 1 to more than 30.Training, validation, and testing distributions are mostly similar but differ slightly in some intervals.
- Limitations: Only a small amount of the one million scenes is currently annotated, limiting broader exploration of 3D object detection.The authors plan to provide more annotations and support additional autonomous-driving tasks.
- Dataset statistics: 6% of scenes were captured on rainy days and 20% were collected at night.Figure 4 summarizes proportions across weather, time, and area domains.
B.1 Models for 3D Object Detection
The validation split reports detection-model results for the ONCE benchmark, but the supplied passage provides only the table reference.
- Validation results: Detection-model results are reported on the ONCE validation split in Table 9.The supplied passage does not include the table’s model names or metric values.
B.2 Self-Supervised Learning for 3D Object Detection
This section presents validation results for self-supervised learning methods.
- Table 10 reports results for self-supervised learning methods.
- The reported results use the validation split.
- The table is associated with the self-supervised learning evaluation section.
B.3 Semi-Supervised Learning for 3D Object Detection
This section presents validation results for semi-supervised learning methods.
- Table 11 reports results for semi-supervised learning methods.
- The reported results use the validation split.
- The table is associated with the semi-supervised learning evaluation section.
C Implementation details
The implementation evaluates multiple 3D detectors and self-supervised or semi-supervised learning schemes using specified data splits, optimization settings, augmentations, and model configurations.
- Models and data splits: Six detector models are trained on the training split and evaluated on validation and testing splits.The models include five single-modality detectors and PointPainting, which uses point clouds and images.
- Detector training: All six detector models use Adam, cosine annealing, an initial learning rate of 0.003, batch size 32, and 80 training epochs.
- Detector training: Detector augmentation includes axis flips, rotations from −45° to +45°, scaling from 0.95 to 1.05, and object cut-and-paste.PointPainting does not apply these augmentations to segmentation maps.
- Models and data splits: The detector implementations cover point-, pillar-, voxel-, point-voxel-, center-, and point-image-based designs.PointRCNN uses point clouds directly; PointPillars uses pillars; SECOND uses voxels; PV-RCNN combines voxels and keypoints; CenterPoints uses center-based assignments; PointPainting combines LiDAR and image semantics.
- Self-supervised learning: Self-supervised experiments pretrain SECOND on 100k, 500k, or 1 million unlabeled scenes before fine-tuning on the training split.The reproduced methods include PointContrast, BYOL, DeepCluster, and SwAV.
- Semi-supervised learning: Semi-supervised experiments use SECOND with pretrained teacher and student models, unlabeled data, and labeled training data.The methods include pseudo-label training, Mean Teacher with EMA-updated teachers, and Noisy Student self-training; training uses method-specific labeled and unlabeled batch sizes and durations.
E Evaluation Metric
The evaluation metric extends 3D average precision by incorporating object orientation, while reporting category- and distance-specific performance.
- Metric design: The proposed metric re-ranks predictions and accounts for object orientations when matching predictions to ground truth.It addresses the issue that opposite orientations can otherwise both match under an IoU-based criterion.
- Reported metrics: The dataset merges car, bus, and truck into the vehicle superclass and reports AP Ori3D for vehicle, pedestrian, and cyclist.Mean AP averages the scores of these three categories, while a five-class evaluation interface remains available.
- Reported metrics: Performance is additionally reported for ground truths and predictions within 30m, 30–50m, and beyond 50m.
- Metric design: Compared with distance-based matching, the metric assigns equal weights to object size, center, and orientation.Compared with the weighted-scoring method, it avoids repeated orientation-factor calculations because orientation already participates in rotated 3D IoU.