PaperlayerView exact revision on arXiv

Source-linked AI summary

SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving

Jianhua Han, Xiwen Liang, Hang Xu, Kai Chen, Lanqing Hong, Jiageng Mao, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Xiaodan Liang, Chunjing Xu

arXiv:2106.11118v3cs.CV

TL;DR

Limited annotation capacity and gaps in the scale and diversity of existing driving datasets motivate SODA10M, a large unlabeled-plus-labeled dataset and benchmark for self/semi-supervised learning. Its representations rank first in 9 of 10 downstream detection and segmentation tasks among compared driving datasets, although some contrastive methods perform equivalently or slightly worse than ImageNet pre-training.

  • Problem

    Autonomous-driving detection faces a gap between rapidly accumulating unlabeled images and the time-intensive labeling process, alongside a lack of suitable large-scale driving data and tailored evaluation.

    Source: §1 · pp. 1–2

  • Method

    The paper releases SODA10M, a large-scale autonomous-driving dataset and benchmark with unlabeled images and a smaller labeled detection set for self/semi-supervised learning.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    SODA10M pre-training ranks first in 9 of 10 downstream detection and segmentation tasks among compared driving datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SODA10M can serve as a pre-training dataset for self-supervised methods evaluated on downstream autonomous-driving tasks.

    Source: §1 · p. 2, §5 · p. 10

    The tested contrastive methods on SODA10M perform equivalently or slightly worse than ImageNet pre-training, and the authors say contrastive losses may need adaptation to complex driving scenes.

    Source: §4.3.1 · p. 8

Abstract

from arXiv · show

Aiming at facilitating a real-world, ever-evolving and scalable autonomous driving system, we present a large-scale dataset for standardizing the evaluation of different self-supervised and semi-supervised approaches by learning from raw data, which is the first and largest dataset to date. Existing autonomous driving systems heavily rely on `perfect' visual perception models (i.e., detection) trained using extensive annotated data to ensure safety. However, it is unrealistic to elaborately label instances of all scenarios and circumstances (i.e., night, extreme weather, cities) when deploying a robust autonomous driving system. Motivated by recent advances of self-supervised and semi-supervised learning, a promising direction is to learn a robust detection model by collaboratively exploiting large-scale unlabeled data and few labeled data. Existing datasets either provide only a small amount of data or covers limited domains with full annotation, hindering the exploration of large-scale pre-trained models. Here, we release a Large-Scale 2D Self/semi-supervised Object Detection dataset for Autonomous driving, named as SODA10M, containing 10 million unlabeled images and 20K images labeled with 6 representative object categories. To improve diversity, the images are collected within 27833 driving hours under different weather conditions, periods and location scenes of 32 different cities. We provide extensive experiments and deep analyses of existing popular self/semi-supervised approaches, and give some interesting findings in autonomous driving scope. Experiments show that SODA10M can serve as a promising pre-training dataset for different self-supervised learning methods, which gives superior performance when fine-tuning with different downstream tasks (i.e., detection, semantic/instance segmentation) in autonomous driving domain. More information can refer to https://soda-2d.github.io.

1 Introduction

The introduction argues that rapid image collection and limited annotation capacity make large-scale unlabeled data important for autonomous-driving detection. SODA10M pairs 10 million road images with a benchmark and reports strong downstream transfer, while findings indicate that self-supervised methods behave differently on driving data than on ImageNet.

1 Introduction

  • 288K images can be captured in eight hours, whereas exhaustive box annotation takes about 136 days for 10 experienced annotators.The acquisition–annotation gap motivates using large-scale unlabeled scenes alongside limited labeled data.Source: §1 · pp. 1–2
  • 10 million road images make SODA10M substantially larger than existing autonomous-driving datasets, with 32 cities and 27,833 driving hours.The paper describes the dataset as diverse across cities, seasons, weather, and driving scenes.Source: §1 · p. 2
  • SODA10M representations rank first in 9 of 10 downstream detection and segmentation tasks among compared driving datasets.The paper presents the dataset as a pre-training resource and benchmark for self/semi-supervised learning.Source: §1 · p. 2
  • DenseCL exceeds MoCo-v1 after ImageNet pre-training, 39.9% vs. 39.0%, but trails it after autonomous-driving pre-training, 38.1% vs. 38.9%.The authors suggest pixel-wise contrastive loss may not suit complex driving scenes.Source: §1 · p. 3
  • The paper contributes a self/semi-supervised autonomous-driving benchmark and reports experiments and analyses of prevailing approaches.The dataset was released and used for the ICCV2021 SSLAD challenge, which attracted more than 130 teams and 500+ submissions.Source: §1 · p. 3

2 Related Work

Related work covers driving datasets, object-detection architectures, and self- and semi-supervised learning approaches. It contrasts general-purpose representation-learning methods with methods that use unlabeled data for detection.

2 Related Work

  • Driving datasets span object-specific detection, instance segmentation, diverse multitask data, and sensor-rich 3D detection and mapping resources.The passage discusses Cityscapes, BDD100K, KITTI, Waymo Open Dataset, and nuScenes.Source: §2 · p. 3
  • Detection methods include one-stage predictors, two-stage RPN-based models, Cascade R-CNN, and transformer-based detectors.Two-stage methods send region proposals to classification and box-regression heads.Source: §2 · p. 3
  • Self-supervised learning includes reconstruction-style pretext tasks and contrastive methods such as MoCo, SimCLR, SwAV, and BYOL.The cited methods vary in their use of negative samples, augmentations, clustering, and prediction objectives.Source: §2 · pp. 3–4
  • Semi-supervised detection uses labeled and unlabeled images through self-training or consistency regularization, with STAC and Unbiased Teacher as examples.STAC uses pseudo-labels to fine-tune a detector, while Unbiased Teacher adds teacher-student mutual learning.Source: §2 · p. 4

3 SODA10M

SODA10M combines a large unlabeled driving-image collection with a smaller labeled detection set, built and checked through a staged collection and annotation process. Its labeled split supports domain adaptation evaluation, while comparisons report broad scale and transfer advantages over other driving datasets.

3 SODA10M

  • SODA10M contains 10M unlabeled images and 20K labeled images split into 5K training, 5K validation, and 10K testing images.The release is intended to support self-supervised and semi-supervised learning research.Source: §3 · p. 4

3.1 Data Collection

  • Images were crowdsourced from taxi drivers, collected every 10 seconds across varied weather, periods, locations, and cities, then sampled to 10M from 100M originals.City and period targets were predefined; other characteristics followed their natural distribution.Source: §3.1 · p. 4

3.2 Data Annotation

  • The annotation pipeline pre-labels images, has annotators correct labels, and uses senior reviewers to check them; six object categories receive tight 2D boxes.The categories are car, truck, pedestrian, tram, cyclist, and tricycle.Source: §3.2 · pp. 4–5

3.3 Data Statistics

  • The 5K-image training set uses clear-weather daytime Shanghai streets, while validation and test sets span three weather conditions, locations, cities, and two periods.The 20K labeled images have disjoint sequence IDs across the train, validation, and test splits.Source: §3.3 · p. 5

3.4 Comparison with Existing Datasets

  • SODA10M has 10M images, 27,833 driving hours, and 32 cities, and performs best in 9 of 10 downstream detection and segmentation tasks as pre-training data.The comparison reports greater scale than BDD100K, nuScenes, and Waymo, and greater driving-condition diversity than nuScenes and Waymo.Source: §3.4 · p. 6

4 Benchmark

The benchmark evaluates supervised, self-supervised, semi-supervised, and video-based approaches on SODA10M, including cross-dataset transfer, domain performance, and training efficiency. Results favor SODA10M in most downstream dataset comparisons, though self-supervised pretraining on it does not consistently surpass ImageNet.

4 Benchmark

  • Experiments use Detectron2 with eight V100 GPUs and 12- or 24-epoch schedules; Figure 3 distinguishes labeled and unlabeled data and shows labeled-data supervised loss in semi-supervised training.Source: §4.1 · p. 6, Figure 3 · p. 6, §4 · p. 6

4.2 Preliminary Supervised Results

  • With a stronger ROI head, Cascade R-CNN exceeds Faster R-CNN by 3.9%, after Faster R-CNN exceeds RetinaNet by 5.3% mAP in the 1x schedule.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3.1 Datasets Comparison

  • SODA10M leads other autonomous-driving pretraining datasets in 9 of 10 downstream tasks; the comparison also suggests diversity matters more than scale.BDD100K performs better than nuScenes and Waymo despite being ten times smaller, which the authors attribute to its greater diversity.Source: §4.3.1 · p. 7, Table 3 · p. 7

4.3.2 Methods Comparison

  • Some contrastive methods improve on SODA10M while others trail ImageNet; DenseCL scores 38.1% versus MoCo-v1’s 38.9% on SODA10M.More pretraining iterations raise SimCLR’s average performance by 1.2% across four downstream tasks.Source: §4.3.2 · p. 8, Table 4 · p. 8

4.3.3 Video-based Self-supervised Methods

  • MoCo-v2 performs better on most downstream tasks than MoCo-v1 and VINCE, while jigsaw-augmented VINCE reaches similar performance to MoCo-v2.The average difference between jigsaw-augmented VINCE and MoCo-v2 is 0.05% over four downstream tasks.Source: §4.3.3 · pp. 8–9, Table 5 · p. 9

4.4 Semi-Supervised Learning Benchmark

  • All semi-supervised methods outperform labeled-only training, and Unbiased Teacher improves over pseudo labeling by 3.4%.Pseudo labeling gains from 50K to 100K unlabeled images but drops 1.4% when expanded from 100K to 500K; STAC improves over pseudo labeling by 2.9%.Source: §4.4 · p. 9, Table 6 · p. 9

4.5.1 Domain Adaptation Results

  • 43.1% daytime vs. 21.1% nighttime performance exposes a domain gap; semi-supervised Unbiased Teacher improves night results by 18.6%, versus 1.6% for MoCo-v1.SODA10M pretraining improves night performance by 1.6% and day performance by 0.7%; ImageNet pretraining improves them by 0.9% and 1.1%, respectively.Source: §4.5.1 · pp. 9–10, Table 7 · p. 10

4.5.2 Performance and Training Efficiency Comparison

  • 46.2% vs. 38.9% mAP and 2.8×8 vs. 8.4×8 GPU days favor Unbiased Teacher over MoCo-v1, despite using 1M vs. 5M unlabeled images.The authors suggest combining self-supervised and semi-supervised methods may improve performance further.Source: §4.5 · p. 9, §4.5.2 · p. 10

5 Conclusion

SODA10M combines extensive unlabeled driving imagery with a smaller, high-quality labeled set and benchmarks self- and semi-supervised learning. The authors report its utility across downstream tasks and identify research directions for these methods.

5 Conclusion

  • SODA10M provides large-scale unlabeled and high-quality labeled driving data, and the paper benchmarks self- and semi-supervised learning on it.The authors describe the dataset as collected across diverse cities, weather, periods, and locations.Source: §5 · p. 10
  • The authors report SODA10M as a promising dataset for training and evaluating self- and semi-supervised methods, with downstream tasks spanning detection and segmentation.They state that it gives superior performance when fine-tuned for different autonomous-driving downstream tasks.Source: §5 · p. 10
  • For future work, the authors identify multi-instance consistency for self-supervised learning, domain adaptation for semi-supervised learning, and efficient training on high-resolution, large-scale images.Source: §5 · p. 10

A SODA10M dataset

SODA10M is presented as a large and diverse autonomous-driving dataset and a benchmark for self- and semi-supervised methods. The authors report strong downstream results while noting that some unlabeled domains lack labeled evaluation data.

A SODA10M dataset

  • The dataset is published under CC BY-NC-SA 4.0, permitting use for non-commercial research.Source: A SODA10M dataset · p. 11
  • Some domains appear in the unlabeled set but not the labeled set, limiting verification of domain-adaptation ability in those domains.The authors plan to add labeled data in those domains.Source: A SODA10M dataset · p. 11
  • DenseCL outperforms MoCo-v1 after ImageNet pre-training (39.9% vs. 39.0%) but underperforms it after SODA10M pre-training (38.1% vs. 38.9%).The authors suggest pixel-wise contrastive loss may not suit complex driving scenarios.Source: A SODA10M dataset · p. 11
  • ImageNet pre-training yields nearly equal reported gains in day and night domains (+1.1% vs. +0.9%), whereas SODA10M pre-training yields a larger night-domain gain (+1.6%) than day-domain gain (+0.7%).The authors attribute this pattern to SODA10M's extensive nighttime imagery and characterize the ImageNet explanation as an assumption.Source: A SODA10M dataset · p. 11
  • SODA10M contains 10M images, 27,833 driving hours, and data from 32 cities, exceeding BDD100K, nuScenes, and Waymo on the reported image, driving-time, and city counts.The compared counts are 0.1M images, 1,111 hours, and 4 cities for BDD100K; 1.4M, 5.5 hours, and 2 cities for nuScenes; and 1M, 6.4 hours, and 3 cities for Waymo.Source: A SODA10M dataset · p. 11
  • As an upstream pre-training dataset, SODA10M achieves the best performance among the compared datasets in 9 of 10 downstream detection and segmentation tasks.The comparison includes BDD100K, nuScenes, and Waymo; the authors also describe SODA10M as more diverse in driving conditions than nuScenes and Waymo and competitive with BDD100K.Source: A SODA10M dataset · pp. 11–12

B Implement Details

The appendix reports common experimental settings for the benchmark and identifies a table listing the codebases used.

B Implement Details

  • Unless otherwise specified, self- and semi-supervised methods use a ResNet50 backbone, with six classes, syncBN enabled, and multi-scale training for supervised and semi-supervised methods.The stated scales are 1920×(864, 907.2, 950.4, 993.6, 1036.8, 1080); experiments use eight Nvidia V100 GPUs and an Intel Xeon Platinum 8168 CPU.Source: B Implement Details · p. 12
  • Experiments use eight Nvidia V100 GPUs with 32GB memory each.Source: B Implement Details · p. 12

B.1 Open-Source Codebase

  • The appendix includes a table of the codebases used in the benchmark.Source: Table 8 · p. 12

B.2 Supervised Methods

The supervised-method appendix specifies the 1x training schedule and points to a table of implementation details.

B.2 Supervised Methods

  • The 1x schedule runs for 12 epochs, with the learning rate reduced tenfold at epochs 8 and 11.The initial learning rate is 0.02.Source: §B.2 · p. 12
  • The 1x schedule uses random crop as its only augmentation and SGD with momentum 0.9.Source: §B.2 · p. 12
  • A table presents implementation details for the supervised-learning benchmark on SODA10M using eight Tesla V100 GPUs.Source: Table 9 · p. 12

B.3 Self-supervised Methods

The benchmark pre-trains image- and video-based self-supervised methods on driving data, then fine-tunes them on SODA10M and other driving datasets. Training schedules and augmentations are adapted to the methods and comparison setting.

B.3 Self-supervised Methods

  • Six image-based methods are pre-trained, generally on 5 million images, with the second 5-million subset optionally used for sequential continued pre-training.The methods are MoCo-v1, MoCo-v2, SimCLR, SwAV, DetCo, and DenseCL; fine-tuning includes SODA10M, BDD100K, and Cityscapes.Source: §B.3 · p. 12
  • Video-based comparisons use MoCo-v1, MoCo-v2, and VINCE, with matched temporal augmentations for the MoCo methods and extra jigsaw augmentation for VINCE.Source: §B.3 · p. 13
  • Pre-training schedules are chosen to approximate the GPU hours of 200 ImageNet epochs for image-based comparisons; video-based methods train for 800 epochs.The image-based schedules are 3700 epochs on BDD100K, 220 on nuScenes, 325 on Waymo, and 60 on SODA unlabeled data; fine-tuning follows the supervised methods and MoCo settings, respectively.Source: §B.3 · p. 13

B.4 Semi-supervised Methods

The semi-supervised benchmark evaluates pseudo-labeling, STAC, and Unbiased Teacher on SODA10M using a one-million-image unlabeled subset. The implementations use method-specific augmentations and SGD.

B.4 Semi-supervised Methods

  • Due to time limits, semi-supervised experiments use 1 million unlabeled SODA10M images, and these methods are described as more efficient than self-supervised methods.Source: §B.4 · p. 13
  • STAC uses color jitter, Gaussian blur, affine transformation, cutout, and random crop; Unbiased Teacher uses color jitter, grayscale, Gaussian blur, random erasing, and random crop.Source: §B.4 · p. 13
  • Both methods use SGD with momentum 0.9, with learning rates of 0.02 for pseudo-labeling, 0.01 for STAC, and 0.02 for Unbiased Teacher.Source: §B.4 · p. 13

C More Experiments

The additional experiments report a Cityscapes segmentation comparison and examine how labeled-set size relates to performance variability. The accompanying domain section describes dataset distributions and driving-condition comparisons, while the terms specify use restrictions.

C More Experiments

D Domain Illustration & Driving Conditions Comparison

F Terms of Use and Licenses

  • The dataset is provided under CC BY-NC-SA 4.0 with additional terms: use is non-commercial, and the dataset or modified versions cannot be distributed, though abstract derivatives may be shared.Source: F Terms of Use and Licenses · pp. 16–17
  • The terms state that reasonable care is taken to scrub PII, including faces and license plates, and prohibit using the dataset to identify or invade anyone's privacy.Source: F Terms of Use and Licenses · p. 17
  • The website and datasets are provided as-is and as-available, without warranties; use is at the user's own risk.Source: F Terms of Use and Licenses · p. 17

Related papers

Loading 2106.11118v3…