PaperlayerView exact revision on arXiv

Source-linked AI summary

BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, Trevor Darrell

arXiv:1805.04687v2cs.CV

TL;DR

Existing autonomous-driving datasets lack comprehensive visual diversity and task coverage for studying multitask learning. BDD100K provides a large, diverse benchmark spanning ten tasks, and evaluations reveal that heterogeneous multitask learning requires special training strategies.

  • Problem

    Existing autonomous-driving datasets have limited scene variation, annotation richness, and geographic distribution for studying diverse multitask learning.

  • Method

    The paper constructs BDD100K, a diverse dataset of over 100K driving video clips with ten tasks and a heterogeneous multitask-learning benchmark.

  • Results

    Evaluations show that existing models require special training strategies to perform heterogeneous tasks jointly.

  • Takeaways & Limitations

    BDD100K supports systematic analysis of multitask learning across driving tasks with varied prediction structures and annotation types.

  • Takeaways & Limitations

    For homogeneous tasks, joint training can provide insignificant gains over single-task training when using larger training sets.

Abstract

from arXiv · show

Datasets drive vision progress, yet existing driving datasets are impoverished in terms of visual content and supported tasks to study multitask learning for autonomous driving. Researchers are usually constrained to study a small set of problems on one dataset, while real-world computer vision applications require performing tasks of various complexities. We construct BDD100K, the largest driving video dataset with 100K videos and 10 tasks to evaluate the exciting progress of image recognition algorithms on autonomous driving. The dataset possesses geographic, environmental, and weather diversity, which is useful for training models that are less likely to be surprised by new conditions. Based on this diverse dataset, we build a benchmark for heterogeneous multitask learning and study how to solve the tasks together. Our experiments show that special training strategies are needed for existing models to perform such heterogeneous tasks. BDD100K opens the door for future studies in this important venue.

1. Introduction

Existing autonomous-driving datasets lack sufficient scene variation, annotation richness, and geographic diversity, while real applications require heterogeneous perception tasks of different complexities. BDD100K addresses these limitations with over 100K diverse annotated driving videos and benchmarks for heterogeneous multitask learning.

  • Motivation: Existing autonomous-driving datasets are limited in scene variation, annotation richness, or geographic distribution, and models trained on them tend to overfit domain characteristics.These limitations make it difficult to leverage deep learning effectively for autonomous driving.
  • Motivation: Real-world applications require heterogeneous perception tasks with different complexities, including detection, segmentation, and tracking.Simple annotations are easier to obtain at scale than instance segmentation or multi-object detection and segmentation tracking.
  • Dataset contribution: BDD100K contains over 100K diverse annotated driving video clips covering realistic scenarios and long-tail variation in appearance and pose.The dataset was created to support algorithmic study on large-scale diverse visual data and multiple tasks.
  • Benchmark and experiments: The paper evaluates existing algorithms across homogeneous, cascaded, and heterogeneous multitask-learning settings.The experiments identify challenges in designing one model to support multiple tasks and show that diverse tasks on one dataset enable new findings.
  • Dataset contribution: BDD100K supports tasks of multiple complexities and serves as an evaluation benchmark for computer vision research in autonomous driving.Its annotations include scene tagging, object bounding boxes, lane markings, drivable areas, segmentation, and tracking.

2. Related Works

Existing visual datasets support representation learning but often provide limited complex visual understanding and driving-condition diversity. BDD100K addresses this gap with annotated benchmarks spanning heterogeneous task types for multitask learning.

  • Visual Datasets: Large-scale general-purpose visual datasets provide millions of images with image-level categorical labels for learning high-level image representations.These datasets are useful for image recognition but do not cover most complex visual understanding tasks.
  • Driving Datasets: Driving datasets may combine multiple data sources, including LiDAR points, yet their diversity across time, locations, and road conditions remains limited.Robust vehicle perception requires varied road conditions across numerous cities.
  • BDD100K: BDD100K contributes video annotations and benchmarks covering pixel-level, region-based, and temporally aware tasks for heterogeneous multitask learning.Its task coverage extends beyond pixel-level or low-dimensional classification and regression.
  • Multitask Learning: Multitask learning seeks to improve generalization by learning from other tasks and has motivated multitask and transfer-learning benchmarks.The Robust Vision Challenge evaluates a single model across six vision challenges.

3. BDD100K

BDD100K is a large-scale, diverse driving-video dataset with 100K videos, high-resolution and high-frame-rate imagery, GPS/IMU trajectories, and annotations spanning multiple autonomous-driving tasks. Its varied environments and temporal annotations support studies of object detection, lane and drivable-area understanding, segmentation, tracking, and tracking-and-segmentation.

  • Dataset overview: BDD100K contains 100K driving videos collected from tens of thousands of drivers, with 720p imagery, 30fps video, and GPS/IMU recordings.The videos are collected through crowdsourcing supported by Nexar 2 and preserve driving trajectories.
  • Dataset diversity and splits: The dataset covers city streets, residential areas, highways, varied weather and times of day, and uses 70K/10K/20K training, validation, and testing videos.The 10th-second frame is annotated for image tasks, while entire sequences support tracking.
  • Object detection: Bounding boxes annotate 10 object categories in reference frames from all 100K videos, with visibility attributes for occluded and truncated instances.These annotations provide object-location information for autonomous-driving and general visual-recognition tasks.
  • Lane and drivable-area understanding: Lane markings use 8 main categories with continuity and direction attributes, while drivable areas distinguish directly drivable from alternatively drivable regions.The two drivable-area categories represent the driver’s current roadway and a lane accessible through changing lanes.
  • Semantic instance segmentation: Pixel-level annotations cover 10,000 sampled video clips and 40 object classes, split into 7K/1K/2K training, validation, and testing images.Each pixel receives a class label, and selected classes additionally receive instance identifiers.
  • Tracking and tracking-and-segmentation: The MOT dataset contains 2,000 videos, about 400K frames, 130.6K track identities, 3.3M bounding boxes, and 49,418 occlusion occurrences, capturing diverse scales, track lengths, and reappearance patterns.BDD100K also provides a MOTS dataset with 90 videos, split into 60 training, 10 validation, and 20 testing videos.

4. Diversity

BDD100K’s geographic, temporal, and scene diversity enables experiments on domain differences within the dataset and against Cityscapes. Results reveal substantial domain discrepancies, especially across time of day and between datasets, showing the need for methods that bridge these gaps.

  • Experimental setup: BDD100K evaluates diversity through object detection across internal domains and semantic segmentation against Cityscapes.The object-detection experiments study domains within BDD100K, while semantic segmentation examines domains between BDD100K and Cityscapes.
  • Object detection: 30K images per training domain are used to train ResNet-50 Faster R-CNN models evaluated with the COCO API.Domains are partitioned by time of day and scene type, with city street and daytime selected as validation domains.
  • Object detection: The city-versus-non-city domain difference is significant, while the daytime-versus-nighttime gap is substantially larger.These results indicate that further work is needed to bridge domain gaps across conditions.
  • Semantic segmentation: Semantic-segmentation models perform much worse when tested on a dataset different from their training dataset, revealing a dramatic domain shift between BDD100K and Cityscapes.Cityscapes was collected in German cities, whereas BDD100K is mainly from the US.

5. Multitask Learning

BDD100K studies multitask learning across homogeneous and heterogeneous autonomous-driving tasks. Joint supervision provides limited gains for similar-output tasks but can substantially improve expensive tasks through additional annotations and upstream-task pretraining.

  • Homogeneous multitask learning: With 10K images, joint lane-marking and drivable-area training raises lane-marking mean ODS-F from 45.41 to 50.40, while drivable-area mean IoU increases only from 64.23 to 64.37.At 20K and 70K images, joint- and single-task differences become insignificant, although performance is generally higher than with 10K images.
  • Homogeneous multitask learning: Because lane marking and drivable-area prediction share similar output structures, extra supervision may provide little new information and motivates heterogeneous multitask learning.The tasks use shared training images but different annotations: lane marking has three attributes, whereas drivable area has two foreground classes.
  • Heterogeneous multitask learning: Joint training increases instance-segmentation AP from 21.8 to 24.5 when the 7K-image instance-segmentation set is combined with the 70K-image detection set.The experiment uses Mask R-CNN with a ResNet-50 backbone and batch-level round-robin training.
  • Cascaded multitask learning: Adding annotations for simple tasks in a cascade improves complicated tasks requiring more expensive labels, including instance segmentation and tracking.More accurate detection can provide better object candidates for tracking, motivating allocation of annotation resources to basic tasks.
  • Joint training of multiple heterogeneous tasks: For MOTS, fine-tuning from instance-segmentation pretraining improves segmentation AP and MOTSP, whereas fine-tuning from MOT pretraining reduces identity switches.Additional examples from the instance-segmentation and MOT datasets further improve the difficult segmentation-tracking task.

6. Conclusion · Appendix

The work presents BDD100K as a large-scale driving video dataset with extensive annotations and establishes a heterogeneous multitask learning benchmark spanning varied prediction structures and driving-system functions.

  • 6. Conclusion: BDD100K is introduced as a large-scale driving video dataset with extensive annotations for heterogeneous tasks.
  • 6. Conclusion: The benchmark covers tasks with different prediction structures that serve distinct aspects of a complete driving system.
  • 6. Conclusion: The experiments analyze homogeneous and cascaded multitask learning scenarios.

A. Dataset Details

This section presents additional details about the BDD100K dataset and its annotations.

  • The section provides more details about the dataset.
  • The section also describes the dataset’s annotations.
  • Dataset and annotation details are presented together in this section.

A.1. Image Tagging

BDD100K’s image tags span diverse weather, scene, and time-of-day attributes, enabling study of visual transfer across domains.

  • Image Tagging: Weather, scene, and time-of-day attributes exhibit visual diversity in BDD100K images, supporting studies of visual transfer between domains.Figure 10 shows the distributions of these attributes.

A.2. Object Detection

BDD100K’s detection dataset contains more pedestrian examples than other compared datasets, while its broader inclusion of non-city scenes yields fewer persons per image than Cityscapes.

  • A.2. Object Detection: BDD100K has more pedestrian examples than the compared detection datasets, but fewer persons per image than Cityscapes because it includes non-city scenes such as highways.Table 10 compares pedestrian counts across detection datasets and attributes the lower per-image density to non-city scenes.

A.3. Lane Marking and Drivable Area · A.4. GPS Trajectory

BDD100K annotates lane-marking attributes motivated by their influence on driving decisions and introduces drivable-area detection as a segmentation task. Its GPS trajectories capture diverse driving behaviors and support training and testing imitation-learning algorithms on real driving data.

  • A.3. Lane Marking and Drivable Area: Lane-marking continuity is labeled independently because it is essential for making a “driving-across” decision.The annotation choice is based on the influence of lane attributes on driving decisions.
  • A.3. Lane Marking and Drivable Area: Lane-marking direction is labeled because it helps distinguish guidance or lane separation from deceleration cues.A marking parallel to the passing car may guide cars and separate lanes, whereas a perpendicular marking can signal deceleration.
  • A.3. Lane Marking and Drivable Area: Drivable-area detection is converted into 3-way segmentation over background, directly drivable, and alternatively drivable regions, while ignoring region IDs.A DRN-D-22 model is trained on 70,000 training images.
  • A.3. Lane Marking and Drivable Area: The segmentation model learns to split roads according to lanes and extrapolate drivable areas into unmarked space.This behavior is shown in the drivable-area prediction examples.
  • A.4. GPS Trajectory: GPS trajectory examples exhibit diverse driving behaviors, including starting, stopping, turning, and passing.These trajectories are shown in Figure 13.
  • A.4. GPS Trajectory: The GPS trajectory data is suitable for training and testing imitation-learning algorithms on real driving data.Its suitability follows from the recorded diversity of driving behaviors.

A.5. Semantic Instance Segmentation … B.1. Tracking

The section characterizes BDD100K’s long-tailed segmentation coverage and extensive tracking annotations, then describes benchmark implementation details for tracking. It also shows that dataset-specific training avoids segmentation errors arising from cross-domain differences.

  • A.5. Semantic Instance Segmentation: BDD100K covers rare segmentation categories while containing many common traffic-object instances, producing a pronounced long-tail distribution.There are almost 60 thousand car instances, a few hundred rider and motorcycle instances, and mere dozens of trailer and train instances.
  • A.5. Semantic Instance Segmentation: Models trained on Cityscapes confuse infrastructure and traffic-sign structures in US scenes, whereas models trained on BDD100K avoid these errors.Cityscapes models may segment sky as building, confuse US highway traffic signs, and over-fit the collecting vehicle’s hood.
  • A.6. Multiple Object Tracking and Segmentation: BDD100K’s tracking annotations cover more than one hundred thousand bounding-box instances and more than two million bounding boxes.The segmentation tracking set contains more than six thousand instances with one hundred thousand polygons.
  • B. Model Details: The paper provides implementation details for its benchmark models.This subsection introduces the model details before the tracking-specific architecture description.
  • B.1. Tracking: The tracking model modifies Faster R-CNN with correlation, bounding-box propagation, and appearance-based association heads.The correlation and regression components estimate short-term frame-to-frame offsets, while the association head learns embeddings for instance re-identification.
  • B.1. Tracking: During training, the tracking model samples frame pairs within an interval of t = 3 frames.This sampling is used for short-term association training.
  • B.1. Tracking: The section includes figures and tables documenting segmentation class distributions, BDD100K MOTS annotations, domain-discrepancy detection results, and lane-marking evaluations.The cited visual and tabular materials cover long-tail classes, example MOTS annotations, object-detection domain discrepancy, and joint lane-marking/drivable-area training.
Loading 1805.04687v2…