PaperlayerView exact revision on arXiv

Source-linked AI summary

SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation

Tao Sun, Mattia Segu, Janis Postels, Yuxuan Wang, Luc Van Gool, Bernt Schiele, Federico Tombari, Fisher Yu

arXiv:2206.08367v1cs.CVcs.LG

TL;DR

Existing driving datasets inadequately capture continuously changing environmental conditions, limiting evaluation of perception generalization under domain shift. SHIFT addresses this gap with a large synthetic, multi-task dataset spanning discrete and continuous shifts, and shows that domain shifts degrade many mainstream perception algorithms as severity increases.

  • Problem

    Existing driving datasets inadequately capture changing environmental conditions, while perception generalization to unforeseen conditions remains limited and domain shifts can cause performance drops.

  • Method

    SHIFT is a synthetic multi-task driving dataset with comprehensive sensors, annotations, and discrete and continuous shifts in weather, lighting, and agent density.

  • Results

    Performance drops across many perception tasks under domain shift, with degradation increasing as shifted conditions become more distant from the source domain.

  • Takeaways & Limitations

    SHIFT supports studying adaptation and uncertainty estimation across multiple autonomous-driving perception tasks under discrete and continuous domain shifts.

Abstract

from arXiv · show

Adapting to a continuously evolving environment is a safety-critical challenge inevitably faced by all autonomous driving systems. Existing image and video driving datasets, however, fall short of capturing the mutable nature of the real world. In this paper, we introduce the largest multi-task synthetic dataset for autonomous driving, SHIFT. It presents discrete and continuous shifts in cloudiness, rain and fog intensity, time of day, and vehicle and pedestrian density. Featuring a comprehensive sensor suite and annotations for several mainstream perception tasks, SHIFT allows investigating the degradation of a perception system performance at increasing levels of domain shift, fostering the development of continuous adaptation strategies to mitigate this problem and assess model robustness and generality. Our dataset and benchmark toolkit are publicly available at www.vis.xyz/shift.

1. Introduction

Existing driving datasets have advanced perception accuracy but inadequately capture generalization failures under changing environmental conditions, especially continuous domain shifts. SHIFT addresses this gap with a large multi-task synthetic dataset designed for discrete and continuous shifts across safety-critical driving factors.

  • Motivation: Driving datasets have fueled perception progress, but generalization to unforeseen environmental conditions remains underwhelming.Self-driving cars must adapt to new environments and identify life-threatening failure cases to prevent crashes.
  • Problem: Domain shifts in weather, lighting, scenery, and road-agent behavior or quantity can cause unforeseeable performance drops outside training conditions.Existing methods include domain generalization, domain adaptation, uncertainty estimation, and out-of-distribution detection, but are often evaluated on toy or synthetically corrupted datasets.
  • Problem: Existing driving sequences are approximately stationary and mainly provide discrete shifts between homogeneous sequences, whereas continuous intra-sequence shifts are certain in the real world.The paper motivates dataset design that quantifies and addresses domain shift at both discrete and continuous levels.
  • SHIFT: SHIFT captures realistic discrete and continuous shifts in time of day, cloudiness, rain, fog strength, and vehicle and pedestrian density.It is introduced as a new synthetic dataset for the continuously evolving nature of the real world.
  • Contributions: SHIFT is presented as the largest synthetic autonomous-driving dataset, with multi-task perception coverage, comprehensive sensors, and inclusive annotations and conditions.The dataset supports research on domain generalization, domain adaptation, uncertainty estimation, continual learning, and continuous test-time learning and adaptation.

2. Related Work

Related work spans realistic and synthetic driving datasets, adverse-condition benchmarks, domain-adaptation methods, and uncertainty estimation. SHIFT addresses limitations in existing datasets by supporting continuous domain shifts across diverse annotations and sequences, while enabling continual adaptation studies.

  • Driving Datasets: Synthetic datasets and video-game benchmarks provide varied sensors or annotations, but typically use short sequences and discrete domain labels.SYNTHIA, AIODrive, GTA-V, and VIPER illustrate simulator- or game-based data generation with differing task and sensor coverage.
  • Adverse Conditions: Adverse-condition datasets evaluate robustness under OOD shifts, but coverage is constrained: ACDC supports only semantic segmentation, whereas BDD100K offers diverse tasks with discrete time-of-day and weather labels.The India Driving Dataset uniquely represents extremely busy roads as an adverse condition.
  • Domain Adaptation: UDA learns transferable features between labeled source and unlabeled target domains, while IncDA uses intermediate domains to reduce gaps between consecutive adaptation steps.SHIFT’s discrete shifts support UDA, and its variations with different strengths along natural axes make it suitable for IncDA.
  • Adaptation and Uncertainty: ContinuousTTA adapts at test time to gradual shifts in incoming streams when target-domain data are unavailable in advance, while uncertainty estimation remains poorly calibrated on difficult datasets.Uncertainty estimation is important for avoiding dangerous autonomous-driving failures, but existing techniques mainly focus on classification on toy datasets.

3. The SHIFT Dataset

SHIFT is a large synthetic driving dataset designed to support multi-task perception and forecasting under environmental domain shifts. It combines comprehensive synchronized sensing, full-frame annotations, and discrete and continuous changes in driving conditions.

  • SHIFT supports multiple image- and video-based perception and forecasting tasks against environmental changes.
  • Sensor suite: Its sensor suite includes 11 synchronized sensors, five multi-view RGB cameras, stereo RGB cameras, optical flow, depth, GNSS, IMU, and 128-channel LiDAR.All cameras provide 90◦ field of view and 1280 × 800 pixel resolution; sensors are captured at 10Hz.
  • Annotations: SHIFT provides full-frame annotations for 2D/3D bounding box trajectories, instance and semantic segmentation, optical flow, and dense depth.The annotations cover each frame in the sequences rather than only selected keyframes.
  • Domain shifts: The dataset captures discrete and continuous shifts in weather, time of day, vehicle and pedestrian density, and camera orientation.Weather changes include cloudiness, rain, and fog intensity.
  • Dataset scale: 70+ hours of driving and 2,500,000 annotated frames make SHIFT the largest synthetic driving dataset available.Each sequence contains 500 frames collected at 10 Hz, equivalent to 50 seconds of driving time.

4. Experiments

Experiments show that perception performance and uncertainty calibration degrade as discrete domain shifts move farther from the source, with compatible trends between SHIFT and BDD100K. Under continuous shifts, test-time adaptation is effective with limited target data but can cause catastrophic forgetting of the source domain.

  • Impact of domain shift: Mainstream perception algorithms suffer performance drops under domain shift, with severity increasing as the test domain moves farther from clear-daytime.Models are trained in clear-daytime and tested under different weather conditions across the supported vision tasks.
  • Uncertainty estimation: Uncertainty calibration worsens farther from the source domain on both SHIFT and BDD100K.Calibration is evaluated with Expected Calibration Error (ECE) for uncertainty estimation methods under distributional shift.
  • Real-world compatibility: SHIFT and BDD100K show compatible domain-shift trends for 2D object detection and semantic segmentation across YOLO v3, Faster R-CNN, Cascade R-CNN, and segmentation methods.BDD100K is used because it provides a large subset of SHIFT’s tasks with discrete real-world domain labels.
  • Continual domain adaptation: TTA is extremely efficient with small target-data availability, whereas Incremental DA outperforms alternatives when enough data becomes available.The comparison evaluates semantic segmentation on clear-daytime → night continuous shifts using increasing numbers of training sequences.
  • Test-time adaptation: TTA improves performance on current target domains, but adapted models cannot recover their original source-domain accuracy after looping back.This demonstrates that catastrophic forgetting affects test-time adaptation despite its effectiveness for on-the-go adaptation.

5. Conclusion … A.2. Annotation details

SHIFT is presented as a multi-task driving dataset for studying adaptation strategies and uncertainty estimation under discrete and continuous domain shifts, with simulation enabling inexpensive data collection and annotation. Its supplementary materials document the dataset, experiments, implementation, sensor layout, and comprehensive annotations across major perception tasks.

  • 5. Conclusion: SHIFT covers important perception tasks under discrete and continuous domain shifts and supports findings on adaptation strategies and uncertainty estimation.The authors note that simulation enables inexpensive data collection and annotation despite imperfect real-world representation.
  • Appendix: The appendix details dataset design, generation, statistics, additional experiments, and implementation for reproducibility.Additional experiments include continual-shift multitask baselines, real-world-data and dataset-size ablations, VIPER comparison, and rainy or foggy failure analysis.
  • A. Dataset Details: A detailed user guide and further information are available at https://www.vis.xyz/shift.
  • A.1. Reference systems and sensor layout: SHIFT uses world, vehicle, and camera reference systems for absolute object positions, 3D annotations, and individual-camera coordinates.The vehicle system follows KITTI’s convention and the right-hand rule; provided vehicle-to-camera matrices transform annotations for other cameras.
  • A.1. Reference systems and sensor layout: The sensor layout includes supported sensor types positioned around the vehicle reference-system origin, with stereo cameras horizontally displaced by 20cm.LiDAR and motion sensors are located at the origin, while most cameras lie on a centered circle.
  • A.2. Annotation details: SHIFT provides 2D/3D boxes, identities, visibility attributes, and fine-grained object classes for six traffic-participant categories.The categories are car, truck, bus, bicycle, motorcycle, and pedestrian.
  • A.2. Annotation details: SHIFT provides 9 DoF 3D box annotations using Euler angles—yaw, roll, and pitch—instead of the 7 DoF used by previous datasets.
  • A.2. Annotation details: The annotation set includes 23-class panoptic segmentation, front-view aligned depth at 1mm resolution, and UV-format optical flow labels.These annotations support segmentation, tracking, monocular and stereo depth estimation, object tracking, and ego-motion tasks.

A.3. Data generation pipeline

SHIFT generates discrete domain shifts by uniformly sampling candidate environmental conditions across four categories, while continuous shifts vary one category through linear interpolation. The pipeline supports controlled sequence lengths, simultaneous shifts, and numerical severity labeling.

  • Discrete shift: Discrete shifts cover time of day, weather, vehicle density, and pedestrian density using candidate environmental domains sampled uniformly within each category.Each candidate corresponds to a defined group of environmental parameters, which may be fixed values or uniformly sampled value sets.
  • Discrete shift: Uniform category-wise sampling preserves candidate marginal distributions, enables data expansion without distribution changes, and gives every sequence subset the same domain distribution.This supports fair experiments on the impact of data amount.
  • Continuous shifts: Continuous shifts change one domain category while keeping the other categories fixed, with the changing parameters produced by linear interpolation between initial, intermediate, and final states.The interpolation uses t values 0, 0.2, and 1, where t ∈ [0, 1] represents shift severity from the minimum to maximum allowed parameter.
  • Continuous shifts: 300, 120, and 30 continuous-shift sequences are provided at 1x, 10x, and 100x lengths, respectively, with a 1x base length of 200 frames.An additional 150 base-length sequences contain simultaneous shifts across multiple domains; longer sequences produce slower domain-shift speeds.
  • Domain labeling details: Domain severity is quantified numerically; weather severity uses percentages from 0% for clear conditions to 100% for the most extreme simulator-supported condition.Weather directions include cloudiness, precipitation, fog density, and fog distance.

A.4. Dataset statistics

SHIFT spans diverse object densities and bounding-box sizes, including 0–30 boxes per frame and box dimensions from 10 to 650 pixels. Compared with BDD100K, it contains more low-density frames and a higher share of very small bounding boxes, while crowded-frame trends are similar.

  • Object density: SHIFT covers object densities from 0 to 30 boxes per frame.Its object-density distribution is measured by bounding boxes per frame.
  • Object density: SHIFT has more frames with fewer bounding boxes on average than BDD100K, while frames with ≥20 boxes/frame show similar trends.The comparison uses BDD100K’s MOT set.
  • Bounding-box size: SHIFT contains bounding boxes ranging from 10 to 650 pixels, with 41.2% smaller than 15 pixels versus 30.9% in BDD100K.This distribution provides challenging conditions for small-object detection and tracking.

B. Additional Experiments

The additional experiments evaluate SHIFT for multitask learning, joint training with real-world data, and the effects of dataset size and sampling rate.

  • Additional Experiments: The experiments cover multitask learning, joint training with real-world data, dataset size, and sampling rate.These studies are presented in Secs. B.1–B.3.

B.1. Multitask learning

SHIFT enables systematic study of multitask learning across four perception tasks under discrete domain shifts. Specific task combinations improve source and out-of-distribution robustness, but optical flow can be harmed and substantial domain-shift degradation remains.

  • Setup: The study evaluates semantic segmentation, instance segmentation, monocular depth estimation, and optical flow estimation in 15 multitask combinations.Each task requires distinct encoding functions, including pixel-level, instance-level, contextual, and motion information.
  • Multitask model: The unified model uses DRN-D-54 as a shared feature extractor with task-specific heads in an encoder-decoder architecture.DRN-D-54 contains sequential residual blocks with dilated convolutions and transposed convolutions for dense prediction.
  • Multitask learning improves robustness: S + D + I raises source performance by 8.6% / 11.0% / 1.6% on semantic segmentation, depth, and instance segmentation, respectively.Similar source improvements occur for S + I and S + D.
  • Multitask learning improves robustness: S + D + I improves OOD performance by 71.6% / 42.8% / 6.3% on semantic segmentation, depth, and instance segmentation, respectively.The larger OOD gains are attributed to learning more general features shareable across tasks; adding instance segmentation typically yields the greatest robustness improvements.
  • Limitations: Optical flow performance drops by -0.2% to -38.9% when jointly trained with other tasks, while instance segmentation improves by at most 10.7% on OOD performance from other tasks.Despite multitask gains, the minimum average OOD performance drop remains ∼40% for all tasks and exceeds 60% under foggy and night conditions.

B.2. Joint training with real-world data

Jointly training on clear daytime BDD100K data and all SHIFT domain variations improves both source-domain and out-of-domain performance. The gains extend across object detection and semantic segmentation, suggesting greater robustness to real-world shifts.

  • Training setup: The study jointly trains on clear daytime BDD100K data and all SHIFT domain variations, then evaluates on other BDD100K domains.Faster R-CNN is used for object detection and DRN-D-54 for semantic segmentation.
  • Object detection: 2.52% and 3.40% are the relative improvements in source-domain and out-of-domain object detection performance, respectively.
  • Semantic segmentation: 51.20% is the source-domain semantic-segmentation mIoU, up from 46.04% with a 10.34% relative improvement.
  • Semantic segmentation: 39.76% is the out-of-domain semantic-segmentation mIoU, rising from 37.37% with a 5.30% relative improvement.
  • Implication: Joint training with SHIFT’s domain variety improves model robustness to real-world shifts when real-world training data cover a limited domain.

B.3. Dataset size · B.4. Comparison with VIPER · B.5. Error analysis for foggy and rainy domains

The dataset-size studies identify sampling-rate saturation and diminishing returns from adding sequences, while comparing SHIFT with VIPER and BDD100K shows SHIFT’s adverse conditions better reflect real-world generalization threats. Error analysis further reveals task- and domain-specific degradation in foggy and rainy conditions.

  • B.3. Dataset size: Dataset-size ablations vary sampling rate and sequence count while training every model on clear-daytime sequences.The ablations examine how dataset design affects performance.
  • B.3. Dataset size: Sampling-rate experiments evaluate semantic segmentation, object detection, and multiple-object tracking on sequences sampled at different frame rates.Different tasks begin to saturate at different sampling rates.
  • B.3. Dataset size: SHIFT is collected at 10Hz and also provides a 1Hz subset for image-based perception tasks.The 1Hz subset follows the sampling-rate experiments for image-based tasks.
  • B.3. Dataset size: Performance increases continuously up to 3000 sequences, but additional sequences yield diminishing gains, potentially because simulator variation is limited.The study varies sequence counts at 1Hz for semantic segmentation and object detection.
  • B.4. Comparison with VIPER: Compared with VIPER and BDD100K, SHIFT’s adverse conditions present a more relevant threat to model generalization under domain shifts.The comparison evaluates segmentation performance using a model trained on the daytime domain.
  • B.5. Error analysis for foggy and rainy domains: Detection and segmentation models behave differently across domain shifts, with foggy and rainy domains showing the largest discrepancy between the two tasks.The analysis accounts for the fact that the tasks use different label sets.
  • B.5. Error analysis for foggy and rainy domains: In the rainy domain, ‘sky’ is the most drastically affected segmentation class, with -69% mIoU relative to clear-daytime conditions.The passage reports the class-specific discrepancy for segmentation in the rainy domain.

C. Implementation Details

The implementation evaluates five autonomous-driving perception tasks using specified models, backbones, training procedures, and task-specific metrics. Object detection uses 50k training frames, while semantic segmentation uses 20k frames until convergence.

  • Object detection: Object detection compares Faster R-CNN, Cascade R-CNN, and YOLO v3 using mAP for 2D bounding boxes.Faster R-CNN and Cascade R-CNN use ResNet-50, YOLO v3 uses DarkNet, and training follows the mmdetection “1x” schedule on 50k frames.
  • Semantic segmentation: Semantic segmentation compares DeepLab v3+, FCN, and DRN-D-54 using mIoU after training on 20k frames for approximately 150 epochs.All three models use ResNet-50 backbones and are trained until convergence.
  • Instance segmentation: Instance segmentation uses Mask R-CNN with a ResNet-50 backbone and evaluates performance using segmentation mAP.Its training routine matches Faster R-CNN’s.
  • Depth estimation: Depth estimation uses AdaBins with a U-Net-like backbone and adaptive-bin prediction, following the official implementation and KITTI benchmark.Evaluation uses SILog on the central Eigen Crop.
  • Optical flow estimation: Optical flow estimation uses RAFT fine-tuned from Things Dataset pretrained weights on 10k frames, evaluated with EPE.The model is fine-tuned using SHIFT data.
Loading 2206.08367v1…