Source-linked AI summary
VisDA: The Visual Domain Adaptation Challenge
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, Kate Saenko
TL;DR
Visual domain adaptation lacks large, diverse benchmarks for transferring models from labeled source data to unlabeled deployment domains. VisDA introduces a large synthetic-to-real challenge covering classification and segmentation, with separate target validation and test domains; it also reports baselines and participant results while identifying supervised ImageNet pre-training as an important scope limitation.
Problem
Domain-adaptation progress is constrained by benchmarks with limited task diversity, small domain shifts, and small dataset sizes.
Method
VisDA constructs a synthetic-to-real unsupervised adaptation benchmark with classification and semantic-segmentation tracks and separate unlabeled target validation and test domains.
Results
The dataset contains over 280K classification images and approximately 30K segmentation images, alongside baseline and challenge-participant performance analyses.
Takeaways & Limitations
VisDA establishes an open benchmark and protocol for evaluating domain-adaptation methods across diverse, unseen target domains and more than one vision task.
Takeaways & Limitations
Challenge systems relied on supervised ImageNet pre-training, which made the synthetic-to-real adaptation problem considerably easier.
Abstract
from arXiv · showhide
We present the 2017 Visual Domain Adaptation (VisDA) dataset and challenge, a large-scale testbed for unsupervised domain adaptation across visual domains. Unsupervised domain adaptation aims to solve the real-world problem of domain shift, where machine learning models trained on one domain must be transferred and adapted to a novel visual domain without additional supervision. The VisDA2017 challenge is focused on the simulation-to-reality shift and has two associated tasks: image classification and image segmentation. The goal in both tracks is to first train a model on simulated, synthetic data in the source domain and then adapt it to perform well on real image data in the unlabeled test domain. Our dataset is the largest one to date for cross-domain object classification, with over 280K images across 12 categories in the combined training, validation and testing domains. The image segmentation dataset is also large-scale with over 30K images across 18 categories in the three domains. We compare VisDA to existing cross-domain adaptation datasets and provide a baseline performance analysis using various domain adaptation models that are currently popular in the field.
1. Introduction
VisDA2017 addresses the limited scale, diversity, and realism of domain-adaptation benchmarks with a large unsupervised simulation-to-real challenge spanning classification and semantic segmentation.
- Dataset shift causes visual-recognition performance to drop when deployment imagery differs in lighting, pose, background, or collection process.
- VisDA evaluates transfer from a labeled source domain to unlabeled target domains across classification and semantic segmentation tasks.Each task provides training, validation, and test domains, with validation and test serving as distinct targets.
- The benchmark introduces a source-to-target task in which models must improve target performance without using target annotations.
- The challenge focuses on synthetic-to-real adaptation, motivated by applications where labeled real imagery is difficult or expensive to collect.Synthetic rendering can produce large quantities of labeled data once the rendering pipeline is established.
- VisDA provides over 280K object-classification images and approximately 30K semantic-segmentation images across three domains.
- The paper reports baseline results, challenge-participant results, open resources, and more difficult dataset and experiment variants for future research.
2. Related Work
Prior domain-adaptation benchmarks often offer limited task diversity, small domain shifts, or small datasets. VisDA responds with broader task coverage and balanced synthetic data designed for synthetic-to-real evaluation.
- Existing visual domain-adaptation benchmarks include shallow-feature and deep-adaptation studies, but the paper focuses on benchmark datasets rather than surveying methods.
- Many popular benchmarks emphasize image classification while comparatively overlooking detection, structure prediction, and sequence labeling.
- Classification Datasets: Office contains 31 object categories across DSLR, WEBCAM, and AMAZON domains.
- Classification Datasets: Prior benchmarks can be limited by small domain shifts and small scale, while some larger web-collected domains remain relatively similar.The paper notes that adaptation performance on the Caltech-Imagenet shift has approached 90% accuracy.
- Semantic Segmentation Datasets: VisDA combines classification and semantic segmentation settings, including GTA5, CityScapes, and Berkeley Deep Drive/Nexar data for segmentation.Semantic-segmentation annotation is labor-intensive, leaving relatively few datasets designed specifically for domain adaptation.
- Synthetic Datasets: VisDA-C offers a substantial balanced synthetic collection with rendering metadata and a deliberately limited set of classes overlapping standard VOC PASCAL categories.
3. VisDA-C: Classification Dataset
VisDA-C is a large-scale 12-category classification testbed for unsupervised adaptation from synthetic renderings to unlabeled real-image domains. Its validation and test domains are distinct, enabling evaluation on unseen target domains and supporting increasingly difficult benchmark variants.
- Dataset Structure: VisDA-C contains three domains sharing 12 object categories: synthetic training images, COCO validation images, and YouTube-BB testing images.The real-domain images are cropped object instances, while the source domain consists of rendered 3D models.
- Evaluation Protocol: Separate validation and test domains prevent hyper-parameter tuning on test data, although over-tuning to one target domain can degrade performance on the other.Validation uses COCO and testing uses YouTube-BB, making the target domains different in both protocol and visual content.
- Dataset Acquisition: 152,397 synthetic images were generated from 1,907 3D models using varied camera angles and lighting conditions.The rendering process used twenty camera yaw-pitch combinations and four light directions per model.
- Dataset Acquisition: 55,388 validation-domain object images were collected from COCO, while smaller patches and the oversized person category were filtered or capped.Images with height or width below 70 pixels were excluded, and person images were reduced to 4,000.
- Baseline Experiments: On validation, source-only AlexNet reached 28.12% accuracy, while Deep CORAL and DAN improved it to 45.53% and 51.62%, respectively.These corresponded to relative improvements of 83.6% and 61.9% over the source-only model, though both remained below in-domain performance.
- Increasing Difficulty: Challenge participants achieved test-domain performance on par with in-domain training, motivating harder variants such as domain swapping and VisDA-C-ext.VisDA-C-ext adds 35,591 COCO images and 4,533 YouTube-BB images, and all evaluated models performed worse on it.
4. VisDA-S: Semantic Segmentation
VisDA-S benchmarks unsupervised semantic segmentation adaptation from synthetic GTA5 imagery to real CityScapes and Nexar dashcam footage. It provides source annotations, target-domain evaluation settings, baseline results, and challenge approaches for this simulation-to-reality shift.
- Dataset and protocol: The benchmark provides pixel-level annotations for 19 semantic classes and follows the classification track’s source, validation-target, and test-target protocol.
- Dataset and protocol: VisDA-S evaluates adaptation from synthetic GTA5 dashcam renderings to real CityScapes validation and Nexar test images.The training domain includes semantic labels, CityScapes provides labeled validation data, and Nexar supplies unlabeled test data.
- Dataset construction: GTA5 contributes 24,966 high-quality labeled frames, while the segmentation setup uses 1,500 annotated 1280 × 720 images matching 19 GTA5 and CityScapes categories.
- Baseline results: 21.6 mIoU is achieved by the front-end dilation source model on CityScapes validation, versus 64.0 oracle mIoU; adaptation raises performance to 25.5 mIoU.
- Challenge methods: The winning challenge pipeline combines frame-level target-image updates, pixel-level discrimination, model ensembling, pyramid spatial pooling, and multi-scale domain discrimination.
5. Conclusion
The paper introduces a large-scale synthetic-to-real dataset for unsupervised domain adaptation and analyzes baseline and state-of-the-art performance. It also proposes dataset variants and open resources to support research on harder adaptation settings, including no supervised pre-training.
- The paper introduces a large-scale synthetic-to-real dataset and reports detailed baseline and current state-of-the-art performance analyses.
- Dataset modifications and alternative setups enable evaluation of more difficult domain-adaptation hypotheses.
- The no-pre-train setup remains a substantial challenge for existing domain-adaptation methods.
- The authors plan to open-source resources, tools, and dataset variants to support adaptation research in diverse settings.
A. Other relevant datasets
The paper situates VisDA among existing cross-domain datasets used for classification and segmentation. These alternatives are often smaller or narrower in domain coverage, and some are not directly comparable because they use nonreal domains or different tasks.
- Existing cross-domain datasets support evaluation of domain-adaptation algorithms for object classification and semantic segmentation.
- Other popular datasets are not directly comparable with VisDA because they use nonreal domains such as digits or faces, or different tasks such as detection.
B. Number of images per category in VisDA-C
The paper documents the number of models and images in each category across VisDA-C’s training, validation, and testing domains, with the distribution also visualized in Figure 3.
- Table 8 reports the number of models and images per category across VisDA-C training, validation, and testing domains.
- The VisDA-C image distribution is also shown as a bar plot in Figure 3.
C. VisDA Image samples
The VisDA image samples illustrate synthetic training domains and real validation or test domains for both classification and segmentation. Figures 5–10 provide additional examples across the considered categories and domains.
- Overview: Figures 2 and 4 present VisDA-C and VisDA-S, while Figures 5–10 show more images for each considered category across all domains.The sample figures cover both challenge components and their domain-specific data.
- Classification: Classification samples pair synthetic CAD-model training images with real MS COCO validation and YouTube-BB test images.The three domains are shown in Figures 5, 6, and 7.
- Segmentation: Segmentation samples pair synthetic GTA5 dashcam training images with real CityScapes validation and NEXAR test images.The three domains are shown in Figures 8, 9, and 10.