PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-ensembling for visual domain adaptation

Geoffrey French, Michal Mackiewicz, Mark Fisher

arXiv:1706.05208v4cs.CV

TL;DR

The paper addresses visual domain adaptation when labeled source data must support prediction on a distinct unlabeled target domain. It adapts mean-teacher self-ensembling with confidence thresholding, class balancing, and other modifications, achieving state-of-the-art results across benchmarks and near-supervised accuracy on small image tasks.

  • Problem

    Visual domain adaptation seeks to transfer knowledge from labeled source data to distinct unlabeled target data, reducing reliance on costly ground-truth annotation.

  • Method

    The approach adapts mean-teacher self-ensembling with confidence thresholding, class balancing, and data augmentation for challenging domain-adaptation scenarios.

  • Results

    The method achieves state-of-the-art results across several benchmarks, with small-image performance approaching supervised learning and a winning VisDA-2017 solution.

  • Takeaways & Limitations

    Self-ensembling can provide strong performance across source and target domains and can approach traditional supervised learning on digit-recognition benchmarks.

  • Takeaways & Limitations

    Class-balance loss is expected to adversely affect target datasets with large class imbalance.

Abstract

from arXiv · show

This paper explores the use of self-ensembling for visual domain adaptation problems. Our technique is derived from the mean teacher variant (Tarvainen et al., 2017) of temporal ensembling (Laine et al;, 2017), a technique that achieved state of the art results in the area of semi-supervised learning. We introduce a number of modifications to their approach for challenging domain adaptation scenarios and evaluate its effectiveness. Our approach achieves state of the art results in a variety of benchmarks, including our winning entry in the VISDA-2017 visual domain adaptation challenge. In small image benchmarks, our algorithm not only outperforms prior art, but can also achieve accuracy that is close to that of a classifier trained in a supervised fashion.

1 Introduction

Visual domain adaptation transfers knowledge from labeled source data to unlabeled target data when labels are costly, and this paper adapts self-ensembling to improve performance in challenging settings.

  • 1 Introduction: Deep-learning vision systems require large labeled datasets, whose ground-truth annotations are expensive to produce.
  • 1 Introduction: Unsupervised domain adaptation transfers knowledge from a labeled source dataset to a distinct unlabeled target dataset while preserving the task objective.It can use abundant labeled synthetic data and unlabeled real data.
  • 1 Introduction: The paper evaluates its approach on visual domain adaptation benchmarks, including the VisDA-17 challenge.
  • 1 Introduction: The approach extends Tarvainen et al.’s self-ensembling method [29] to domain adaptation and adds confidence thresholding and class balancing for harder scenarios.The paper reports state-of-the-art results across several benchmarks, with some results approaching supervised learning.

2 Related work

The paper builds on self-ensembling methods that enforce prediction consistency and situates them among domain-adaptation approaches based on feature learning, adversarial training, generative transformation, and distribution alignment.

  • 2.1 Self-ensembling for semi-supervised learning: Self-ensembling trains networks to produce consistent predictions for unlabeled samples under augmentation, dropout, noise, or adversarial perturbations [18].
  • 2.1 Self-ensembling for semi-supervised learning: Laine et al.’s Π-model compares predictions from two differently perturbed passes, while their temporal model uses historical prediction averages [14].Both methods achieved state-of-the-art results on SVHN and CIFAR-10 semi-supervised benchmarks.
  • 2.1 Self-ensembling for semi-supervised learning: Tarvainen et al. [29] replace prediction averaging with an exponential moving average of network weights, using gradient-trained student and teacher networks.The student is trained to match the teacher’s predictions through a mean-square consistency loss.
  • 2.2 Domain adaptation: Domain-adaptation methods include auto-encoders, gradient-reversal domain classifiers, tri-training, GAN-based transformations, and feature-distribution alignment.These approaches seek domain-invariant representations, transformed samples, or reduced source-target distribution differences.
  • 2.2 Domain adaptation: The related-work landscape includes methods that minimize feature-distribution differences using covariance alignment, Maximum Mean Discrepancy, or domain-specific batch-normalization statistics.
  • 2.2 Domain adaptation: Unlike semi-supervised learning with mixed labeled and unlabeled samples, domain adaptation uses separate labeled-source and unlabeled-target datasets with different distributions.The paper’s variant therefore uses separate source and target paths.

3 Method

The method adapts mean teacher self-ensembling to domain adaptation by separating labeled source and unlabeled target data, then adds confidence thresholding, augmentation choices, and class balancing for difficult scenarios.

  • 3.1 Adapting to domain adaptation: The model applies cross-entropy to labeled source samples and mean-squared self-ensembling loss to unlabeled target samples using student–teacher prediction agreement.The teacher weights are an exponential moving average of the student weights, and source and target datasets are handled separately.
  • 3.1 Adapting to domain adaptation: State-of-the-art results were achieved in 5 of 8 small image benchmarks with the initial mean-teacher domain-adaptation model, while three benchmarks required further modifications.The remaining difficult benchmarks were MNIST → SVHN, STL → CIFAR-10, and Syn-digits → SVHN.
  • 3.2 Confidence thresholding: Confidence thresholding stabilizes training by masking the self-ensembling loss for target samples whose teacher confidence falls below 0.968.The authors hypothesize that filtering low-confidence predictions increases the signal-to-noise ratio, although confidence does not guarantee correctness.
  • 3.2 Confidence thresholding: Confidence thresholding achieves state-of-the-art results on STL →CIFAR-10 and Syn-digits →SVHN, while slightly reducing performance on MNIST ↔USPS and SVHN → MNIST.The authors recommend it as a replacement for the Gaussian ramp-up because it stabilizes more challenging scenarios.
  • 3.3 Data augmentation: Translations and horizontal flips improve several benchmarks, whereas affine augmentation helps digit and traffic-sign datasets but can impair photographic-domain and VisDA-17 performance.Three augmentation schemes were evaluated: minimal Gaussian noise, standard translations and flips, and affine transformations.
  • 3.4 Class balance loss: A class-balance loss penalizes imbalanced target predictions by matching the mini-batch mean class probabilities to a uniform distribution.Its strength is scaled by the fraction of target samples that pass the confidence threshold, addressing MNIST →SVHN collapse toward the ‘1’ class.

4 Experiments

Across small-image benchmarks and VisDA-2017, self-ensembling achieves strong domain-adaptation results, often approaching or surpassing supervised target-domain baselines. Performance depends on dataset-specific augmentation, confidence thresholding, and class balancing.

  • Small image datasets: Self-ensembling outpaces prior methods on MNIST ↔USPS and reaches performance nearly matching supervised learning with target labels.
  • Small image datasets: Self-ensembling on CIFAR-10 →STL surpasses both the source-trained baseline and the target-trained theoretical maximum, supporting a regularisation effect.
  • Small image datasets: 2.9% error reduced from 6.9% on Syn-Digits →SVHN, slightly outperforming the 3.4% supervised train-on-target error.
  • Small image datasets: The MNIST →SVHN path is harder because grayscale, uniform MNIST digits differ from variable-sized, colored SVHN digits, requiring class balancing and additional augmentation.
  • VisDA-2017 visual domain adaptation challenge: VisDA-2017 adapts labeled 3D renderings to real validation and test images across 12 classes, using a pretrained ResNet-152 with a 12-unit output layer.

5 Conclusions

The paper concludes that self-ensembling delivers state-of-the-art domain adaptation and near-supervised accuracy on digit benchmarks. It interprets the method as label propagation after source–target distribution alignment.

  • The algorithm achieves state-of-the-art results across benchmarks and nearly matches traditional supervised learning on MNIST- and SVHN-targeted digit tasks.
  • The approach works with randomly initialized and pretrained network architectures and produces strong performance on both source and target samples.
  • Effective adaptation is framed as first aligning source and target distributions, then refining correspondence through self-ensembling label propagation.

A.1 Small image datasets

The paper’s small-image experiments use datasets summarized in Table 3.

  • Table 3 describes the datasets used in the paper’s experiments.

A.2 Data preparation

Several datasets require preprocessing to align image resolution, format, classes, or regions of interest before adaptation.

  • USPS images are up-scaled from 16 × 16 to 28 × 28 to match MNIST.
  • STL images are down-scaled to 32×32, while unmatched frog and monkey classes are removed to form a 9-class problem.
  • GTSRB sign regions are extracted from bounding-box annotations and scaled to 40 × 40 to match Syn-Signs.
  • MNIST images are padded to 32 × 32 and converted to RGB by replicating the grayscale channel for SVHN compatibility.

B.1 Training procedure

Training used fixed optimization settings, dataset-specific batch sizes and hyperparameters, and separate augmentation schemes for reduced and competition configurations.

  • B.1 Training procedure: 300 epochs with Adam, a 0.001 learning rate, and mini-batches of 256 samples were used generally, reduced to 128 for Syn-digits →SVHN and Syn-signs →GTSRB.The self-ensembling loss was weighted by 3 and class balancing by 0.005.
  • B.1 Training procedure: Early stopping selected the epoch with the highest proportion of samples passing the confidence threshold, using that epoch’s target test-set performance as the final score.
  • B.1 Training procedure: The VisDA-17 configuration used batch size 56, self-ensembling weight 10, confidence threshold 0.9, class-balancing weight 0.01, and learning rates of 10^-5 and 10^-6 for new and pretrained layers.
  • B.1 Training procedure: The reduced augmentation scheme scaled images to 176 pixels on the smallest dimension before randomly cropping 160 × 160 regions, with random uniform scaling in [0.75, 1.333].Random affine transformations were omitted because they increased car–truck confusion in validation.
  • B.1 Training procedure: Competition augmentation additionally applied intensity scaling in [0.75, 1.333], normally distributed rotations with standard deviation 0.2π, and random desaturation to greyscale.
  • B.1 Training procedure: Colour-space augmentation included rotations around randomly chosen axes with standard deviation 0.05π and random offsets after ResNet-152 standardisation.

D Network architectures

The paper documents network architectures, benchmark-result tables, dataset resources, and VisDA-17 validation and test results, but the supplied architecture passage gives no layer-level details.

  • D Network architectures: The supplied material identifies architecture tables for MNIST ↔USPS, MNIST ↔SVHN, CIFAR-10 ↔STL, Syn-Digits →SVHN, and Syn-signs →GTSRB.
  • D Network architectures: The related-results material records prior-work markers and notes that RevGrad results were drawn from two papers to cover all displayed experiments.
  • D Network architectures: Table 1 reports small-image benchmark accuracy as mean ± standard deviation over 5 independent runs, using MT, CT, TF, TFA, and minimal-augmentation variants.
  • D Network architectures: The supplied table metadata identifies Table 2 as VisDA-17 performance over 5 independent runs and points to full validation and test results in Tables 4 and 5.
  • D Network architectures: The dataset and supplementary table references include datasets, full VisDA-17 validation and test results, and architecture tables, but no underlying cell values are supplied here.
Loading 1706.05208v4…