PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoDIAL: Automatic DomaIn Alignment Layers

Fabio Maria Carlucci, Lorenzo Porzi, Barbara Caputo, Elisa Ricci, Samuel Rota Bulò

arXiv:1704.08082v3cs.CV

TL;DR

Domain adaptation is needed because classifiers trained on source data generalize poorly when source and target distributions differ, especially without target labels. AutoDIAL embeds Domain Alignment layers that match feature distributions to a reference and learn alignment strength across network levels, while exploiting unlabeled target data. Experiments report that it outperforms state-of-the-art methods and consistently improves performance across CNN architectures.

  • Problem

    Domain adaptation addresses poor classifier generalization when source and target distributions differ, including settings where target labels are unavailable.

  • Method

    AutoDIAL embeds Domain Alignment layers that match source and target feature distributions to a canonical reference while automatically learning alignment degree at each network level.

  • Results

    AutoDIAL outperforms state-of-the-art methods and consistently improves domain-adaptation performance across different CNN architectures.

  • Takeaways & Limitations

    The framework provides unsupervised adaptation that simultaneously aligns feature representations and exploits unlabeled target data during learning.

  • Takeaways & Limitations

    The paper focuses on unsupervised domain adaptation, while semi-supervised use is identified as future work.

Abstract

from arXiv · show

Classifiers trained on given databases perform poorly when tested on data acquired in different settings. This is explained in domain adaptation through a shift among distributions of the source and target domains. Attempts to align them have traditionally resulted in works reducing the domain shift by introducing appropriate loss terms, measuring the discrepancies between source and target distributions, in the objective function. Here we take a different route, proposing to align the learned representations by embedding in any given network specific Domain Alignment Layers, designed to match the source and target feature distributions to a reference one. Opposite to previous works which define a priori in which layers adaptation should be performed, our method is able to automatically learn the degree of feature alignment required at different levels of the deep network. Thorough experiments on different public benchmarks, in the unsupervised setting, confirm the power of our approach.

1. Introduction

Unsupervised domain adaptation addresses the failure of classifiers to generalize when source and target distributions differ. AutoDIAL embeds Domain Alignment layers that align representations and automatically learn the required adaptation across network levels.

  • Domain shift persists because real-world training and test data often are not identically distributed, even with powerful deep models.
  • Unsupervised domain adaptation is especially relevant when target-domain labels are tedious or impossible to obtain.
  • AutoDIAL embeds Domain Alignment layers at different network levels to align source and target feature distributions to a canonical reference.
  • The learned alignment parameters automatically determine how much adaptation is applied at each network level, rather than fixing adapted layers in advance.
  • AutoDIAL combines feature alignment with an entropy loss that exploits unlabeled target data by encouraging separated target classes.
  • AutoDIAL outperforms state-of-the-art methods and consistently improves domain-adaptation performance across different CNN architectures.

2. Related Work

Related work mainly reduces source–target discrepancy through instance re-weighting or feature alignment. AutoDIAL extends normalization-based adaptation by learning alignment strength at each network level and using target data through a parameter prior.

  • Unsupervised domain adaptation has traditionally used instance re-weighting, assigning source samples importance according to their similarity to target data.
  • Feature-alignment methods project source and target data into a common subspace to reduce distributional distance across shallow and deep models.
  • Deep feature-alignment approaches commonly minimize Maximum Mean Discrepancy or use domain-adversarial learning to maximize domain confusion.
  • AutoDIAL generalizes Adaptive Batch Normalization by automatically tuning alignment at each deep-network level and adding a prior that uses target samples during training.

3. Automatic DomaIn Alignment Layers

The method uses domain-specific layers within paired source and target networks to align feature distributions while learning how much alignment each network level requires. Training combines source supervision with target-domain information through a target-dependent prior and entropy loss.

  • Unsupervised domain adaptation uses labeled source data and unlabeled target data, whose differing distributions make direct target classification difficult.
  • DA-layers transform source and target features toward a reference distribution while the two predictors share network structure and weights.
  • The mixing factor α controls whether normalization is independent across domains or coupled, allowing alignment strength to vary by network level.α = 1 gives independent alignment, whereas α = 0.5 yields coupled normalization; α is learned during training.
  • Unlike methods that preselect adapted layers, the approach learns both where adaptation occurs and how strongly it is applied.
  • Training estimates shared parameters from source and target observations using a source likelihood, a target-dependent prior, and source log-loss plus target entropy loss.The target-dependent prior favors models with well-separated classes and incorporates unlabeled target information in the discriminative model.

4. Experiments

Experiments evaluate AutoDIAL on three public domain-adaptation benchmarks using AlexNet and Inception-BN, including ablations of its alignment and entropy components. Across benchmarks, AutoDIAL generally achieves the strongest or near-strongest performance, while learned alignment is concentrated more strongly in lower layers.

  • Experimental Setup: The evaluation covers three publicly available datasets, including Office31, Office-Caltech, and the Caltech-ImageNet Cross-Dataset Testbed.Office31 uses all source/target domain combinations; Office-Caltech restricts combinations involving Caltech256; Caltech-ImageNet is evaluated in two settings.
  • Experimental Setup: AutoDIAL is applied to AlexNet and Inception-BN, with DA-layers appended to AlexNet fully connected layers and replacing batch-normalization layers in Inception-BN.The experiments use pretrained networks, with the first three Inception blocks frozen and their alignment coefficient fixed at 0.5.
  • Analysis of the proposed method: Lower layers learn α values closer to 1, indicating greater adaptation than layers nearer the classifier, while α converges within a few training iterations.This pattern is more pronounced for AutoDIAL-AlexNet than for AutoDIAL-Inception-BN.
  • Comparison with State of the Art methods: Among AlexNet-based Office31 methods, AutoDIAL has the best average performance, and it also outperforms all baselines with Inception-BN.Compared with AdaBN, the reported improvement is attributed to learned alignment parameters α and entropy regularization.
  • Comparison with State of the Art methods: 90.6% vs 90.4% on average, AutoDIAL and RTN have roughly the same Office-Caltech performance while significantly outperforming the other baselines.The comparison is restricted to AlexNet-based deep methods.
  • Comparison with State of the Art methods: AutoDIAL significantly outperforms previous methods and sets the new state of the art on the Caltech-ImageNet subset.The paper attributes this result to the alignment strategy combined with entropy regularization, beyond the use of Inception-BN alone.

5. Conclusions

AutoDIAL is an unsupervised deep domain-adaptation framework that aligns source and target feature distributions while learning where and how strongly to adapt. Experiments report that it outperforms state-of-the-art methods, with semi-supervised use identified as a future direction.

  • 5. Conclusions: AutoDIAL introduces Domain Alignment layers that match source and target distributions to a reference distribution.The layers are embedded at multiple network levels.
  • 5. Conclusions: The network learns alignment parameters that determine the required adaptation degree at each layer.This lets the CNN align representations while deciding where adaptation should occur.
  • 5. Conclusions: AutoDIAL uses target data both for DA-layer statistics and an entropy loss on unlabeled samples.The entropy loss promotes high-confidence classifications.
  • 5. Conclusions: The experiments demonstrate that AutoDIAL outperforms state-of-the-art domain-adaptation methods.The evaluation includes results summarized for AlexNet-based, Inception-based, Office-Caltech, and Cross-Dataset Testbed experiments.
  • 5. Conclusions: The paper focuses on unsupervised domain adaptation, while semi-supervised use is identified as a future direction.The authors state that the approach can also be exploited when some target labels are available.

Appendix

The appendix adds implementation formulas, an additional benchmark, and feature-distribution visualizations for AutoDIAL.

  • Appendix: The appendix provides explicit batch-statistics formulas and back-propagation equations for DA-layers.These materials are collected in Section A.
  • Appendix: Section B reports results on the SVHN–MNIST benchmark.This extends the empirical evaluation beyond the main-paper benchmarks.
  • Appendix: Section C provides examples of feature distributions learned by AutoDIAL–Inception-BN on Office31.The examples are intended to show learned feature distributions.

Appendix A. DA-layers formulas

Appendix A makes the DA-layer computation explicit by defining source-target batch statistics, cross-normalized outputs, and their derivatives for learning.

  • Appendix A. DA-layers formulas: The DA-layer equations rewrite the main-paper formulation with explicit source and target sample indexes.The rewritten formulation supports sample-level definitions of the layer computation.
  • Appendix A. DA-layers formulas: The batch statistics mix source and target means and variances using the domain-mixing parameter α.The statistics are computed from ns source and nt target samples in a batch.
  • Appendix A. DA-layers formulas: The appendix derives gradients with respect to the inputs and α, enabling the alignment parameters to be learned by back-propagation.As with standard Batch Normalization, the gradients do not depend on the layer inputs for the stated implementation property.

Appendix B. Results on the SVHN – MNIST benchmark

The appendix evaluates AutoDIAL on SVHN-to-MNIST transfer using all SVHN images as source and all MNIST images as target. AutoDIAL sets a new state of the art while using only one extra learnable parameter per DA-layer.

  • Appendix B. Results on the SVHN – MNIST benchmark: The SVHN-to-MNIST experiment uses all SVHN images as source data and all MNIST images as target data.The evaluation follows the experimental protocol in [10].
  • Appendix B. Results on the SVHN – MNIST benchmark: The comparison includes CORAL, DAN, DANN, DRCN, DSN, and ATN.These methods serve as the benchmark baselines.
  • Appendix B. Results on the SVHN – MNIST benchmark: AutoDIAL sets a new state of the art on the SVHN-to-MNIST benchmark.The result is reported in Table 6.
  • Appendix B. Results on the SVHN – MNIST benchmark: AutoDIAL outperforms capacity-expanding ATN and DSN while using only a single extra learnable parameter in each DA-layer.The learned α values are higher near the network bottom and lower near the top, with a steeper transition between convolutional and fully connected layers.

Appendix C. Feature distributions

AutoDIAL’s Domain Alignment Layers closely align source and target feature distributions across network layers. The learned alignment effect varies with layer depth rather than being imposed uniformly.

  • Feature distributions: Feature histograms from source and target sets are very closely matched after passing through AutoDIAL’s Domain Alignment Layers.The comparison uses randomly sampled features from different layers of AutoDIAL–Inception-BN trained on Amazon–DSLR.
  • Feature distributions: The alignment effect appears to depend on the network level rather than requiring every layer to pursue the same degree of adaptation.
  • Feature distributions: The learned α parameters are analyzed as a function of layer depth on the SVHN–MNIST dataset.
  • Feature distributions: The feature-distribution analysis examines randomly sampled source and target features from different AutoDIAL–Inception-BN layers on Office 31.
Loading 1704.08082v3…