PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-Specific Batch Normalization for Unsupervised Domain Adaptation

Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, Bohyung Han

arXiv:1906.03950v1cs.LGcs.AIstat.ML

TL;DR

Unsupervised domain adaptation must transfer knowledge from labeled source data to unlabeled target data despite domain shift and shared-network limitations. The paper introduces DSBN with two-stage pseudo-labeling and multi-task training, reporting state-of-the-art performance on standard benchmarks and extension to multiple sources.

  • Problem

    Unsupervised domain adaptation transfers knowledge from annotated source domains to unlabeled target domains, but many methods share the whole network despite differing domain characteristics.

  • Method

    DSBN assigns separate batch-normalization branches to domains while sharing other parameters, using pseudo-label estimation followed by multi-task classification training.

  • Results

    The framework achieves state-of-the-art performance on standard benchmarks including Office-31 and VisDA-C and reports significantly improved results in multi-source adaptation.

  • Takeaways & Limitations

    DSBN is generically applicable to deep neural network adaptation methods with batch-normalization layers and can be extended to multiple source domains.

Abstract

from arXiv · show

We propose a novel unsupervised domain adaptation framework based on domain-specific batch normalization in deep neural networks. We aim to adapt to both domains by specializing batch normalization layers in convolutional neural networks while allowing them to share all other model parameters, which is realized by a two-stage algorithm. In the first stage, we estimate pseudo-labels for the examples in the target domain using an external unsupervised domain adaptation algorithm---for example, MSTN or CPUA---integrating the proposed domain-specific batch normalization. The second stage learns the final models using a multi-task classification loss for the source and target domains. Note that the two domains have separate batch normalization layers in both stages. Our framework can be easily incorporated into the domain adaptation techniques based on deep neural networks with batch normalization layers. We also present that our approach can be extended to the problem with multiple source domains. The proposed algorithm is evaluated on multiple benchmark datasets and achieves the state-of-the-art accuracy in the standard setting and the multi-source domain adaption scenario.

1. Introduction

Unsupervised domain adaptation transfers knowledge from labeled source data to unlabeled target data despite domain shift. The paper separates domain-specific information with DSBN and trains the resulting framework in two stages, achieving state-of-the-art benchmark performance and supporting multiple sources.

  • Motivation: Unsupervised domain adaptation transfers knowledge from annotated source domains to unlabeled target domains affected by domain shift.Domain shift means that source and target datasets have different characteristics.
  • Motivation: Shared whole-network training can limit adaptation because source and target domains contain different characteristics that may not fit one model.The paper motivates separating domain-specific information from domain-invariant information.
  • Method: Domain-Specific Batch Normalization uses separate batch-normalization branches for each domain while remaining applicable to neural networks with batch-normalization layers.DSBN captures domain-specific information through BN parameters and transforms domain-specific data into domain-invariant representations.
  • Method: The two-stage method first estimates target pseudo-labels with an existing adaptation network and then trains a multi-task classifier using source labels and target pseudo-labels.The second stage can iteratively refine target labels.
  • Results: The framework extends to multiple source domains and achieves state-of-the-art performance on standard benchmarks including Office-31 and VisDA-C.The reported benchmark result integrates the framework with two recent domain adaptation techniques.

2. Related Work

Related work addresses domain adaptation through global or local distribution alignment and through separation of domain-specific and shared representations. The paper instead uses batch-normalization parameters to capture domain-specific information within a pseudo-label-based framework.

  • Alignment approaches: Existing approaches learn domain-invariant representations by aligning source and target distributions globally or locally.Global methods include discrepancy and adversarial objectives, while local methods align examples with shared class semantics.
  • Domain-specific representations: Other methods jointly learn domain-specific and domain-invariant information by separating network components or imposing layer-specific properties.Examples include Domain Separation Network and Collaborative and Adversarial Network.
  • Batch normalization: Batch-normalization parameters have been used to model domain-specific information through target-statistics re-estimation, distribution alignment, or latent-source discovery.These approaches provide related uses of batch normalization in adaptation.
  • Pseudo-label methods: Pseudo-label-based methods estimate target labels and learn domain-specific target models using semantic matching, class-probability alignment, or related objectives.MSTN uses semantic matching and domain-adversarial losses, whereas CPUA uses class scores for adversarial learning.
  • Paper positioning: The proposed framework aims to produce more reliable pseudo-labels by capturing domain-specific information with domain-specific batch normalization.This connects the paper’s DSBN mechanism to its pseudo-labeling strategy.

3. Preliminaries

The preliminaries formulate adaptation from labeled source data and unlabeled target data, then review MSTN and CPUA as integration backbones. MSTN combines classification, domain-adversarial, and semantic-matching losses, while CPUA aligns class probabilities and addresses class imbalance.

  • Problem setting: Unsupervised domain adaptation uses labeled source data and unlabeled target data to classify target examples by transferring supervised source knowledge.The source and target datasets are denoted X_S and X_T, with cardinalities n_S and n_T.
  • MSTN: MSTN aligns centroids of same classes across domains using pseudo-labels for unlabeled target samples.Its objective includes source classification, domain-adversarial, and semantic-matching terms.
  • MSTN: MSTN’s total loss is L = L_cls(X_S) + λL_da(X_S, X_T) + λL_sm(X_S, X_T).L_cls is source cross-entropy; L_da encourages domain confusion; L_sm aligns same-class centroids.
  • CPUA: CPUA aligns class probabilities across domains and introduces class-weighted loss terms to address class imbalance using class priors.Target class fractions are computed from pseudo-labels.
  • Notation: The classification network F, cross-entropy loss ℓ, and domain discriminator D define the components used by the reviewed adaptation objectives.These symbols are explicitly identified in the preliminaries.

4. Domain-Specific Batch Normalization

DSBN separates domain-specific batch-normalization statistics and affine parameters while sharing the rest of the network, allowing common information and domain-specific properties to be modeled separately. It can be integrated into existing adaptation networks and extended to multiple source domains.

  • Batch Normalization: BN normalizes mini-batch activations and transforms them with affine parameters γ and β, using running statistics during testing.Sharing these statistics across domains is inappropriate when domain shift is significant.
  • Domain-Specific Batch Normalization: DSBN keeps non-BN network parameters shared while using domain-specific BN parameters to capture domain-specific properties.The design is intended to remove domain-specific information effectively and support learning domain-invariant features.
  • Domain-Specific Batch Normalization: DSBN assigns separate BN branches, affine parameters, and training statistics to the source and target domains.Each example selects the branch corresponding to its domain, and the corresponding estimated statistics are used at testing.
  • Implementation: An existing BN-based classification network becomes domain-specific by replacing its BN layers with DSBN layers and supplying domain labels.The resulting network is specialized to either the source or target domain through the domain variable d ∈{S, T}.
  • Multi-Source Extension: For multiple source domains, DSBN adds more domain branches, while the training loss sums source-domain losses and retains the existing adaptation procedure.The alignment loss can be any loss for aligning source and target domains.

5. Domain Adaptation with DSBN

The framework first generates target pseudo-labels with a DSBN-equipped adaptation network, then trains source and target models using supervised classification losses. Iterative refinement updates target pseudo-labels with later model predictions and is reported to improve target accuracy.

  • Stage 1: The first stage trains an existing BN-based adaptation network with DSBN to generate initial pseudo-labels for unlabeled target data.MSTN and CPUA are used as example initial pseudo-label generators after replacing their BN layers with DSBN.
  • Stage 2: The second stage trains final source and target models using source ground-truth labels and target pseudo-labels with two classification losses.The target pseudo-labels are progressively refined during training.
  • Accuracy Improvement: The final second-stage model generally recognizes target-domain images more accurately because it uses initial pseudo-labels for training rather than only weak domain-alignment information.This comparison motivates iterative use of the second-stage model for further pseudo-label refinement.
  • Stage 2: Self-Training: The second-stage network is used iteratively to update target pseudo-labels, and the authors empirically observe improved target-domain classification accuracy.The approach begins with initial labels and progressively relies more on the second-stage predictions through λ, which increases from 0 to 1.

6. Experiments

Experiments evaluate DSBN on standard and multi-source domain-adaptation benchmarks, with ablations and visualization analyzing its effectiveness. The framework consistently improves baseline performance and achieves state-of-the-art results across reported settings.

  • Experimental Settings: The experiments use VisDA-C, Office-31, and Office-Home, with ResNet-101 for VisDA-C and ResNet-50 for the other datasets.VisDA-C contains two domains, Office-31 three, and Office-Home four.
  • Results: DSBN combined with MSTN achieves state-of-the-art performance on VisDA-C and reliably recognizes hard classes including knife, person, skate, and truck.The comparison uses first-stage and two-stage variants with MSTN and CPUA as pseudo-label generators.
  • Results: Models trained with DSBN in both stages achieve state-of-the-art performance on Office-31 and consistently outperform two baseline models.The framework improves performance when applied to both MSTN and CPUA.
  • Multiple Source Domains: In multi-source adaptation, DSBN models consistently outperform BN models across settings, with source-domain separation especially advantageous for the hard Office-31 task A.The separate setting assigns one DSBN branch to each source domain and the target domain.
  • Ablation Study: Ablations show that DSBN is crucial in both training stages, while second-stage DSBN produces large additional gains and ordinary BN is not helpful.The benefit is more pronounced for hard classes.
  • Feature Visualization and Iterative Learning: DSBN aligns same-class examples from the two domains more effectively in feature space, and iterative pseudo-label refinement further improves target classification accuracy.The visualization compares BN and DSBN representations, while iterative learning updates labels from the preceding model.

7. Conclusion

The paper introduces DSBN, which uses separate batch-normalization branches for each domain while sharing other network parameters. Combined with two-stage training, the framework performs strongly on standard benchmarks and extends to multi-source adaptation.

  • Conclusion: DSBN assigns separate batch-normalization branches to domains while sharing all other network parameters.The framework is designed for deep neural networks with batch-normalization layers.
  • Conclusion: The two-stage DSBN framework applied to MSTN and CPUA demonstrates outstanding performance on standard benchmark datasets.The first stage generates target pseudo-labels and the second stage learns final models using source labels and target pseudo-labels.
  • Conclusion: The framework extends to multi-source domain adaptation and reports significantly improved results compared with other methods.
Loading 1906.03950v1…