PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Class Weight Bias: Weighted Maximum Mean Discrepancy for Unsupervised Domain Adaptation

Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, Wangmeng Zuo

arXiv:1705.00609v1cs.CV

TL;DR

The paper addresses the failure of conventional MMD-based domain adaptation to account for differing class priors across source and target domains. It proposes weighted MMD with class-specific source weights and classification EM training, and reports that weighted approaches outperform MMD across architectures and datasets.

  • Problem

    Existing MMD-based domain adaptation methods generally ignore changing class prior distributions, or class weight bias, across domains.

  • Method

    Weighted MMD introduces class-specific auxiliary source weights and uses classification EM to assign target pseudo-labels, estimate weights, and update model parameters.

  • Results

    Weighted MMD outperforms MMD for domain adaptation across various CNN architectures and datasets.

  • Takeaways & Limitations

    Accounting for class prior distributions provides a domain discrepancy metric that the paper reports as better suited to domain adaptation under class weight bias.

  • Takeaways & Limitations

    Future work includes evaluating weighted MMD on non-CNN UDA models and applying it to other distribution-discrepancy tasks such as image generation.

Abstract

from arXiv · show

In domain adaptation, maximum mean discrepancy (MMD) has been widely adopted as a discrepancy metric between the distributions of source and target domains. However, existing MMD-based domain adaptation methods generally ignore the changes of class prior distributions, i.e., class weight bias across domains. This remains an open problem but ubiquitous for domain adaptation, which can be caused by changes in sample selection criteria and application scenarios. We show that MMD cannot account for class weight bias and results in degraded domain adaptation performance. To address this issue, a weighted MMD model is proposed in this paper. Specifically, we introduce class-specific auxiliary weights into the original MMD for exploiting the class prior probability on source and target domains, whose challenge lies in the fact that the class label in target domain is unavailable. To account for it, our proposed weighted MMD model is defined by introducing an auxiliary weight for each class in the source domain, and a classification EM algorithm is suggested by alternating between assigning the pseudo-labels, estimating auxiliary weights and updating model parameters. Extensive experiments demonstrate the superiority of our weighted MMD over conventional MMD for domain adaptation.

1. Introduction

The paper identifies class-prior differences as a neglected source of bias in MMD-based domain adaptation and proposes weighted MMD with class-specific source weights. A classification EM procedure estimates pseudo-labels, auxiliary weights, and model parameters, while experiments report improved adaptation performance.

  • Motivation: Class prior distributions vary across domains, creating class weight bias that existing MMD-based adaptation methods generally ignore.The bias can arise from changes in sample selection criteria and application scenarios.
  • Motivation: MMD can minimize discrepancy by preserving source class weights, which may wrongly estimate target samples when class weights differ.The paper identifies this behavior as unreasonable for domain adaptation under class weight bias.
  • Proposed approach: Weighted MMD introduces class-specific auxiliary weights to reweight source samples toward the target class-weight distribution.The approach addresses unknown target labels through auxiliary weight estimation jointly optimized with model parameters.
  • Proposed approach: A classification EM scheme alternates among target pseudo-label assignment, auxiliary-weight estimation, and model-parameter updates.The CEM procedure uses E-step, C-step, and M-step operations within the weighted domain adaptation network.
  • Results: Weighted MMD outperforms conventional MMD across various CNN architectures and datasets.The experiments verify the superiority of weighted MMD for domain adaptation.

2. Preliminaries and Related Work

The preliminaries define MMD as an RKHS-based distribution discrepancy used to learn domain-invariant representations in unsupervised adaptation. The related-work discussion highlights computational constraints and the unaddressed assumption of equal class priors, distinguishing weighted MMD through class-level source reweighting.

  • Preliminaries: Unsupervised domain adaptation trains transferable CNN features without target labels, commonly by minimizing source-target distribution discrepancy.The paper focuses on the setting where all target-sample labels are unknown during training.
  • MMD: MMD compares source and target distributions by mapping data into a reproducing kernel Hilbert space and matching mean embeddings.It is presented as an effective non-parametric distribution-comparison metric.
  • MMD: Pairwise MMD computation has quadratic time complexity, while an unbiased linear-complexity approximation supports mini-batch gradient computation.This approximation makes MMD suitable for CNN-based adaptation with stochastic gradient descent.
  • Limitations of conventional MMD: Existing MMD-based UDA methods assume equal source and target class priors, an assumption that may fail in real-world applications and degrade performance.The paper illustrates this issue as class weight bias across domains.
  • Related work: Unlike sample-specific reweighting or selection methods, weighted MMD assigns class-specific weights to source data to alleviate class weight bias.This distinction is presented as the method’s related-work novelty.

3. Weighted Maximum Mean Discrepancy

Weighted MMD measures domain discrepancy while accounting for differences in class prior probabilities. It constructs a source reference distribution with target-domain class weights and source-domain class-conditional distributions.

  • Motivation: In UDA, class-conditional discrepancy is preferable, but unknown target labels make ordinary MMD between marginal source and target distributions the usual choice.The class-conditional distributions are treated as the proper domain-discrepancy metric, while target-label unavailability motivates marginal MMD.
  • Motivation: When source and target class weights differ, ordinary MMD cannot reliably handle the resulting class weight bias.The proposed alternative addresses cases where source and target class prior probabilities are not equal.
  • Weighted MMD construction: Weighted MMD introduces class-specific auxiliary weights to reweight source samples and match the target domain’s class weights.The reweighted source distribution retains source class-conditional distributions while adopting target class weights.
  • Weighted MMD construction: The weighted MMD objective estimates the auxiliary weights and measures discrepancy between the reweighted source distribution and the target distribution.An empirical weighted MMD estimator and a linear-time approximation are provided.

4. Weighted Domain Adaptation Network

The weighted domain adaptation network integrates weighted MMD into a CNN classifier for unsupervised adaptation. Classification EM alternates between target pseudo-labeling, auxiliary-weight estimation, and model-parameter updates.

  • Model: WDAN extends a semi-supervised logistic-regression model by adding a weighted MMD term and incorporating CNN features.The model addresses unknown target class distributions during training.
  • Model: Weighted MMD regularizers are added to higher CNN layers because features become more task-specific and dataset-biased toward the network’s top layers.The regularizers are applied to architecture-specific higher-layer representations.
  • Objective: The WDAN objective combines source and target softmax losses with weighted MMD regularizers, using source class weights estimated from labeled source data.The source class weight is estimated as Mc/M, where Mc is the number of samples in class c and M is the total source-sample count.
  • Optimization: Classification EM alternates between estimating target class posteriors, assigning pseudo-labels, estimating auxiliary weights, and updating model parameters.The E-step computes posterior probabilities, the C-step assigns maximum-posterior pseudo-labels and estimates weights, and the M-step updates parameters by minimizing the objective.
  • Optimization: The extended classification-EM procedure can converge to a stationary value, and experiments empirically validate its auxiliary-weight estimation.This procedure adds auxiliary-weight estimation to the usual classification-EM pseudo-labeling step.

5. Experiments

The experiments evaluate WDAN across four UDA benchmarks and multiple CNN architectures, including analyses of sensitivity, class-weight-bias robustness, and feature visualization. The reported results use standard benchmark settings and compare methods across architectures.

  • Evaluation setup: WDAN is evaluated on Office-10+Caltech-10, Office31, ImageCLEF, and Digit Recognition benchmarks.The study also analyzes hyper-parameter sensitivity, robustness to class weight bias, and feature visualization.
  • Evaluation setup: The evaluation uses LeNet, AlexNet, GoogLeNet, and VGGnet-16 CNN architectures.Some architectures use ImageNet-pretrained models, while LeNet is trained from scratch.
  • Results: Table 1 reports results in percent for different methods using AlexNet, GoogLeNet, and VGGnet-16 on Office-10+Caltech-10.The table notes that some LapCNN, DDC, and DAN results are duplicated from prior work, while the star denotes use of ground-truth class distributions as priors.

5.1. Comparison with State-of-the-arts

Across multiple benchmarks, architectures, and class counts, WDAN generally achieves the strongest domain-adaptation performance and outperforms conventional MMD-based baselines.

  • ImageCLEF: 1.4%, 1.2%, and 0.9% average gains over GoogLeNet, DDC, and DAN, respectively, are reported on ImageCLEF.The comparison uses GoogLeNet-based implementations and reports averaged performance differences.
  • Digit Recognition: 11.7%, 10.4%, and 3.7% average improvements over LeNet, SA, and DAN, respectively, are reported across four digit-recognition tasks.WDAN achieves the best performance on all four tasks.
  • Office-31: WDAN outperforms its MMD counterpart on Office-31, supporting its effectiveness with 31 classes.The experiments cover all six Office-31 adaptation tasks using AlexNet.

5.2. Empirical Analysis

Empirical analyses examine λ sensitivity, class-weight-bias robustness, and learned feature geometry, with WDAN consistently favored over DAN in the reported comparisons.

  • Effect of Parameter λ: WDAN consistently outperforms DAN, while their best results occur at λ = 0.4 and λ = 0.1, respectively.WDAN and DAN outperform AlexNet when λ < 1.2 and λ < 1.0, respectively, showing the importance of balancing empirical loss and the MMD regularizer.
  • Impact of Class Weight Bias: Increasing class-weight bias significantly degrades conventional MMD-based methods, whereas WDAN is more robust.The test fixes source class weights at 0.5 each and gradually changes the target class distribution.
  • Feature Visualization: WDAN features preserve more class discrepancy distance than DAN features in the target-domain t-SNE visualization.The paper attributes this to the weighted MMD regularizer not minimizing class-weight bias as DAN does.

6. Conclusion

The paper addresses class weight bias in UDA with weighted MMD and WDAN, reporting improved performance over DAN across domain adaptation tasks. It also identifies unresolved extensions beyond CNN-based UDA and image classification.

  • 6. Conclusion: WDAN uses weighted MMD to reduce class weight bias by constructing a reference source distribution based on the target distribution.The method jointly assigns pseudo-labels, estimates auxiliary weights, and learns model parameters using a modified CEM algorithm.
  • 6. Conclusion: Empirical results show that WDAN outperforms its MMD counterpart, DAN, across various domain adaptation tasks.
  • 6. Conclusion: Future work includes evaluating weighted MMD on non-CNN UDA models and applying distribution-discrepancy measures to tasks such as image generation.
Loading 1705.00609v1…