PaperlayerView exact revision on arXiv

Source-linked AI summary

Balanced Distribution Adaptation for Transfer Learning

Jindong Wang, Yiqiang Chen, Shuji Hao, Wenjie Feng, Zhiqi Shen

arXiv:1807.00516v1cs.LGstat.ML

TL;DR

Transfer learning methods often treat marginal and conditional distribution discrepancies equally and commonly assume balanced classes, limiting adaptation for varied tasks. The paper proposes BDA to adaptively weight these discrepancies and W-BDA to adapt class weights, with experiments showing strong performance across transfer-learning tasks.

  • Problem

    Existing transfer-learning methods may equally weight marginal and conditional discrepancies and assume balanced class distributions, despite variation across tasks and domains.

  • Method

    BDA adaptively weights marginal and conditional distribution adaptation, while W-BDA extends it by adaptively changing each class's weight.

  • Results

    BDA outperforms most existing methods on 15 of 16 tasks, averaging 55.56% classification accuracy and a 2.38% improvement over JDA.

  • Takeaways & Limitations

    Experiments on five image datasets demonstrate the superiority of BDA and W-BDA over several state-of-the-art methods.

Abstract

from arXiv · show

Transfer learning has achieved promising results by leveraging knowledge from the source domain to annotate the target domain which has few or none labels. Existing methods often seek to minimize the distribution divergence between domains, such as the marginal distribution, the conditional distribution or both. However, these two distances are often treated equally in existing algorithms, which will result in poor performance in real applications. Moreover, existing methods usually assume that the dataset is balanced, which also limits their performances on imbalanced tasks that are quite common in real problems. To tackle the distribution adaptation problem, in this paper, we propose a novel transfer learning approach, named as Balanced Distribution \underline{A}daptation~(BDA), which can adaptively leverage the importance of the marginal and conditional distribution discrepancies, and several existing methods can be treated as special cases of BDA. Based on BDA, we also propose a novel Weighted Balanced Distribution Adaptation~(W-BDA) algorithm to tackle the class imbalance issue in transfer learning. W-BDA not only considers the distribution adaptation between domains but also adaptively changes the weight of each class. To evaluate the proposed methods, we conduct extensive experiments on several transfer learning tasks, which demonstrate the effectiveness of our proposed algorithms over several state-of-the-art methods.

I. INTRODUCTION

The paper identifies two challenges in transfer learning: equally weighting marginal and conditional distribution discrepancies, and adapting across imbalanced class proportions. It proposes BDA and W-BDA to address these challenges and evaluates them on five image datasets.

  • Motivation: Existing distribution adaptation methods often treat marginal and conditional discrepancies equally, although their relative importance varies across transfer tasks.The paper states that marginal distributions may dominate when domains are dissimilar, while conditional distributions need more attention when domains are similar.
  • Motivation: Class imbalance across domains is common, but existing methods often assume balanced classes or correct bias in only one domain.The paper identifies this as a separate challenge that can hinder transfer learning effectiveness.
  • Contributions: BDA adaptively weights marginal and conditional distribution adaptation, with several existing methods represented as special cases.This lets BDA adjust distribution adaptation to specific transfer learning tasks.
  • Contributions: W-BDA extends BDA by adaptively changing each class's weight while performing distribution adaptation.The method targets class imbalance in transfer learning without abandoning domain distribution adaptation.
  • Evaluation: Experiments on five image datasets evaluate BDA and W-BDA against other state-of-the-art methods.The paper reports extensive experiments designed to assess both proposed methods.

II. RELATED WORK

The related work covers feature-based transfer learning, emphasizing joint distribution adaptation and class-imbalance methods. It positions the proposed approach as a feature-based method that adaptively handles distribution importance and class weights.

  • Scope: Feature-based transfer learning methods are discussed in relation to applications including activity recognition, incremental learning, and online learning.The paper focuses its detailed related-work discussion on feature-based transfer learning.
  • Joint distribution adaptation: Joint distribution adaptation methods match marginal and conditional distributions but commonly combine them without modeling their differing importance.The paper argues that this may limit generalization when the two distributions have large discrepancies.
  • Class imbalance problem: Prior class-imbalance approaches differ in whether they reweight samples, classes, domains, or source and target priors.The paper contrasts its feature-based method with CCSL, source-only prior adaptation, partially labeled-target methods, and prediction-based sample weighting.
  • Problem formulation: The proposed transfer-learning formulation assumes shared feature and label spaces but different marginal and conditional distributions, with unlabeled target labels to be learned.Balanced distribution adaptation minimizes discrepancies between both domain-level marginal and conditional distributions.

B. Balanced Distribution Adaptation

BDA minimizes marginal and conditional distribution discrepancies while adaptively weighting their importance for each transfer task. It estimates these discrepancies with MMD and uses target-domain predictions to approximate conditional information without target labels.

  • BDA uses a balance factor µ to adapt the relative importance of marginal and conditional distribution discrepancies.µ approaches 0 when domains are more dissimilar and 1 when domains are more similar.
  • Because target labels are unavailable, BDA approximates conditional information with class-conditional distributions and iteratively refined soft target predictions.A base classifier trained on the source domain supplies initial target-domain soft labels.
  • Maximum mean discrepancy empirically estimates both distribution divergences in BDA.MMD is used as a nonparametric measurement of the two discrepancies.
  • The objective combines marginal and conditional adaptation terms with a regularization term and constraints on the transformed data and µ.The first constraint preserves inner properties of the original data, while the second constrains the balance factor.
  • The transformed representation is obtained by solving a generalized eigendecomposition problem and selecting the d smallest eigenvectors.The data matrix combines source and target samples, while MMD matrices encode marginal and class-conditional discrepancies.
  • The balance factor µ lacks an effective direct estimation method and is selected experimentally or by cross-validation in real applications.The paper treats µ as data-dependent rather than a free parameter like λ.

C. Weighted Balanced Distribution Adaptation

W-BDA extends BDA to class-imbalanced transfer learning by balancing class proportions while adapting marginal and conditional distributions. It uses an iterative feature transformation and classifier-refinement procedure, with kernelization available for nonlinear problems.

  • Weighted Balanced Distribution Adaptation: W-BDA approximates conditional distributions using class priors from the source and target domains to balance class proportions.The method assumes marginal distributions remain unchanged during this weighting step.
  • Weighted Balanced Distribution Adaptation: The W-BDA objective embeds its weighting matrix into a trace optimization problem.
  • Weighted Balanced Distribution Adaptation: Eq. (10) considers both class sample counts and class priors, providing a more accurate conditional-distribution approximation under class imbalance.
  • Weighted Balanced Distribution Adaptation: For nonlinear problems, W-BDA can use a kernel map with a kernel matrix constructed by a linear or RBF kernel.
  • Weighted Balanced Distribution Adaptation: BDA and W-BDA solve eigendecomposition problems, use the d smallest eigenvectors to construct A, and iteratively retrain a classifier until convergence.The classifier is trained on the transformed labeled source data, while target soft labels are updated during the procedure.

IV. EXPERIMENTS

The experiments evaluate the proposed methods on transfer learning tasks using extensive empirical comparisons.

  • IV. EXPERIMENTS: The paper evaluates the proposed methods through extensive experiments.

A. Datasets

The evaluation uses five digit and object datasets spanning handwritten digits, object recognition, and real-world domain adaptation.

  • A. Datasets: The datasets include USPS, MNIST, COIL20, Office, and Caltech-256 for digit and object recognition tasks.Office contains Amazon, Webcam, and DSLR domains; the datasets vary substantially in image counts and category counts.

B. Comparison Methods

The comparison covers traditional classifiers and several state-of-the-art transfer learning methods, including marginal, component, subspace, and joint-distribution approaches.

  • B. Comparison Methods: 1NN and PCA are traditional learning methods, whereas GFK, TCA, JDA, and TSL are state-of-the-art transfer learning approaches.

C. Implementation Details

The experiments use dimensionality reduction followed by 1NN, with method-specific settings for BDA and W-BDA. BDA and W-BDA are evaluated across transfer-learning tasks, including the 16-task accuracy comparison in Table II.

  • PCA, TCA, JDA, TSL, and BDA use dimensionality reduction followed by 1NN classification.
  • GFK applies 1NN after constructing the geodesic flow kernel.
  • BDA and W-BDA search µ over {0, 0.1, · · ·, 0.9, 1.0}.
  • The comparison study sets d = 100, with λ = 0.1 for MNIST + USPS and Office + Caltech, and λ = 0.01 for COIL20.
  • Table II reports accuracy (%) for BDA and other methods on 16 tasks.

D. Performance Evaluation of BDA

Across 16 cross-domain tasks, BDA outperforms most existing methods and achieves the highest reported performance among the compared approaches. The results also describe differences among TCA, JDA, TSL, and GFK across task types.

  • 1) Classification accuracy:: BDA outperforms existing methods on 15 out of 16 tasks, with 55.56% average classification accuracy and a 2.38% improvement over JDA.JDA is treated as the BDA special case µ = 0.5, while BDA adjusts µ across tasks.
  • 1) Classification accuracy:: JDA adapts marginal and conditional distributions with equal weight, whereas BDA adjusts their balance parameter for different transfer-learning scenarios.
  • 1) Classification accuracy:: TCA is a BDA special case with µ = 0 because it adapts only the marginal distribution, and its performance is worse than JDA and BDA.
  • 1) Classification accuracy:: GFK performs better on object-recognition tasks, while its smooth transit may be insufficient for digit tasks when marginal distribution distance is large.
  • 1) Classification accuracy:: All transfer-learning methods outperform traditional learning approaches, with BDA achieving the best performance among them.

2) Effectiveness of distribution adaptation:

BDA’s balance factor µ controls the relative emphasis on marginal and conditional distribution adaptation. Experiments show that its optimal value varies across tasks, while some tasks exhibit multiple optima or unstable trends.

  • 2) Effectiveness of distribution adaptation:: MMD distances decrease for TCA, JDA, and BDA, while BDA achieves the best performance and JDA has a larger MMD distance than BDA.TCA’s MMD distance decreases less because it only adapts the marginal distribution and requires no iteration.
  • 2) Effectiveness of distribution adaptation:: The optimal µ varies across tasks, demonstrating the importance of balancing marginal and conditional distributions rather than treating them equally.JDA corresponds to the equal-weight special case µ = 0.5.
  • 2) Effectiveness of distribution adaptation:: For A →W, optimal µ = 0.8 indicates similar marginal distributions and greater dependence on conditional distributions.
  • 2) Effectiveness of distribution adaptation:: For M →U, optimal µ = 0.1 indicates that marginal distributions contribute most to the discrepancy.
  • 2) Effectiveness of distribution adaptation:: Some tasks have more than one optimal µ, and the tendency of µ is not always stable in CO2 →CO1.The paper identifies these issues as future research problems.

F. Effectiveness of Weighted BDA

W-BDA extends BDA to class-imbalanced transfer learning by adaptively weighting classes. It improves performance on imbalanced tasks and remains comparable to BDA when class distributions are balanced.

  • F. Effectiveness of Weighted BDA: On the first four highly imbalanced tasks, W-BDA improves performance by adaptively weighting each class.
  • F. Effectiveness of Weighted BDA: On two tasks with relatively balanced class distributions, W-BDA achieves comparable results.
  • F. Effectiveness of Weighted BDA: BDA outperforms JDA in all reported situations because it handles the discrepancy between marginal and conditional distributions.
  • F. Effectiveness of Weighted BDA: Sensitivity evaluations for feature dimension d and regularization parameter λ are omitted due to page limits, although both methods are relatively robust to them.
  • F. Effectiveness of Weighted BDA: W-BDA adaptively changes each class’s weight while performing distribution adaptation to address class imbalance.
  • F. Effectiveness of Weighted BDA: The conclusion reports extensive experiments on five image datasets demonstrating superiority over several state-of-the-art methods.
Loading 1807.00516v1…