PaperlayerView exact revision on arXiv

Source-linked AI summary

Transfer Learning with Dynamic Distribution Adaptation

Jindong Wang, Yiqiang Chen, Wenjie Feng, Han Yu, Meiyu Huang, Qiang Yang

arXiv:1909.08531v1cs.LGstat.ML

TL;DR

Transfer learning must handle source-target distribution differences, but existing approaches do not quantitatively assess the relative importance of marginal and conditional distributions. The paper proposes DDA, with MDDA for traditional and DDAN for deep transfer learning, and reports strong performance across digit, sentiment, and image-classification tasks. It also identifies differing distribution contributions as useful evidence for future transfer-learning research.

  • Problem

    Existing transfer-learning methods often align marginal or conditional distributions, or weight both equally, without quantitatively evaluating their relative importance.

  • Method

    DDA dynamically estimates marginal and conditional distribution weights and supports MDDA for traditional transfer learning and DDAN for deep transfer learning.

  • Results

    MDDA and DDAN achieve the best performance against other state-of-the-art traditional and deep transfer-learning methods on digit recognition, sentiment analysis, and image classification.

  • Takeaways & Limitations

    Marginal and conditional distributions can contribute differently to domain divergence, and quantitatively weighting them supports transfer learning.

  • Takeaways & Limitations

    The quantitative evaluation of the adaptive factor currently supports only source and target domains with the same label space.

Abstract

from arXiv · show

Transfer learning aims to learn robust classifiers for the target domain by leveraging knowledge from a source domain. Since the source and the target domains are usually from different distributions, existing methods mainly focus on adapting the cross-domain marginal or conditional distributions. However, in real applications, the marginal and conditional distributions usually have different contributions to the domain discrepancy. Existing methods fail to quantitatively evaluate the different importance of these two distributions, which will result in unsatisfactory transfer performance. In this paper, we propose a novel concept called Dynamic Distribution Adaptation (DDA), which is capable of quantitatively evaluating the relative importance of each distribution. DDA can be easily incorporated into the framework of structural risk minimization to solve transfer learning problems. On the basis of DDA, we propose two novel learning algorithms: (1) Manifold Dynamic Distribution Adaptation (MDDA) for traditional transfer learning, and (2) Dynamic Distribution Adaptation Network (DDAN) for deep transfer learning. Extensive experiments demonstrate that MDDA and DDAN significantly improve the transfer learning performance and setup a strong baseline over the latest deep and adversarial methods on digits recognition, sentiment analysis, and image classification. More importantly, it is shown that marginal and conditional distributions have different contributions to the domain divergence, and our DDA is able to provide good quantitative evaluation of their relative importance which leads to better performance. We believe this observation can be helpful for future research in transfer learning.

1 INTRODUCTION

Transfer learning addresses domain-shifted data by leveraging labeled source domains, but existing methods do not quantify the differing importance of marginal and conditional distributions. The paper introduces DDA and its MDDA and DDAN extensions, reporting improved performance across several transfer-learning tasks.

  • Motivation: Transfer learning leverages labeled source-domain samples to learn classifiers for a new target domain despite distribution divergence.Traditional machine-learning algorithms assume training and testing data share distributions, whereas transfer learning reduces domain divergence.
  • Research gap: Existing methods align marginal or conditional distributions, while methods aligning both commonly assign them equal weights.This equal-weight assumption fails to evaluate their relative importance when their contributions to domain divergence differ.
  • Proposed approach: Dynamic Distribution Adaptation (DDA) quantitatively learns marginal and conditional distribution weights using H∆H divergence during representation learning.DDA iteratively optimizes distribution-importance learning and feature learning to obtain a domain-invariant transfer classifier.
  • Proposed approach: MDDA applies DDA to traditional transfer learning, while DDAN applies it to deep transfer learning within structural risk minimization.MDDA uses the Grassmann manifold, and DDAN uses a deep neural network for end-to-end transfer classification.
  • Results: Experiments on digit classification, sentiment analysis, and image classification show that MDDA and DDAN outperform many state-of-the-art traditional and deep methods.The paper also reports that marginal and conditional distributions have different effects and that DDA assigns them quantitative weights.

2 RELATED WORK

Prior transfer-learning research includes instance reweighting, feature transformation, metric learning, distribution alignment, and deep or adversarial representation learning. DDA differs by dynamically evaluating the relative importance of marginal and conditional distributions rather than treating them equally.

  • Transfer-learning methods: Transfer-learning methods include instance reweighting, feature transformation, and transfer metric learning.Feature transformation seeks shared subspaces, while metric learning learns transferable distances between domains.
  • Subspace and manifold learning: Subspace and manifold methods learn domain-invariant representations, but prior approaches may ignore distribution alignment or conditional information.Examples include SA, SDA, GFK, and Grassmann-based adaptation methods.
  • Distribution alignment: Distribution-alignment methods target marginal, conditional, or joint distribution divergence.JDA aligns marginal and conditional distributions with equal weights, while BDA manually weights them.
  • Dynamic distribution adaptation: DDA dynamically evaluates the relative importance of marginal and conditional distributions, unlike methods that align them equally or only in the original feature space.The paper states that MDDA can perform alignment in a manifold to address feature distortion.
  • Deep and adversarial transfer learning: Deep transfer methods use losses such as MMD, CORAL, CMD, or adversarial objectives to learn transferable representations.DDC, DAN, Deep CORAL, CMD, and DANN represent these deep and adversarial approaches.

3 PRELIMINARIES

The paper formulates transfer learning with source and target domains whose marginal and conditional distributions may both differ, then incorporates distribution divergence into structural risk minimization. MMD provides the empirical divergence measure used in the framework.

  • Transfer-learning setting: The general transfer-learning setting allows both marginal and conditional distributions to differ between source and target domains.The objective is to learn a transferable classifier minimizing target-domain risk without target labels.
  • Structural risk minimization: Structural risk minimization combines sample loss with regularization in a kernel-induced Hilbert space.The labeled domain is the source domain because the target domain has no labels.
  • Distribution divergence: The regularization term includes source-target distribution divergence alongside other regularization, with trade-off parameters controlling their contributions.The divergence term is denoted Df(·,·), while Rf(·,·) denotes other regularization.
  • Maximum mean discrepancy: Maximum mean discrepancy (MMD) is adopted as a non-parametric measure for empirically calculating distribution divergence.MMD is defined through differences between mean embeddings in a reproducing kernel Hilbert space and estimated from samples.

4 DYNAMIC DISTRIBUTION ADAPTATION

Dynamic Distribution Adaptation (DDA) quantitatively adjusts the relative importance of marginal and conditional distributions during transfer learning. It supports both manifold-based MDDA for traditional learning and end-to-end DDAN for deep learning, with practical trade-offs in computation, feature extraction, and label-space assumptions.

  • 4 DYNAMIC DISTRIBUTION ADAPTATION: DDA dynamically learns distribution weights by measuring domain divergence, rather than assuming marginal and conditional distributions contribute equally.The adaptive factor is learned while representations are learned, allowing distribution alignment to reflect domain-specific importance.
  • 4 DYNAMIC DISTRIBUTION ADAPTATION: Random guessing and max-min averaging provide feasible alternatives for estimating the adaptive factor but are computationally prohibitive and lack guaranteed results.These alternatives repeatedly run the adaptation algorithm, unlike the proposed per-iteration evaluation.
  • 4 DYNAMIC DISTRIBUTION ADAPTATION: MDDA estimates relative distribution importance with A-distance measurements of marginal and class-conditional domain discrepancies.The estimate is recomputed during each adaptation iteration because feature distributions can change after conditional alignment.
  • 4 DYNAMIC DISTRIBUTION ADAPTATION: The quantitative evaluation currently assumes identical source and target label spaces, leaving open-set and partial transfer extensions for future work.The paper suggests outlier detection and similarity-based selection as possible extensions when label spaces differ.
  • 4 DYNAMIC DISTRIBUTION ADAPTATION: MDDA learns transferable manifold representations, while DDAN jointly learns features and a classifier through a deep neural network.DDAN aligns domains during end-to-end training and computes adaptation between mini-batches for streaming-data practicality.
  • 4 DYNAMIC DISTRIBUTION ADAPTATION: DDAN must avoid updating the adaptive factor from poorly matched source and target mini-batch labels, which can cause mode collapse or gradient explosion.The paper identifies this risk when batch size approaches the number of classes.
  • 4 DYNAMIC DISTRIBUTION ADAPTATION: MDDA suits resource-constrained devices but depends on feature extraction, whereas DDAN avoids separate extraction and classifier training and suits large-scale datasets.MDDA can also use deep features; DDAN is intended for cloud training followed by device inference.

5 EXPERIMENTS AND EVALUATIONS

The experiments evaluate MDDA on large-scale public datasets, with source code made available and a deep version assessed in the final part of the section.

  • 5 EXPERIMENTS AND EVALUATIONS: The evaluation focuses primarily on MDDA across large-scale public datasets and also assesses its deep version.The authors state that most contributions can be covered by the MDDA evaluation.

5.1 Experimental Setup

The study evaluates transfer learning across five public benchmark datasets comprising 38 tasks, comparing MDDA and DDAN with traditional and deep baselines. MDDA uses manifold features, while DDAN performs end-to-end adaptation from images with ResNet-50.

  • Datasets: The experiments use five public datasets spanning digit recognition, sentiment analysis, and image classification, totaling 38 transfer tasks.The datasets are USPS+MNIST, Amazon review, Office-31, ImageCLEF-DA, and Office-Home.
  • Datasets: USPS and MNIST provide two digit-recognition tasks, while Amazon reviews provide 12 cross-domain sentiment-analysis tasks.USPS and MNIST contain handwritten digits; Amazon covers Kitchen, DVDs, Books, and Electronics domains.
  • Datasets: Office-31, ImageCLEF-DA, and Office-Home contribute 24 image-classification tasks across multiple visual domains.These datasets contribute 6, 6, and 12 tasks, respectively.
  • Comparison methods: MDDA is compared with traditional and deep transfer-learning methods, including distribution-alignment, subspace-learning, baseline-network, and adversarial approaches.Comparison methods include TCA, JDA, GFK, CORAL, SCA, JGSA, AlexNet, ResNet, DDC, DAN, DANN, ADDA, and JAN.
  • Implementation: MDDA uses dataset-specific extracted features, whereas DDAN accepts original images and uses ResNet-50 as its baseline network.MDDA uses SURF, marginalized-denoising-autoencoder, or fine-tuned ResNet-50 features depending on the dataset.
  • Implementation: The evaluation uses target-domain classification accuracy, with MDDA and DDAN trained under specified manifold, kernel, regularization, and optimization settings.MDDA uses T = 10 iterations and an RBF kernel; DDAN uses learning rate 0.01, batch size 32, and weight decay 5e-4.

5.2 Results and Analysis

MDDA substantially outperforms existing methods across digit, sentiment, and image-transfer benchmarks, while DDAN also performs strongly on most image tasks. The results support the effectiveness of dynamic distribution adaptation for reducing domain divergence.

  • Digit datasets: 8.9% is MDDA’s improvement over the best digit-recognition method, JGSA, on USPS+MNIST tasks.The paper reports that MDDA significantly outperforms existing methods on these digit tasks.
  • Analysis: The results identify distribution alignment and subspace-learning methods as generally inferior to MDDA, while traditional transfer methods can still outperform baseline networks in some settings.The paper attributes remaining limitations to domain shift and feature distortion after manifold or subspace learning.
  • Sentiment datasets: 6.0% is MDDA’s improvement over the best Amazon-review baseline, CORAL, in cross-domain sentiment analysis.The authors associate this result with reduced divergence between text domains.
  • Image datasets: MDDA outperforms all traditional and deep comparison methods on 20/24 image-classification tasks.The image results cover Office-31, ImageCLEF-DA, and Office-Home.
  • Image datasets: 77.3% is MDDA’s average classification accuracy across all image tasks.On Office-Home, MDDA exceeds CDAN by 4.5%.
  • Deep adaptation: DDAN substantially outperforms traditional and deep methods on most tasks without adversarial training.The authors describe this architecture as more efficient than methods requiring adversarial neural networks because those methods need more time to converge.

5.3 Evaluation of Dynamic Distribution Adaptation

The evaluation tests whether marginal and conditional distributions contribute differently to transfer divergence and whether DDA can estimate their relative weight. Results indicate task-dependent importance, effective estimation, and lower search cost, while DDAN may require more tuning than MDDA.

  • Evaluation design: The experiments search µ values because no ground truth exists for the marginal-versus-conditional distribution weight.Performance under different µ values is used as an indirect check of the evaluation procedure.
  • Evaluation accuracy: DDA’s µ evaluation is significantly better than random search and average search.The comparison is reported using Fig. 4(b) and Table 7.
  • Distribution importance: The optimal µ varies across tasks, and multiple µ values can sometimes produce the same classification result.This observation indicates that the best distribution weighting is task-dependent.
  • Evaluation efficiency: DDA requires evaluation only once per algorithm iteration, whereas random and average search rerun the full MDDA or DDAN algorithm several times.The paper therefore reports DDA as more efficient for estimating µ.
  • Method comparison: On image-classification datasets, MDDA performs slightly better than DDAN under the reported tuning process.The authors note that MDDA is easier to tune and expect more extensive DDAN tuning could match or exceed MDDA.

5.4 Ablation Study

The ablation study finds that DDA is the most important component in both MDDA and DDAN, while manifold learning and Laplacian regularization generally provide smaller improvements.

  • DDA is the most important component and dramatically increases transfer-learning results in the ablation study.
  • MDDA's manifold feature learning provides marginal improvement while helping eliminate feature distortion in the original space.
  • DDAN's DDA component outperforms DAN, which adapts only marginal distributions.
  • Laplacian regularization generally provides marginal improvements, except on the USPS+MNIST digit datasets.The regularization is included because it helps the algorithms converge quickly.

5.5 Parameter Sensitivity and Convergence Analysis

MDDA and DDAN remain robust across broad hyperparameter ranges, with MDDA reaching steady performance in fewer than 10 iterations and DDAN converging quickly with steady performance.

  • MDDA is robust to manifold subspace dimensions d from 10 to 100 and neighbor counts p from 2 to 64.The authors state these parameters can be selected without in-depth application-specific knowledge.
  • MDDA achieves robust performance across wide regularization ranges, with best choices λ ∈[0.5, 1, 000], η ∈[0.01, 1], and ρ ∈[0.01, 5].The reported sensitivity analysis directly shows the tendency for λ; ρ and η follow the same tendency.
  • T < 10 iterations are sufficient for MDDA to reach steady performance, indicating a training advantage in cross-domain tasks.
  • DDAN is robust to its key parameters λ, p, and ρ, so its hyperparameters do not need to be cherry-picked in real applications.The authors note that this matters because tuning deep-learning hyperparameters is time-consuming.
  • DDAN converges quickly while maintaining steady performance.
  • Both MDDA and DDAN achieve efficient computing time while outperforming the comparison methods in the reported experiments.For image classification, the reported times for ARTL, JGSA, and MDDA include deep feature extraction and algorithm runtime.

6 CONCLUSIONS AND FUTURE WORK

The paper introduces DDA to learn the relative importance of marginal and conditional distributions, and builds MDDA and DDAN for traditional and deep transfer learning. Experiments across three task types report state-of-the-art performance, while future work extends DDA to broader transfer settings.

  • DDA dynamically evaluates the relative importance of source and target distributions in transfer learning.
  • MDDA applies DDA to traditional transfer learning, while DDAN applies it to deep transfer learning.
  • Experiments on digit recognition, sentiment analysis, and image classification report the best performance against state-of-the-art traditional and deep transfer-learning methods.
  • Future work will extend DDA to heterogeneous transfer learning and more complex transfer-learning situations.
Loading 1909.08531v1…