PaperlayerView exact revision on arXiv

Source-linked AI summary

Transfer Learning with Dynamic Adversarial Adaptation Network

Chaohui Yu, Jindong Wang, Yiqiang Chen, Meiyu Huang

arXiv:1909.08184v1cs.LGstat.ML

TL;DR

Deep adversarial adaptation must address cases where global and local domain distributions contribute differently, a relative importance that existing methods did not dynamically quantify. DAAN introduces a dynamic adversarial factor to learn domain-invariant representations and estimate that importance, with experiments showing superior performance across benchmark tasks.

  • Problem

    Existing adversarial domain adaptation methods align global or local distributions, but lack a dynamic quantitative evaluation of their relative importance when contributions differ across domains.

  • Method

    DAAN performs end-to-end adversarial training to learn domain-invariant features while using a Dynamic Adversarial Factor to evaluate marginal and conditional distribution importance.

  • Results

    DAAN outperforms comparison methods on most ImageCLEF-DA and Office-Home transfer tasks, while its ablation results show that equal or single-distribution alignment is insufficient.

  • Takeaways & Limitations

    Dynamic distribution alignment enables adversarial transfer learning to account for the relative importance of global and local distributions.

Abstract

from arXiv · show

The recent advances in deep transfer learning reveal that adversarial learning can be embedded into deep networks to learn more transferable features to reduce the distribution discrepancy between two domains. Existing adversarial domain adaptation methods either learn a single domain discriminator to align the global source and target distributions or pay attention to align subdomains based on multiple discriminators. However, in real applications, the marginal (global) and conditional (local) distributions between domains are often contributing differently to the adaptation. There is currently no method to dynamically and quantitatively evaluate the relative importance of these two distributions for adversarial learning. In this paper, we propose a novel Dynamic Adversarial Adaptation Network (DAAN) to dynamically learn domain-invariant representations while quantitatively evaluate the relative importance of global and local domain distributions. To the best of our knowledge, DAAN is the first attempt to perform dynamic adversarial distribution adaptation for deep adversarial learning. DAAN is extremely easy to implement and train in real applications. We theoretically analyze the effectiveness of DAAN, and it can also be explained in an attention strategy. Extensive experiments demonstrate that DAAN achieves better classification accuracy compared to state-of-the-art deep and adversarial methods. Results also imply the necessity and effectiveness of the dynamic distribution adaptation in adversarial transfer learning.

I. INTRODUCTION

Deep transfer learning uses adversarial adaptation to learn transferable, domain-invariant features, but existing methods typically emphasize either global or local alignment. DAAN dynamically evaluates their relative importance and achieves stronger benchmark performance.

  • Transfer learning leverages labeled source-domain samples because acquiring sufficient labeled target-domain data is expensive and time-consuming.
  • Existing adversarial methods generally align either global source-target distributions with one discriminator or local subdomains with multiple discriminators.DANN represents global adversarial learning, whereas other methods focus on subdomain alignment.
  • Marginal and conditional distributions can contribute differently to domain discrepancy, yet prior adversarial methods lacked a dynamic quantitative evaluation of their relative importance.Kernel-based BDA and MEDA adaptively align both distributions but incur computational or scalability limitations.
  • DAAN learns domain-invariant features through end-to-end adversarial training and introduces a Dynamic Adversarial Factor to quantify global-versus-local distribution importance.The method is trained with backpropagation and SGD in linear time.
  • DAAN is theoretically analyzed, can be interpreted through an attention strategy, and is designed for easy implementation in deep learning libraries.The attention interpretation concerns learning the importance of marginal and conditional distributions.
  • Extensive experiments report that DAAN outperforms comparison methods on most ImageCLEF-DA and Office-Home transfer tasks.The experiments assess both classification accuracy and estimation of the dynamic adversarial factor.

A. Unsupervised Domain Adaptation

Unsupervised domain adaptation learns discriminative models despite domain shifts, using traditional or deep approaches. Prior methods include subspace and distribution alignment, discrepancy-based deep adaptation, and adversarial alignment with global or multiple discriminators.

  • Unsupervised domain adaptation learns a discriminative model when source and target domains differ, using labeled source examples and unlabeled target examples.
  • Traditional methods include subspace learning, which aligns representations or statistics, and distribution alignment, which matches domain distributions.Examples include SA, SDA, CORAL, TCA, and JDA.
  • JDA matches marginal and conditional distributions but treats them equally, while BDA and MEDA adaptively evaluate their different effects.The cited approaches differ in how they weight the two distributions.
  • Deep domain adaptation commonly reduces distribution divergence using discrepancy measures such as CORAL, KL divergence, MMD, and CMD.
  • Adversarial domain adaptation uses a discriminator to distinguish source from target while a feature extractor learns domain-invariant features and a label classifier minimizes classification loss.The discriminator minimizes its domain loss, whereas the feature extractor maximizes that loss.
  • DANN aligns whole domains with a global discriminator, MADA performs fine-grained alignment with multiple discriminators, and related methods construct multiple feature spaces.

B. Adversarial Learning for Domain Adaptation

Adversarial domain adaptation trains feature extractors to confuse domain discriminators while preserving source-label discrimination. DAAN extends this framework by dynamically weighting global and local distribution alignment to address their differing contributions and scalability challenges.

  • Adversarial learning: Adversarial adaptation trains the feature extractor to maximize domain-discriminator loss, while the discriminator minimizes it and the label-classifier loss is minimized.This creates domain-invariant features while retaining source-domain label information.
  • Existing approaches: Existing methods separately emphasize marginal alignment with global discriminators or conditional alignment with multiple discriminators.DANN is cited as a global-alignment example, whereas other approaches align conditional or subdomain distributions.
  • Adaptation challenge: Both marginal and conditional distributions support domain-invariant feature learning, but their relative contributions can differ substantially across applications.The paper therefore frames dynamic, quantitative evaluation of their importance as a central adaptation challenge.
  • Adaptation challenge: MEDA estimates marginal and conditional weights but requires 1+C extra linear classifiers per iteration and pseudo-inverse computation over all samples.These requirements make MEDA computationally expensive, time-consuming, and unsuitable for large-scale or online deployment.
  • DAAN: DAAN combines a feature extractor, label classifier, global discriminator, local discriminators, and a Dynamic Adversarial Factor for adversarial adaptation.The factor evaluates the relative importance of marginal and conditional distributions, while GRL supports efficient training.

1) Label Classifier:

DAAN uses a source-supervised label classifier alongside global and local domain-distribution components. The classifier minimizes cross-entropy over source samples, while the architecture includes the feature extractor, discriminators, and gradient-reversal mechanism.

  • Label Classifier: The label classifier Gy predicts source-domain labels from features produced by the feature extractor Gf.Its training uses the supervised information available for the source domain.
  • Label Classifier: The label-classifier objective is a cross-entropy loss over source-domain samples and C classes.Pxi→c denotes the probability that xi belongs to class c.
  • DAAN architecture: DAAN’s architecture includes a feature extractor Gf, label classifier Gy, global discriminator Gd, and C local subdomain discriminators.The figure identifies these modules by blue, orange, purple, and green components, respectively.
  • Global Domain Discriminator: The global domain discriminator aligns marginal distributions between source and target domains using a cross-entropy domain-discriminator loss.Ld is the domain-discriminator loss, Gf is the feature extractor, and di is the input sample’s domain label.

3) Local Domain Discriminator:

DAAN’s local discriminator aligns class-conditional source and target distributions through class-wise discriminators weighted by label predictions. Its dynamic factor estimates whether global or local alignment should receive greater emphasis using discriminator losses and A-distances.

  • Local discriminator: The local domain discriminator aligns conditional distributions by matching source and target data associated with each class.It captures multi-mode structure and enables finer-grained adaptation than a global discriminator.
  • Local discriminator: Each class-wise discriminator receives attention weights derived from the label predictor’s output for each sample.The local loss uses predicted class probabilities together with domain labels.
  • Dynamic factor: The dynamic adversarial factor quantitatively evaluates the relative importance of marginal and conditional distributions using global and local discriminator distances.Global distributions correspond to marginal distributions, while local distributions correspond to conditional distributions.
  • Dynamic factor: DAAN updates the factor within the network from deep adversarial representations and discriminator losses, avoiding extra classifiers required by MEDA.The factor is initialized at 1, target pseudo-labels are obtained after each epoch, and distances are recalculated during training.

E. Learning Procedure

DAAN combines label prediction with global and local adversarial objectives, weighting the two alignment losses through a self-calculated factor. It can be optimized efficiently with SGD, while limiting the manually specified hyperparameters to λ.

  • Learning objective: DAAN integrates a label classifier, global domain discriminator, and local subdomain discriminator into one learning objective.The objective combines the classification and two domain-alignment components.
  • Learning objective: λ is the trade-off parameter in DAAN’s learning objective.The dynamic factor ω weights global versus local alignment but is calculated by the network.
  • Dynamic weighting: When ω approaches 0, global alignment dominates and DAAN degenerates to DANN; when ω approaches 1, local class-wise alignment dominates and DAAN degenerates to MADA.The paper associates these limits with different relative discrepancies between global and local distributions.
  • Optimization: DAAN can be trained with stochastic gradient descent by optimizing the integrated objective directly or alternating between the two objectives.The gradients are computed through the learning procedure described for the combined objective.

F. Discussions

The discussion connects DAAN’s dynamic alignment to H-divergence and attention: it estimates global and local domain discrepancies while learning their relative importance. Compared with fixed global, local, and kernel-based approaches, DAAN performs adaptive adversarial alignment without extra classifiers.

  • Theoretical analysis: DAAN’s risk analysis treats its dynamic distribution adaptation as a dynamic version of H-divergence over global and local domain divergences.The divergences can be approximately measured by the A-distances used in the global and local discriminators.
  • Attention interpretation: DAAN can be interpreted as an attention mechanism that learns the importance of marginal and conditional distributions with neural networks.This interpretation frames the dynamic factor as selecting which distributional alignment deserves more attention.
  • Comparison: Unlike global-only or local-only adversarial methods, DAAN quantitatively calculates the relative importance of global and local distributions through ω.Its dynamic alignment is designed to adapt to the differing roles of marginal and conditional distributions.
  • Comparison: Compared with MEDA, DAAN fine-tunes ω using deep adversarial representations without training extra classifiers.The paper describes this design as more accurate, easy, and efficient.

IV. EXPERIMENTS

The experiments evaluate DAAN for unsupervised domain adaptation on ImageCLEF-DA and Office-Home against several deep and adversarial baselines. DAAN outperforms the comparison methods on most transfer tasks in both datasets.

  • Datasets: DAAN is evaluated on ImageCLEF-DA and Office-Home for unsupervised domain adaptation.The experiments use all domain combinations, producing 6 ImageCLEF-DA tasks and 12 Office-Home tasks.
  • Datasets: ImageCLEF-DA contains 12 shared categories across Caltech-256, ImageNet ILSVRC 2012, and Pascal VOC 2012, with 50 images per category in each domain.The six transfer tasks are I→P, P→I, I→C, C→I, P→C, and C→P.
  • Datasets: Office-Home contains 15,588 images across Artistic, Clip Art, Product, and Real-World domains with 65 object categories.The study constructs 12 transfer tasks from all domain combinations.
  • Results: DAAN outperforms all comparison methods on most Office-Home tasks.The baselines include CAN, MEDA, deep residual learning, DDC, DAN, RTN, DANN, D-CORAL, JAN, and MADA.
  • Results: DAAN outperforms all comparison methods on most ImageCLEF-DA transfer tasks and also surpasses DANN and MADA.The reported ImageCLEF-DA comparisons use ResNet-based classification accuracy.

C. Implementation Details

The experiments use standardized deep-learning initialization and optimization settings, and report DAAN’s performance against comparison methods across benchmark tasks. The results indicate broad superiority over competing approaches.

  • All deep methods are implemented in PyTorch and fine-tuned from ImageNet-pretrained ResNet-50 models.
  • The classifier is trained from scratch with a learning rate 10 times higher than that of the other layers.Mini-batch SGD with momentum 0.9 is used for optimization.
  • DAAN outperforms all comparison methods on most ImageCLEF-DA transfer tasks, including DANN and MADA.The comparison is based on classification accuracy using ResNet features.
  • DAAN also outperforms all comparison methods on most Office-Home tasks.
  • The study compares DAAN with adversarial and non-adversarial methods and reports better performance than recent methods such as MEDA and CAN.

E. Analysis of the Importance of the Dynamic Adversarial Factor ω

The analysis tests whether adversarial adaptation should weight marginal and conditional distributions differently and whether DAAN can estimate that weighting effectively. Results show that the optimal factor varies across tasks and that dynamic estimation is useful.

  • Classification accuracy varies with ω across Office-Home and ImageCLEF-DA tasks, supporting consideration of different marginal and conditional distribution effects.
  • The optimal ω varies across tasks and may have several optimal values even within the same task.The paper attributes this possibility to different feature representations learned when calculating ω.
  • DAAN’s ω calculation is compared with Random guessing, Average search, and MEDA, while grid search supplies an optimal-reference result.
  • The reported comparison concludes that DAAN’s evaluation of ω is more effective and efficient.

F. Effectiveness Analysis

DAAN is evaluated through ablation, feature visualization, and convergence analysis. The paper reports dynamic alignment benefits, rapid stabilization, and applicability to classification-related transfer-learning tasks.

  • Ablation study: Ablation results indicate that aligning only marginal or conditional distributions, or weighting them equally, is insufficient for the evaluated datasets.DAAN dynamically aligns the two distributions and achieves better performance in the comparison.
  • Feature visualization: The t-SNE analysis compares JAN and DAAN activations for the 12-class task P → I, with red source samples and blue target samples.
  • Convergence analysis: The convergence analysis tracks ω and loss across iterations for DAAN and MEDA on the same DAAN architecture.
  • Convergence analysis: DAAN reaches quick, steady convergence after 20 epochs.
  • Convergence analysis: The dynamic adversarial factor ω stabilizes after several iterations, whereas MEDA’s adaptive factor takes more iterations.
  • Conclusions and future work: The conclusion presents DAAN as an adversarial transfer-learning method that quantitatively evaluates marginal and conditional distribution importance while learning domain-invariant representations.
  • Conclusions and future work: The authors state that DAAN can be applied to classification-related applications including object detection, image segmentation, and visual tracking.
Loading 1909.08184v1…