PaperlayerView exact revision on arXiv

Source-linked AI summary

Transferable Semantic Augmentation for Domain Adaptation

Shuang Li, Mixue Xie, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Wei Li

arXiv:2103.12562v1cs.CV

TL;DR

Domain adaptation transfers knowledge from labeled source data to unlabeled target data, while many methods assume a shared classifier. TSA implicitly augments source features toward target semantics to adapt the classifier. Across several cross-domain benchmarks, it consistently improves performance.

  • Problem

    Many domain-adaptation methods assume a shared classifier, while relatively little attention has been paid to adapting the source-supervised classifier.

    Source: §1 · p. 1

  • Method

    TSA uses class-wise semantic directions based on inter-domain feature mean differences and target intra-class covariance, implementing source augmentation through an expected transferable cross-entropy loss.

    Source: §1 · p. 2

  • Results

    TSA consistently improves performance across several cross-domain benchmarks; on Office-Home, BSP gains 4.9% improvement and achieves 71.2% average accuracy.

    Source: §1 · p. 2, §4.3 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    TSA adds no extra network modules, has negligible computational cost, and can be plugged into various domain-adaptation methods.

    Source: §1 · p. 2, §5 · p. 8

    Source and target data come from different distributions, so their corresponding ideal classifiers may differ.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Domain adaptation has been widely explored by transferring the knowledge from a label-rich source domain to a related but unlabeled target domain. Most existing domain adaptation algorithms attend to adapting feature representations across two domains with the guidance of a shared source-supervised classifier. However, such classifier limits the generalization ability towards unlabeled target recognition. To remedy this, we propose a Transferable Semantic Augmentation (TSA) approach to enhance the classifier adaptation ability through implicitly generating source features towards target semantics. Specifically, TSA is inspired by the fact that deep feature transformation towards a certain direction can be represented as meaningful semantic altering in the original input space. Thus, source features can be augmented to effectively equip with target semantics to train a more transferable classifier. To achieve this, for each class, we first use the inter-domain feature mean difference and target intra-class feature covariance to construct a multivariate normal distribution. Then we augment source features with random directions sampled from the distribution class-wisely. Interestingly, such source augmentation is implicitly implemented through an expected transferable cross-entropy loss over the augmented source distribution, where an upper bound of the expected loss is derived and minimized, introducing negligible computational overhead. As a light-weight and general technique, TSA can be easily plugged into various domain adaptation methods, bringing remarkable improvements. Comprehensive experiments on cross-domain benchmarks validate the efficacy of TSA.

1. Introduction

Domain adaptation transfers knowledge from a labeled source domain to an unlabeled target, but many methods assume a shared classifier and rarely address classifier adaptation. TSA implicitly augments source features toward target semantics and applies that augmentation through an efficient transferable loss.

1. Introduction

  • TSA addresses the limited attention to adapting source-supervised classifiers, while prior classifier-adaptation approaches may require complex architectures with high computational overhead.Most existing methods assume a shared classifier; cited alternatives include classifier residual learning and SymNets.Source: §1 · pp. 1–2
  • TSA samples class-wise semantic directions using inter-domain feature mean differences and target intra-class covariance to guide source features toward target semantics.A memory module estimates class-wise means and covariance from pseudo-labeled target samples.Source: §1 · p. 2
  • Minimizing an upper bound on expected transferable cross-entropy implements source augmentation without explicitly generating augmented features, with negligible extra computational overhead.The loss is defined over the augmented source distribution.Source: §1 · p. 2
  • TSA introduces no extra network modules over the backbone, making it simple to implement and computationally efficient.Source: §1 · p. 2
  • Extensive experiments on several cross-domain benchmarks, including Office-Home, Office-31, VisDA-2017, and digits, demonstrate that TSA can consistently yield significant performance improvements.TSA is described as a lightweight, general technique that can be plugged into various DA methods to significantly boost their performances.Source: §1 · p. 2

2. Related Work

Feature-adaptation methods mainly align domain representations through discrepancy minimization or adversarial learning. Classifier adaptation instead addresses the restrictive shared-classifier assumption; TSA augments source features toward target semantics without auxiliary modules or explicit sample generation.

2.1. Feature Adaptation

  • Feature-adaptation work mainly minimizes statistical discrepancies or uses adversarial learning to learn domain-invariant representations.Examples include MMD-based alignment and adversarial feature learning.Source: §2.1 · p. 2
  • Unlike domain-invariant feature learning, TSA uses augmented source features to improve the generalization ability of the source classifier.Source: §2.1 · p. 2

2.2. Classifier Adaptation

  • Classifier adaptation spans target-style sample generation and non-generative strategies such as residual classifier perturbation or multi-classifier alignment.The described methods include image translation, augmented features, residual learning, and task-specific classifiers.Source: §2.2 · pp. 2–3
  • TSA implicitly generates target-semantic source features by minimizing a robust transferable cross-entropy loss, without auxiliary network modules or explicit sample generation.The described semantics include object shapes, colors, and backgrounds.Source: §2.2 · p. 3

3. Transferable Semantic Augmentation

TSA addresses classifier adaptation by augmenting labeled source features toward target semantics in deep feature space. It uses class-wise target guidance to construct transformation directions and learns through an expected loss rather than explicitly generating many augmented features.

3.1. Motivation and Preliminaries

  • TSA frames classifier adaptation as guiding source features toward target styles such as shape, color, visual angle, or background.The method uses deep-feature transformations as a way to represent semantic changes in the input space.Source: §3.1 · p. 3

3.2.1 Class-wise Overall Semantic Guidance.

  • Class-wise inter-domain feature mean differences provide overall semantic guidance for aligning augmented source features with target features.Target means are estimated using pseudo-labels to address the target domain's lack of labels.Source: §3.2.1 · p. 4

3.2.2 Semantic Transformation Direction Learning.

  • TSA samples class-wise transformation directions from a multivariate normal distribution using the mean difference and target intra-class covariance.The mean difference addresses overall semantic bias, while target covariance captures within-class semantic variation.Source: §3.2.2 · p. 4

3.2.3 Memory Module.

  • A memory module caches recent features and replaces older entries each batch, avoiding the out-of-date feature estimates possible with iterative accumulation.The paper describes the replacement cost as negligible memory overhead.Source: §3.2.3 · p. 4

3.2.4 Sampling Strategy.

  • The augmentation strength λ increases from 0 to λ0 over training, reducing reliance on less accurate target mean and covariance estimates early on.The schedule is λ = (t/T) × λ0, with t the current and T the maximum iteration.Source: §3.2.4 · p. 4

3.3. Transferable Cross-Entropy Loss Learning

  • Rather than explicitly generating many augmented features, TSA derives an upper-bound surrogate for expected transferable cross-entropy and minimizes it to adapt the source classifier.The derivation uses Jensen’s inequality; the resulting loss can serve as a plug-in for other domain adaptation methods.Source: §3.3 · pp. 4–5
  • TSA differs from ISDA by targeting classifier adaptation under unlabeled target data and domain shift, and its memory module estimates mean and covariance more accurately in experiments.The paper also reports that directly applying ISDA in the domain-adaptation setting performs worse.Source: §3.3 · p. 5

3.4. Overall Formulation

The overall formulation uses target-data mutual information alongside TSA’s adaptation objective. The paper then relates TSA to a domain-adaptation bound on target generalization error.

3.4. Overall Formulation

  • Mutual information maximization on target data encourages stronger correlations between target features and predictions, which the paper says benefits semantic augmentation.The stated rationale is that extracted features then contain more prediction-relevant semantics while ignoring trivial semantics.Source: §3.4 · p. 5
  • Because the target domain is unlabeled, the average target predictions approximate its ground-truth distribution.Source: §3.4 · p. 5
  • The overall TSA objective includes a trade-off parameter β, whose component effects are analyzed in the ablation study.Source: §3.4 · p. 5

3.5. Theoretical Insight

  • The theoretical discussion introduces a domain-adaptation upper bound on target generalization error, ϵt.The bound is stated for a classifier h in hypothesis space H.Source: §3.5 · p. 5
  • The bound’s terms include source generalization error, the H∆H-distance between domains, and the error of an ideal joint hypothesis on source and target.Source: §3.5 · p. 5
  • The paper argues that TSA’s augmented source features can fill the domain gap, further bound domain distance, and support joint minimization of the ideal hypothesis’s source and target errors.The authors conclude that TSA therefore accords with the domain-adaptation theory and enhances transferability.Source: §3.5 · p. 5

4. Experiment

Experiments report gains from TSA on Office-Home and Office-31, alongside analyses of semantic augmentation, classifier boundaries, and estimation behavior.

4.1. Datasets

  • The evaluation covers Office-Home, Office-31, VisDA-2017, and four digit transfers, using benchmark-specific source and target domains.Source: §4.1 · pp. 5–6

4.3. Results

  • On Office-Home, BSP+TSA improves accuracy by 4.9% to 71.2% on average, and TSA consistently improves all four evaluated methods.Source: Table 1 · p. 6, §4.3 · p. 6

4.2. Implementation Details

  • Experiments use ImageNet-pretrained ResNets for Office-Home, Office-31, and VisDA-2017, and train digit-task networks from scratch; results average three trials.Source: §4.2 · p. 6

4.3. Results

  • On Office-31, TSA improves DANN by 7.2% and surpasses both feature-adaptation and recent classifier-adaptation methods.Source: Table 2 · p. 6, §4.3 · p. 6

4.4. Analysis

  • For SVHN-to-MNIST, augmented images alter backgrounds, colors, and number shapes, with some resembling target images.Source: Figure 3 · p. 7, §4.4 · p. 8
  • On Office-31, the full sampling strategy performs best; mean difference alone misses target semantic variation, and adding the mutual-information loss is reported as more effective.Source: Table 5 · p. 7, §4.4 · pp. 7–8
  • In the inter-twinning-moons experiment, TSA shifts classification boundaries toward target data, whereas source-only and DANN misclassify many target samples.Source: Figure 4 · p. 8, §4.4 · p. 8
  • The memory-module estimates better approximate ideal mean and covariance estimates, while TSA remains competitive across a wide range of λ0 and β values.Source: Figure 5 · p. 8, §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

TSA improves classifier adaptation by optimizing a transferable loss over implicitly augmented source features, with negligible computational cost; experiments show efficacy across cross-domain datasets.

  • TSA optimizes a transferable loss over implicitly augmented source features, with negligible computational costs, and experiments demonstrate efficacy and versatility across cross-domain datasets.Source: §5 · p. 8

A. Training of TSA

TSA is trained end-to-end by minimizing an overall objective that combines its transferable loss with mutual-information maximization.

A. Training of TSA

B. Memory Cost and Performance Gain

On Office-31, the memory-based estimator improves performance over the ResNet-50 baseline with little additional GPU memory, and outperforms iterative estimation. Its memory module replaces older batches, addressing bias from accumulated outdated features.

B. Memory Cost and Performance Gain

C. Reverse Mapping Algorithm

The reverse mapping algorithm visualizes augmented deep features as images by searching the pixel space with a generator and feature extractor. The reported visualizations show diverse, meaningful, and target-specific augmented features.

C. Reverse Mapping Algorithm

  • The reverse mapping algorithm searches pixel space for images corresponding to augmented features in deep feature space.Source: C. Reverse Mapping Algorithm · p. 12
  • A source-pretrained generator produces fake images, while a task-pretrained feature extractor maps fake and source images to deep features; both modules are frozen during the search.For SVHN→MNIST, the generator is pretrained on SVHN and the extractor on the adaptation task.Source: C. Reverse Mapping Algorithm · p. 12
  • The algorithm first finds noise corresponding to a source image by matching its generated-image features to source-image features, using mean square error at feature and pixel levels to initialize the search.Source: C. Reverse Mapping Algorithm · p. 12
  • After TSA augments the source feature, the search optimizes noise for that augmented feature, and the resulting generated image visualizes it.Source: C. Reverse Mapping Algorithm · p. 12
  • The reverse-mapping noise variable has dimension 128; the generator uses WGAN-GP structure and optimization uses SGD with momentum 0.9.Source: C. Reverse Mapping Algorithm · p. 12
  • The reported visualizations show diverse, meaningful, and even target-specific augmented features.Source: C. Reverse Mapping Algorithm · p. 12

D. Necessity Stress Test of Target Data

On Office-31, accuracy rises as more target data is used for training, while using 80% and 100% yields comparable performance. The authors suggest using less target data to save computation and memory costs.

D. Necessity Stress Test of Target Data

  • The stress test varies the proportion of target data used for training from 0% to 100% and evaluates with the entire target dataset.For each class, a random proportion of target data is sampled alongside the original source dataset.Source: D. Necessity Stress Test of Target Data · p. 12
  • 80% and 100% target-data training yield comparable performance on Office-31, motivating reduced target-data use to save computation and memory costs.Accuracy increases as the proportion of target training data grows.Source: D. Necessity Stress Test of Target Data · p. 13, Table 7 · p. 13

E. Visualization of Features

On A→W, TSA is associated with more domain-indistinguishable and discriminative learned features for DANN and CDAN. The visualizations compare source and target features across the four methods.

E. Visualization of Features

  • With TSA, DANN and CDAN representations are more indistinguishable across domains and have sharper class boundaries.The authors describe this as evidence that TSA facilitates more transferable and discriminative features alongside classifier adaptation.Source: E. Visualization of Features · p. 13, Figure 9 · p. 13
  • Without TSA, DANN and CDAN do not perfectly align the domains because several classes mismatch.The A→W (Office-31) t-SNE visualizations show source features in blue and target features in red.Source: E. Visualization of Features · p. 13, Figure 9 · p. 13

Related papers

Loading 2103.12562v1…