Source-linked AI summary
Domain Impression: A Source Data Free Domain Adaptation Method
Vinod K Kurmi, Venkatesh K Subramanian, Vinay P Namboodiri
TL;DR
Domain adaptation usually requires source samples, but storage, privacy, and sharing constraints can make them unavailable. This paper instead uses a source-trained classifier to generate proxy samples through a generative framework and adapts the classifier with target data; it reports comparability with full-source baselines and applicability to existing adaptation methods.
Problem
The problem is source-data-free domain adaptation when source samples cannot be accessed and target data are unlabeled, despite a source-trained classifier being available.
Method
The method uses generation and adaptation modules: a classifier-guided GAN learns source joint-distribution proxy samples, which support target-domain classifier adaptation.
Results
The method is reported as comparable to baselines that use full source-sample information while not using source samples during adaptation.
Takeaways & Limitations
The framework is generic and can be applied with existing domain adaptation models, using generated domain impressions instead of source samples.
Takeaways & Limitations
The formulation assumes that the source dataset is unavailable during adaptation and only an unlabeled target dataset is provided at training time.
Abstract
from arXiv · showhide
Unsupervised Domain adaptation methods solve the adaptation problem for an unlabeled target set, assuming that the source dataset is available with all labels. However, the availability of actual source samples is not always possible in practical cases. It could be due to memory constraints, privacy concerns, and challenges in sharing data. This practical scenario creates a bottleneck in the domain adaptation problem. This paper addresses this challenging scenario by proposing a domain adaptation technique that does not need any source data. Instead of the source data, we are only provided with a classifier that is trained on the source data. Our proposed approach is based on a generative framework, where the trained classifier is used for generating samples from the source classes. We learn the joint distribution of data by using the energy-based modeling of the trained classifier. At the same time, a new classifier is also adapted for the target domain. We perform various ablation analysis under different experimental setups and demonstrate that the proposed approach achieves better results than the baseline models in this extremely novel scenario.
1. Introduction
The paper addresses domain adaptation when source samples cannot be accessed by using a trained source classifier to generate proxy samples, then adapting the classifier with target data. The framework is presented as generic and comparable to source-data baselines without using source samples.
- Dataset bias causes models trained on one dataset to perform poorly on other datasets, motivating domain adaptation.
- Source samples may be unavailable because of storage requirements, sharing challenges, privacy concerns, and dataset-handling issues.The paper identifies ImageNet’s nearly 14 million images and hundreds of gigabytes of storage as an example.
- The proposed method uses a source-trained classifier as an energy-based model and a GAN to learn the source joint distribution and generate labeled proxy samples.The energy is defined using LogSumExp values from the classifier, and generated samples replace source data during adaptation.
- The framework adapts the classifier for the target domain using generated dummy labeled samples together with target samples, without using source samples.
- The generated domain impression can be applied to existing domain adaptation methods, making the framework generic across adaptation algorithms.
- Comparisons and ablation analyses report that the method is comparable to baselines using full source-sample information while omitting source samples.
2. Literature Survey
Prior work addresses domain adaptation through reconstruction, adversarial learning, privacy preservation, and source-data-free strategies. The paper distinguishes its setting by considering unavailable source samples together with unavailable target labels.
- Domain adaptation research includes reconstruction methods that jointly address classification and generation.
- Adversarial methods use GANs and related discriminator-based approaches, while some methods address mode collapse through multiple discriminators.
- Privacy-oriented approaches transform data into privacy-preserving domains or combine multiple private sources through federated transfer learning.
- Some source-data-free methods assume access to target labels, whereas the paper considers source samples unavailable and target labels unavailable.
- Other source-data-free approaches generate target-style data or align target and source representations without accessing source data.
3. Background: Generative model from Discriminative model
The background separates discriminative modeling of class-conditional probabilities from generative modeling of the joint data-label distribution. It then expresses the joint likelihood through the classifier and an energy-based model.
- Discriminative models focus on estimating the class-conditional distribution p(y|x) and classification boundaries.Here x denotes the input and y denotes the label.
- Generative models learn the joint distribution p(x, y) from the data-generation process.
- The joint log likelihood is decomposed into log pθ(x) and log pθ(y|x) using Bayes’ theorem.
- The classifier supplies pθ(y|x) through cross-entropy, while log pθ(x) is represented with an energy-based model.θ denotes the model parameters.
- The energy function maps an input x to a scalar and is defined as the LogSumExp of the trained classifier’s logits.
4. Source Data Free Adaptation
The method separates source-data-free adaptation into generation of proxy source samples and adaptation of a classifier to unlabeled target data. It combines classifier-guided conditional generation with adversarial and domain-discriminative losses while keeping the pre-trained classifier fixed.
- Framework structure: The framework divides source-data-free adaptation into a Generation module for obtaining samples and an Adaptation module for adapting the classifier to the target domain.The two modules are presented as the core structure of the approach.
- Problem setting: The method assumes the source dataset is unavailable and uses only a source-trained classifier together with unlabeled target data during adaptation.The source classifier is retained while the original source samples are not accessible.
- Optimization design: In the framework, the generator, GAN discriminator, feature extractor, classifier, and domain discriminator are trainable, while the pre-trained classifier remains frozen.The latent noise vector supplies the generator, and the pre-trained classifier provides class supervision for generated samples.
- Generation module: The generation module learns the source joint distribution p(x, y) with a conditional GAN guided by the pre-trained classifier and class labels.Conditional generation is used because an unconditional GAN does not guarantee samples from the desired class.
- Generation module: The generator is optimized using likelihood, adversarial, and cross-entropy objectives so generated samples are class-consistent and more natural.The adversarial discriminator uses target-domain samples, while cross-entropy updates the generator without updating the pre-trained classifier.
- Adaptation module: The adaptation module uses generated proxy samples and target samples with domain-discriminative training to learn domain-invariant features through a gradient reversal layer.The domain discriminator receives the loss gradient, whereas the feature extractor receives its negative gradient; the adaptive classifier is trained on generated samples.
5. Results and Discussion
The evaluation covers digit and Office-31 adaptation tasks, comparing source-data-free variants with baselines and methods that access source data. Results report competitive performance, improvements over no adaptation, and sensitivity to generated-sample counts and loss functions.
- Datasets: The experiments evaluate MNIST→MNIST-M, SVHN→MNIST, MNIST→SVHN, MNIST→USPS, and all six Office-31 adaptation tasks.Office-31 contains Amazon, Webcam, and DSLR domains with 31 classes each.
- Digit adaptation: Table 1 compares classification accuracy for source-data-free models against a no-adaptation baseline and state-of-the-art methods using source data.SDDA-P initializes the classifier with pretrained weights, while SDDA initializes it randomly.
- Digit adaptation: The proposed model performs comparably to baselines using full source information while improving target performance over baselines, including approximately 25% on MNIST→MNIST-M.The paper also reports improvements with a large margin on other adaptation tasks.
- Ablations: In the SDDA-G variant, performance initially improves as generated-sample counts increase, then slightly deteriorates at larger counts.The variant first learns the generative model and then uses generated samples to train the adaptation module.
- Office-31 adaptation: Office-31 results report approximately 3% and 1.5% improvements over the no-adaptation baseline on A→W and A→D without source-data access.The reported results cover all six Office-31 adaptation tasks.
6. Analysis
The analysis evaluates the proposed source-data-free method through loss, domain-discrepancy, generation, sample-count, density, and adaptation-parameter studies. Results indicate class-discriminative, diverse generated samples, closer target-domain alignment after adaptation, and compatibility with multiple adaptation frameworks.
- Loss-function ablation: The generative adversarial loss is necessary for convergence, while adding likelihood-based loss improves MNIST→MNIST-M adaptation.Without adversarial loss, the generator cannot be trained because there is no adversarial discriminator.
- Adaptation frameworks: The proposed method can be plugged into MMD, IDDA, Wasserstein DA, and GRL frameworks for MNIST→MNIST-M adaptation.The analysis uses DCGAN architectures for both generator and classifier.
- Distribution discrepancy: Source and generated domains remain closer, while the target domain becomes closer after adaptation than before adaptation.The comparison uses proxy A-distance between source-target, source-generated, and target-generated domains.
- Generated-sample count: 6000 generated samples produce the best reported MNIST→MNIST-M performance in the separate-generation experiment.This experiment trains generation and adaptation modules separately and also evaluates MNIST→SVHN.
- Image generation visualization: Generated images are class-discriminative and diverse, supporting class structure and avoiding mode collapse during source-domain impression generation.The target samples serve as real data for GAN training, while cross-entropy from the pretrained classifier supplies class structure.
- Adaptation parameter: The model is not very sensitive to the adaptation parameter λ, with better performance reported at λ = 1 for SDDA.This ablation concerns the MNIST→MNIST-M adaptation task.
- Generated-sample density: Generated samples have similar density around source and target data with non-adapted features, but slightly higher target density with adapted features.Density is estimated within an ε-neighborhood using convolutional-layer features.
7. Conclusion
The paper proposes a source data-free adaptation method to address the availability of source data, and presents it as a generic approach applicable with existing domain adaptation models. The authors characterize it as a novel direction for domain adaptation without source-data availability.
- Contribution: The proposed method addresses the source-data availability challenge in domain adaptation without requiring access to source datasets.The conclusion identifies source-data availability as a critical challenge for existing domain adaptation techniques.
- Scope: The approach is generic and can be applied with existing domain adaptation models.The authors present the method as a source data-free adaptation framework applicable across models.
- Conclusion: The work opens an avenue for further research on domain adaptation without source data.This is the conclusion's stated interpretation of the reported results.