Source-linked AI summary
Semi-supervised Domain Adaptation via Minimax Entropy
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, Kate Saenko
TL;DR
In SSDA, existing adaptation methods can fail when only a few target labels are available. MME adversarially adapts prototype-based classifiers and achieves state-of-the-art performance across reported benchmarks.
Problem
Existing unsupervised domain adaptation methods can fail to learn discriminative target boundaries and may underperform simple source-target training when few target labels are available.
Method
MME alternately maximizes unlabeled-target conditional entropy with respect to a prototype-based classifier and minimizes it with respect to the feature encoder.
Results
MME sets a new SSDA state of the art, reducing error by 8.5% versus few-shot baselines, 8.8% versus alignment methods, and 11.3% versus joint source-target training in one scenario.
Takeaways & Limitations
MME outperformed reported baselines across DomainNet adaptation scenarios and most network settings, as well as on Office-Home and Office.
Takeaways & Limitations
The approach assumes a single domain-invariant prototype for each class, with source labels dominating the initial prototype estimates.
Abstract
from arXiv · showhide
Contemporary domain adaptation methods are very effective at aligning feature distributions of source and target domains without any target supervision. However, we show that these techniques perform poorly when even a few labeled examples are available in the target. To address this semi-supervised domain adaptation (SSDA) setting, we propose a novel Minimax Entropy (MME) approach that adversarially optimizes an adaptive few-shot model. Our base model consists of a feature encoding network, followed by a classification layer that computes the features' similarity to estimated prototypes (representatives of each class). Adaptation is achieved by alternately maximizing the conditional entropy of unlabeled target data with respect to the classifier and minimizing it with respect to the feature encoder. We empirically demonstrate the superiority of our method over many baselines, including conventional feature alignment and few-shot methods, setting a new state of the art for SSDA.
1. Introduction
Semi-supervised domain adaptation methods can underperform when a few target labels are available, motivating Minimax Entropy, which adapts prototype-based classifiers through adversarial entropy optimization.
- UDA methods align source and target distributions but can fail to learn discriminative target-domain class boundaries when limited target labels are available.
- MME is an adversarial SSDA method that optimizes conditional entropy on unlabeled target data while also using the task loss.
- The model uses cosine-similarity classification, with class-specific weight vectors serving as estimated prototypes for each class.
- MME estimates domain-invariant prototypes by moving classifier weight vectors toward unlabeled target features through entropy maximization.
- 8.5% error reduction versus baseline few-shot methods, 8.8% versus alignment methods, and 11.3% versus joint source-target training were reported in one adaptation scenario.
- The paper identifies limitations of existing domain adaptation methods and reports superiority over existing methods on benchmark datasets.
2. Related Work
Related work spans unsupervised domain adaptation, semi-supervised learning, and few-shot learning, while MME targets domain shift with limited target supervision.
- UDA methods align source and target feature distributions, often using domain classifiers, but target-domain class boundaries can remain ambiguous.
- Semi-supervised learning methods improve learning with unlabeled data but generally do not address domain shift.
- Conditional entropy minimization can fail under large domain gaps, whereas MME is presented as an entropy-minimization variant for domain adaptation.
- Few-shot learning uses a few labeled examples without unlabeled examples and targets novel classes, whereas SSDA adapts to the same classes in a new domain.
3. Minimax Entropy Domain Adaptation
The method adapts a classifier with estimated class prototypes by adversarially maximizing target entropy for the classifier and minimizing it for the feature extractor, alongside supervised classification. This minimax process is connected to reducing domain divergence without an explicit domain classifier.
- Model architecture: The base model combines a feature extractor with a normalized similarity-based classifier whose weight vectors act as estimated class prototypes.The classifier uses class-specific weight vectors and a temperature parameter to produce probabilistic predictions from normalized features.
- Method motivation: The SSDA objective uses labeled source and target data for classification while exploiting unlabeled target examples for adaptation.The classification loss alone does not learn discriminative features for the entire target domain, motivating minimax entropy training.
- Minimax entropy training: The classifier maximizes entropy on unlabeled target data to estimate domain-invariant prototypes, while the feature extractor minimizes entropy to cluster target features around them.Repeating prototype estimation and feature clustering produces discriminative target representations.
- Minimax entropy training: Both classifier and feature extractor retain supervised classification training while adversarially optimizing the conditional entropy of unlabeled target predictions.A gradient reversal layer implements the opposing entropy gradients in one forward and back-propagation pass, with λ controlling the trade-off.
- Theoretical connection: The theoretical formulation bounds domain divergence through the proportion of target examples whose entropy exceeds γ and motivates maximizing target entropy with respect to the classifier.The paper reports that experiments observe reduced domain divergence.
- Theoretical connection: The method can be interpreted as minimizing domain divergence through entropy-based minimax training, despite using neither a domain classifier nor a domain-classification loss.The associated domain classifier assigns domain labels according to whether prediction entropy exceeds threshold γ.
4. Experiments
Experiments evaluate MME across DomainNet, Office-Home, and Office under supervised and unsupervised adaptation settings. MME generally outperforms competing baselines, while analyses associate its gains with prototype-based discriminative features and domain alignment.
- Experimental setup: Experiments use one-shot and three-shot target supervision across DomainNet, plus averaged adaptation scenarios on Office-Home and Office.DomainNet experiments cover four domains; target labels and unlabeled examples are used according to the SSDA setup.
- Main results: MME outperformed other baselines in nearly all DomainNet adaptation scenarios and across three networks, with one exception matching ENT.On average, MME exceeded S+T by 9.5% in ResNet one-shot and 8.9% in ResNet three-shot settings.
- Main results: MME also outperformed baselines on Office-Home and Office when performance was averaged across adaptation scenarios.The reported tables summarize accuracy across the respective datasets’ adaptation scenarios.
- Comparison with entropy minimization: Entropy minimization can help ResNet, but MME performs much better when domain gaps are large, including C→S, S→P, and R→S.The passage attributes ResNet’s stronger entropy-minimization results partly to discriminative pretrained features and batch normalization’s alignment effect.
- Varying labeled examples: With a few labeled target examples, MME works much better than S+T, while ENT requires 5 labeled examples per class to improve performance.As labeled examples increase, ENT narrows the performance gap because prototype estimation becomes more accurate without adaptation.
- Analysis: Feature analyses indicate that MME produces more discriminative target features, lower entropy than baselines except ENT, and reduced domain divergence relative to S+T.The visualizations compare class prototypes, target features, source-target alignment, covariance eigenvalues, entropy, and domain divergence.
5. Conclusion
The paper concludes that MME adversarially adapts a prototype-based few-shot model for SSDA. It reports superiority over conventional feature-alignment and few-shot baselines, establishing a new SSDA state of the art.
- Conclusion: MME adversarially optimizes a prototype-based few-shot model for semi-supervised domain adaptation.Its model combines a feature encoder with a classifier measuring similarity to estimated class prototypes.
- Conclusion: MME alternates maximizing unlabeled-target conditional entropy for the classifier with minimizing it for the feature encoder.This minimax procedure is the paper’s adaptation mechanism.
- Conclusion: MME outperforms conventional feature-alignment and few-shot methods, setting a new state of the art for SSDA.The conclusion describes this superiority as an empirical finding across the evaluated baselines.
1. Datasets
The supplemental material provides examples of the datasets used in the experiments and lists the DomainNet classes used.
- Datasets: Examples of the experimental datasets are provided in Figure 7.The passage refers readers to the supplemental material for those dataset examples.
- Datasets: The supplemental material includes a list of classes used in the DomainNet experiments.The class list accompanies the dataset examples.
2. Implementation Detail
The supplemental material describes implementation choices for MME and baseline methods, including optimization, architecture, and training-data usage. It also reports repeated-run and split-based evaluation details.
- MME implementation: The implementation uses learning-rate annealing and reports main-paper performance from one-time training.Additional material reports averages and variances across multiple runs and different dataset splits.
- MME implementation: MME implementation replaces or augments backbone classification layers and uses momentum optimization with distinct learning rates for fully connected and other layers.The reported learning rates are 0.01 for fully connected layers and 0.001 for other layers in the described settings.
- Baselines: S+T trains on labeled source and target examples with cross-entropy, whereas ENT minimizes unlabeled-example entropy across the entire network.The comparison distinguishes methods that use labeled data only from standard entropy minimization.
- Baselines: DANN and CDAN train domain classifiers to distinguish source examples from unlabeled target examples, while ADR uses sensitivity loss with opposing optimization objectives for C and F.CDAN uses its official implementation with modifications for labeled target examples.
3. Additional Results Analysis
Additional analyses show that the method outperforms baselines across benchmark settings, remains stable across runs and splits, and maintains performance under different training choices. Its advantage is linked to domain-gap reduction rather than entropy minimization alone.
- Our method outperformed baselines in almost all Office-Home and Office settings, including all Office settings.
- λ=0.1 produced the best validation result and was used in all experiments as the trade-off between classification loss and unlabeled-target entropy.
- During training, our method gradually increased performance, whereas other methods quickly converged.
- VAT showed no effectiveness in SSDA, while entropy minimization depended strongly on batch-normalization choice and performed worse with Joint BN.The proposed method remained effective with Joint BN, consistent with its design to reduce the domain gap.
- The method consistently performed better than other methods across different labeled-target training splits and remained stable over three runs.