Source-linked AI summary
Adversarial Dropout Regularization
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, Kate Saenko
TL;DR
The paper addresses the problem that domain critics can align target features without respecting class boundaries, producing ambiguous representations. It proposes Adversarial Dropout Regularization, which uses dropout on the classifier to detect non-discriminative features and trains the generator to avoid them. ADR improves unsupervised domain adaptation across classification and segmentation experiments and also works for GAN-based semi-supervised learning.
Problem
Existing adversarial domain adaptation critics distinguish source from target features without considering class boundaries, which can produce ambiguous features near those boundaries.
Method
ADR uses dropout to turn the classifier into a critic sensitive to non-discriminative features, guiding the generator to align target features away from decision boundaries.
Results
ADR outperformed baseline methods, achieved state-of-the-art results on three datasets, and was effective for GAN-based semi-supervised learning.
Takeaways & Limitations
ADR provides a general alignment approach for classification and semantic segmentation that does not require domain labels and can extend to generative-model training.
Takeaways & Limitations
The method assumes access to labeled source images and corresponding labels alongside unlabeled target images.
Abstract
from arXiv · showhide
We present a method for transferring neural representations from label-rich source domains to unlabeled target domains. Recent adversarial methods proposed for this task learn to align features across domains by fooling a special domain critic network. However, a drawback of this approach is that the critic simply labels the generated features as in-domain or not, without considering the boundaries between classes. This can lead to ambiguous features being generated near class boundaries, reducing target classification accuracy. We propose a novel approach, Adversarial Dropout Regularization (ADR), to encourage the generator to output more discriminative features for the target domain. Our key idea is to replace the critic with one that detects non-discriminative features, using dropout on the classifier network. The generator then learns to avoid these areas of the feature space and thus creates better features. We apply our ADR approach to the problem of unsupervised domain adaptation for image classification and semantic segmentation tasks, and demonstrate significant improvement over the state of the art. We also show that our approach can be used to train Generative Adversarial Networks for semi-supervised learning.
1 INTRODUCTION
Unsupervised domain adaptation must transfer representations from labeled source domains to unlabeled targets despite distribution shifts. ADR addresses a limitation of adversarial alignment by using classifier decision-boundary information to generate more discriminative target features.
- Motivation: Unsupervised domain adaptation improves source models when target labels are unavailable and source and target input distributions differ.Simulated driving images provide labeled source data, but models trained on them may transfer poorly to real target images.
- Prior approach: Adversarial alignment methods use a feature encoder, classifier, and domain critic to align source and target features by fooling the critic.The critic predicts whether generated features come from the source or target domain.
- Limitation: Domain critics ignore category information, allowing generated features near class boundaries that are likely to be misclassified.The paper argues that successful alignment must account for class decision boundaries, although target labels are unavailable.
- Proposed method: ADR detects boundary-near points by perturbing the classifier and measuring changes in posterior class probabilities.Samples near a decision boundary are expected to show the largest posterior discrepancy.
- Proposed method: The resulting critic encourages the generator to avoid non-discriminative regions and align target features away from decision boundaries.Unlike global distribution matching, ADR uses boundary information to promote low-density separation.
- Contributions: ADR applies to classification, semantic segmentation, semi-supervised learning, and GAN training, with experiments reporting state-of-the-art results in difficult domain shifts.The approach is presented as general and straightforward to apply across these settings.
2 RELATED WORK
Prior work aligns domain feature distributions and uses low-density separation or entropy minimization to improve decisions on unlabeled samples. ADR instead uses dropout adversarially so the classifier detects noise-sensitive, non-discriminative features and guides feature generation.
- Domain Adaptation: Domain adaptation methods align source and target feature distributions, motivated by theory linking lower domain divergence to a lower upper bound on target error.Deep-learning methods commonly perform distribution matching in hidden network layers.
- Low-density Separation: Low-density separation places decision boundaries where unlabeled samples are sparse to obtain discriminative representations.ADR extends this goal to deep domain adaptation and is related to entropy minimization.
- Low-density Separation: ADR detects target samples sensitive to slight boundary movement rather than relying only on classifier-output entropy.Features robust to the movement may remain relatively closer to the boundary than source features.
- Low-density Separation: The entropy baseline trains the generator to minimize target entropy while the critic maximizes it, enabling direct comparison with ADR.The baseline was created by modifying a method proposed by Springenberg.
- Dropout: Standard dropout regularizes networks by randomly dropping units, while ADR uses dropout adversarially to make the classifier sensitive to noise and the generator produce noise-robust features.The paper distinguishes this use from conventional dropout regularization.
3 METHOD
ADR uses dropout to turn the classifier into a critic that detects target features near class boundaries, while adversarial training makes the generator avoid those regions. The method alternates source classification, critic sensitivity maximization, and generator sensitivity minimization.
- Classifier selection via dropout: The classifier C also acts as the critic by producing two dropout-based predictions for each target feature.Different dropout masks select classifiers C1 and C2, yielding posterior outputs p1 and p2.
- Classifier selection via dropout: ADR measures critic sensitivity with the symmetric KL divergence between the two posterior predictions.The critic maximizes this prediction difference because target samples near decision boundaries are more sensitive to classifier perturbations.
- Training procedure: C and G first minimize source classification loss so the classifier learns discriminative source features.This source objective is retained when C is trained as a critic.
- Training procedure: During critic training, C maximizes target sensitivity while retaining the source classification objective.The sensitivity term detects target samples near decision boundaries, and the classification term preserves discriminative source representations.
- Training procedure: During generator training, G minimizes target sensitivity to move target representations away from decision boundaries and also uses a conditional entropy term.The method repeats this generator update n times within each mini-batch training cycle.
- Insight and improvement: ADR encourages classifier neurons to learn different input characteristics, while minimizing sensitivity drives G toward features carrying categorical information.Non-categorical cues such as texture, contrast, or color differences can otherwise be exploited by C to increase sensitivity.
4 EXPERIMENTS
Experiments evaluate ADR on toy data, digit and object classification, semantic segmentation, and semi-supervised GAN training. Across these settings, ADR improves target-domain discrimination and performance relative to source-only, competing, or baseline methods.
- 4.2 UNSUPERVISED DOMAIN ADAPTATION FOR CLASSIFICATION: As sensitivity loss increases during digit adaptation, target accuracy improves while classifier C′ remains more accurate than critic C.The critic detects non-discriminative samples, while C′ is trained not to be sensitive to dropout noise.
- 4.2 UNSUPERVISED DOMAIN ADAPTATION FOR CLASSIFICATION: ADR outperforms existing distribution-matching methods and ENT when fine-tuning both networks on VisDA classification.The source-only embedding separates target classes poorly under domain shift, whereas ADR produces clearly improved separation.
- 4.2 UNSUPERVISED DOMAIN ADAPTATION FOR CLASSIFICATION: ADR improves mean IoU over source-only and competing models for GTA5-to-Cityscapes semantic segmentation.ADR produces clean segmentations, while ENT achieves a mean IoU of 15.0 and performs worse than the source-only model.
- 4.3 SEMI-SUPERVISED LEARNING USING GANS: ADR-generated SVHN images are mostly clear and diverse, and its critic outperforms other models on SVHN while matching state-of-the-art accuracy on CIFAR10.Some SVHN images are significantly blurred; CIFAR10 images are less realistic, though objects appear in most images.
5 CONCLUSION
The paper introduces Adversarial Dropout Regularization (ADR) to align feature distributions while generating discriminative target-domain features. ADR is general, requires no domain labels, outperforms baseline methods on three datasets, and also works for GAN-based semi-supervised learning.
- ADR uses a critic that detects samples near class boundaries and a feature generator that fools the critic.
- ADR aligns feature distributions without requiring domain labels and applies across multiple tasks.
- ADR outperformed baseline methods, including entropy minimization, and achieved state-of-the-art results on three datasets.
- ADR was also effective for training Generative Adversarial Networks in semi-supervised learning.
A ENTROPY BASED METHOD FOR DOMAIN ADAPTATION
The entropy-based baseline trains a classifier to move target samples toward or away from decision boundaries through opposing entropy objectives. It replaces ADR’s sensitivity term with classifier-output entropy and uses four generator updates per mini-batch.
- The critic maximizes target-sample entropy to move target samples near the decision boundary, while the generator minimizes entropy to move them away.
- This baseline replaces ADR’s sensitivity term d(p1, p2) with the entropy of the classifier output in its adversarial loss.
- The generator update hyperparameter n is set to 4 adversarial-loss updates per mini-batch, which worked well across settings.
B DIGITS CLASSIFICATION TRAINING DETAIL
The digits experiments follow an established adaptation protocol using standard splits or sampled subsets, with balanced source-target mini-batches and batch size 128.
- SVHN-to-MNIST uses the standard training splits for training and the MNIST test split for evaluation.
- MNIST-to-USPS adaptation samples 2,000 MNIST images and 1,800 USPS images.
- Each mini-batch contains equal numbers of source and target samples, with batch size 128 for both domains.
C OBJECT CLASSIFICATION TRAINING DETAIL
Object-classification training uses SGD and task-specific batch sizes, while comparison methods receive separate reporting protocols. Kernel changes showed little effect, whereas domain-classifier accuracy declined after one epoch under large domain differences.
- SGD uses learning rate 1.0 × 10^-3, with batch size 32 for ResNet101 and 24 for ResNext due to GPU-memory limits.
- The MMD model uses five RBF kernels, and changing kernel counts or parameters produced no significant performance difference.
- The domain-classification baseline reports accuracy after one epoch because accuracy dropped significantly afterward, possibly from the large synthetic-to-real domain difference.
- The ENT baseline reports accuracy after five epoch updates because its accuracy drops significantly after approximately five epochs.
D SEGMENTATION EXPERIMENTS DETAIL
The semantic-segmentation implementation modifies FCN for ResNet and integrates features from multiple ResNet layers. ResNet50 serves as the generator, while convolution and upsampling layers form the critic network.
- The implementation modifies FCN architecture to work with a ResNet structure.
- Features from ResBlock 2–4, the first convolution layer, and the maxpooling layer are integrated.
- ResNet50 is treated as the generator, while the remaining convolution and upsampling layers constitute the critic network.
- Input images were resized to 512x1024 because of GPU-memory limits.
E GAN TRAINING
The GAN-training setup uses the critic architecture of Salimans et al. with a fixed batch size and Adam optimization. A separate classifier follows the critic’s conv6 layer outside the adversarial-learning process.
- The critic architecture follows the design proposed by Salimans et al. (2016).
- Training uses batch size 100 and Adam with learning rate 2.0 × 1.0−4.
- A classifier is constructed after the critic’s conv6 layer and is separate from the adversarial-learning process.