Source-linked AI summary
SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptation
Viraj Prabhu, Shivam Khare, Deeksha Kartik, Judy Hoffman
TL;DR
UDA methods often struggle when covariate shift is accompanied by label distribution shift because unreliable pseudo-labels can cause error accumulation. SENTRY uses transformation-consistency committees to select target instances for entropy minimization or maximization, achieving state-of-the-art results on 27/31 domain shifts.
Problem
UDA methods offer limited success under cross-domain label distribution shift, while noisy pseudo-label self-training can accumulate errors and misalign domains.
Method
SENTRY judges target-instance reliability through predictive consistency across random image transformations, then minimizes entropy for consistent instances and maximizes it for inconsistent ones.
Results
SENTRY significantly improves upon the state of the art across 27/31 shifts from standard and label-distribution-shift UDA benchmarks.
Takeaways & Limitations
Selective confidence adjustment provides a UDA strategy for simultaneous data and label distribution shift, complemented by pseudo-label-based target class balancing.
Abstract
from arXiv · showhide
Many existing approaches for unsupervised domain adaptation (UDA) focus on adapting under only data distribution shift and offer limited success under additional cross-domain label distribution shift. Recent work based on self-training using target pseudo-labels has shown promise, but on challenging shifts pseudo-labels may be highly unreliable, and using them for self-training may cause error accumulation and domain misalignment. We propose Selective Entropy Optimization via Committee Consistency (SENTRY), a UDA algorithm that judges the reliability of a target instance based on its predictive consistency under a committee of random image transformations. Our algorithm then selectively minimizes predictive entropy to increase confidence on highly consistent target instances, while maximizing predictive entropy to reduce confidence on highly inconsistent ones. In combination with pseudo-label based approximate target class balancing, our approach leads to significant improvements over the state-of-the-art on 27/31 domain shifts from standard UDA benchmarks as well as benchmarks designed to stress-test adaptation under label distribution shift.
1. Introduction
UDA methods often assume unchanged label distributions, limiting adaptation when domains differ in both data and label distributions. SENTRY addresses unreliable self-training by selecting target instances through transformation consistency and applying entropy optimization selectively.
- Motivation: Unconstrained pseudo-label self-training can accumulate early mistakes and produce substantial domain misalignment under challenging shifts.Prior approaches may rely on noisy pseudo-labels or miscalibrated predictions.
- Motivation: SENTRY targets UDA under simultaneous data and label distribution shift, a setting where standard distribution-matching assumptions may fail.Many real-world target datasets are imbalanced or follow naturally occurring power-law label distributions.
- Method: SENTRY identifies reliable target instances using predictive consistency across random, label-preserving image transformations.It avoids relying solely on model confidence, which can be miscalibrated under domain shift.
- Method: The method minimizes entropy on highly consistent instances and maximizes entropy on highly inconsistent ones to increase or reduce confidence selectively.This selective objective is designed to avoid reinforcing predictions for likely misaligned instances.
- Contributions: 27/31 domain shifts: SENTRY sets a new state-of-the-art across standard and label-distribution-shift UDA benchmarks.The reported benchmarks include DomainNet, OfficeHome, and VisDA.
2. Related Work
Related UDA work commonly aligns domains through feature discrepancy or adversarial distribution matching, but these methods have limited success under label distribution shift. Other approaches address label shift through weighting, generation, relaxed alignment, or self-training, while SENTRY tackles simultaneous covariate and label distribution shift without additional assumptions.
- Unsupervised Domain Adaptation: Distribution-matching UDA methods achieve limited success when source and target label distributions differ.These methods generally align feature spaces or domain distributions while relying on assumptions that are strained under LDS.
- UDA under Label Distribution Shift: Prior UDA methods under LDS include class-weighted discrepancy measures, pairwise feature-matching generative approaches, and asymmetrically relaxed distribution alignment.Some work assumes label shift without changes in the conditional input distribution.
- Self-training for UDA: Self-training methods adapt using confident target pseudo-labels, confidence regularization, or conditional entropy minimization.These approaches have been explored as promising strategies for UDA under LDS.
3. Approach
SENTRY adapts a source-trained classifier to an unlabeled target domain by evaluating prediction consistency across transformed target images. It selectively minimizes entropy for consistent instances, maximizes it for inconsistent instances, and combines this with source supervision and pseudo-label-based balancing.
- Notation: For each target image, the initial pseudo-label is the model’s argmax prediction over classes.These pseudo-labels are updated during optimization and support target sampling.
- Preliminaries: UDA via entropy minimization: The UDA pipeline first trains on labeled source data and then adapts with unlabeled target data using conditional entropy minimization as a conventional baseline.Naive entropy minimization can reinforce systematic misalignment under strong domain shifts.
- SENTRY: Selective Entropy Optimization via Committee Consistency: SENTRY evaluates each target instance with a committee of random transformed versions and majority-vote consistency against the original prediction.A target instance is marked consistent when most augmented predictions match the original prediction, and inconsistent otherwise.
- SENTRY: Selective Entropy Optimization via Committee Consistency: Consistent instances receive entropy minimization, whereas inconsistent instances receive entropy maximization to reduce confidence on likely misaligned examples.The objective uses a positive entropy term for consistent samples and a negative entropy term for inconsistent samples.
- SENTRY Optimization: SENTRY’s optimization combines source cross-entropy, target information-entropy regularization, and selective entropy optimization.The implementation updates pseudo-labels, enqueues predictions, and resamples the target set during training.
- Overcoming LDS via pseudo class balancing: Class-balanced source sampling and pseudo class-balanced target sampling address label imbalance during adaptation.Target balancing approximates the target label distribution using pseudo-labels.
4. Experiments
SENTRY is evaluated on standard and label-shifted UDA benchmarks, controlled imbalance settings, ablations, and adaptation dynamics. It consistently improves over prior methods, while selective min-max entropy optimization and pseudo-label balancing are central to its gains.
- Results on label-shifted DA benchmarks: SENTRY outperforms prior methods across label-shifted benchmarks, winning every DomainNet shift and 5 of 6 OfficeHome RS→UT shifts.It exceeds the next-best method by 3.55% mean accuracy on DomainNet and 3.58% on OfficeHome RS→UT.
- Varying degree of label imbalance: Under controlled SVHN→MNIST label imbalance, SENTRY achieves 13.6% higher mean accuracy than the next competing method despite worsening performance at higher imbalance factors.The study uses identical data amounts across imbalance factors and reports per-class average accuracy.
- Ablations: Ablations show that selective entropy optimization is crucial: minimizing entropy on all samples is 10.7% and 11.9% worse than SENTRY, while adding maximization improves selective minimization by 1.8% and 1.5%.Oracle experiments further show that selective min-max optimization exceeds minimization alone by 2% and 2.4%.
- Ablations: Pseudo-label class-balanced sampling improves per-class accuracy over random sampling by 0.91% and 0.52% on the two ablation shifts.Without the target information-entropy regularizer, the gaps increase to 2.9% and 3.7%.
- Analysis: SENTRY’s consistency-based selection becomes more selective over adaptation, with minimization selections increasing and maximization selections decreasing across epochs.The method requires k additional forward passes per iteration but no additional backward passes, avoiding sizeable computational overhead over standard entropy minimization.
5. Conclusion
SENTRY addresses UDA under simultaneous data and label distribution shift by selectively trusting consistent target predictions and downweighting inconsistent ones. It improves over state-of-the-art methods across 27 of 31 benchmark shifts.
- SENTRY significantly improves upon the state of the art across 27/31 shifts from several UDA benchmarks.
6. Appendix
The appendix analyzes SENTRY’s components, performance, representations, datasets, and comparison methods. It reports gains from class balancing, broad benchmark improvements, improved alignment under long-tailed shifts, and limitations affecting distribution-matching methods and evaluation protocols.
- Ablations: Class balancing contributes small gains individually, while using both source and target balancing performs best.Source balancing adds +0.5% and +0.3% on C→S, while target pseudo class-balancing adds +2% and +0.1% on Rw→Cl.
- Per-class accuracy: SENTRY outperforms InstaPBM on 37/40 DomainNet Clipart→Sketch categories and remains competitive on the others.
- SVHN→MNIST-LT: 95.65 average accuracy after SENTRY versus 68.05 before adaptation is reported on SVHN→MNIST-LT with imbalance factor 20.
- Representation analysis: SENTRY improves cross-domain alignment across classes on SVHN→MNIST-LT, including head and tail classes, while target accuracy rises from 68.1% to 95.7%.
- Qualitative analysis: Highly consistent target examples tend to have canonical poses and appearances, whereas inconsistent examples are harder to recognize.
- OfficeHome results: SENTRY achieves state-of-the-art performance on 9/12 standard OfficeHome shifts and improves over InstaPBM by 2.5% overall.
- DM methods under LDS: Distribution-matching methods are not expected to succeed under label distribution shift because a target-error bound term becomes unbounded.