Adaptive Methods for Real-World Domain Generalization

1. Introduction

• Domain generalization learns from heterogeneous training domains to classify samples from unseen test domains, and domain-invariant learning seeks features that support a universal classifier.Source: §1 · p. 1
• A universal classifier that minimizes average risk may be far from the optimum for a particular test domain, especially when domain distributions vary.The proposed adaptive classifier uses a few unlabeled samples from the test domain and requires no further training.Source: §1 · p. 1
• The approach first embeds each domain using a few unlabeled samples, then uses the embedding as supplementary information to train an adaptive classifier.Source: §1 · p. 1
• The paper adapts low-shot prototypical learning [39] to construct domain embeddings from unlabeled domain samples and provides an algorithm for training adaptive classifiers with them.Source: §1 · p. 1

2. We justify our design choices with novel theoreti-

• The paper distinguishes its setting from unsupervised domain adaptation, which assumes unlabeled samples from unseen domains are available during training.Source: §2 · p. 1
• The paper uses kernel mean embeddings and product-kernel risk minimization to derive generalization bounds on average risk for adaptive classifiers.Source: §2 · p. 2
• Geo-YFCC contains 1.1M samples across 40 training, 7 validation, and 15 test domains, with domain shift, label shift, and long-tailed class distributions.It is constructed from YFCC100M [42] and is described as substantially more challenging than incumbent benchmarks.Source: §2 · p. 2
• On existing benchmarks and two large-scale benchmarks, adaptive classification with finetuning (ERM) demonstrates strength over existing approaches, while naive ERM can also be a powerful baseline.Source: §2 · p. 2

2. Related Work

• Prior domain-generalization work includes invariant representations [41] [25], meta learning [24], invariant risk minimization [2], distributionally robust optimization [38], mixup [46] [47] [44], and causal matching [32].The paper instead focuses on classifiers specialized to each target domain.Source: §2 · p. 2
• The paper describes adaptive classification as complementary to invariant learning and says it can be combined with invariant approaches.Source: §2 · p. 2
• Kernel mean embeddings were proposed for adaptive domain generalization; this paper complements theoretical work [12] with an efficient classification algorithm requiring only a few unsupervised samples.It also extends results of to a larger class of functions.Source: §2 · p. 2
• The paper proposes domain-generalization benchmarks based on ImageNet LSVRC12 and YFCC100M [42], addressing large-scale computer-vision challenges including long-tailed distributions and long training times [1].Extensive hyperparameter tuning and optimal model selection are difficult in large-scale settings because training takes a long time.Source: §2 · p. 2

3. Approach

• A target-domain optimal classifier can lie far from the universal classifier, motivating adaptation to reduce this adaptivity gap.The gap may remain large even when training includes domains with classifiers close to the target optimum, because the universal classifier minimizes overall risk.Source: §3 · p. 2

3.1. Prototypical Domain Embeddings

• Prototypical domain embeddings use domain identities to cluster same-domain samples and separate different domains, targeting expressive and consistent embeddings.Embedding construction and test-domain prototypes require only unlabeled samples, with no retraining for each test domain.Source: §3.1 · pp. 3–4

3.2. ERM on Augmented Inputs

• DA-ERM concatenates image features with a domain embedding and trains the classifier with regular ERM; test-domain embeddings come from a small unlabeled sample.The test embedding may also be computed from a small sample of the total test set.Source: §3.2 · p. 4

3.3. Theoretical Guarantees

• Embedding approximation error is bounded at order σ/√n, and the analysis gives adaptive-classifier generalization bounds with tighter dependence on n than prior results.The approximation bound applies when the feature second moment is bounded by σ²; the generalization bound incorporates feature variance σ.Source: §3.3 · pp. 4–5

4. Large Scale Benchmarks

• Large-scale benchmarks address the scarcity of domain-generalization evaluations and the infeasibility of leave-one-domain-out model selection at scale.The largest dataset then in use had 7 domains and 5 classes.Source: §4 · p. 5

4.1. Geo-YFCC

• Geo-YFCC contains 1,147,059 images across 1,261 categories and 62 country domains, with 45 training, 7 validation, and 15 test domains.The benchmark exhibits long-tailed label distributions, label shift, and covariate shift across domains. Images are selected from YFCC100M using geotags and keyword-filtered ImageNet-5K categories.Source: §4.1 · p. 5

4.2. Long-Tailed ImageNet (LT-ImageNet)

• LT-ImageNet introduces controlled label shifts and long-tailed class distributions by varying head- and tail-class sampling across training domains.Validation and test domains restrict classes to those present in training, so the setting does not tackle zero-shot domain generalization.Source: §4.2 · p. 5

5.1. Small-Scale Benchmarks

• Across five small-scale datasets, DA-ERM improves average accuracy by 1.2% over ERM and performs better or competitively with the best domain-invariant approach on each dataset.The comparisons follow DOMAINBED and average results over eight independent trials.Source: §5.1 · pp. 6–7, Table 1 · p. 7
• DA-MMD improves by 2.3% over MMD (62.2% vs. 64.5%), and DA-CORAL improves by 0.9% over CORAL.These variants replace DA-ERM's cross-entropy loss with the corresponding domain-invariant loss.Source: §5.1 · p. 7

5.2. Large-Scale Benchmarks

• DA-ERM improves LT-ImageNet top-1 accuracy by 2.3% on train domains (71.1% vs. 73.4%) and 2.4% on test domains (53.7% vs 56.1%).The comparisons use the dataset's default settings.Source: §5.2 · p. 7, Table 2 · p. 7
• On Geo-YFCC test domains, DA-ERM improves by an absolute 1% over benchmark techniques, while maintaining training-domain performance.The benchmark has 1,261 classes and significant country-level domain differences.Source: §5.2 · p. 7, Table 2 · p. 7

5.3.1 Dataset Ablations

• With a fixed number of points per domain, DA-ERM's advantage grows as training-domain count increases; with a fixed total point count, it outperforms other algorithms as domains increase, then dips slightly beyond 100 domains.The authors relate the dip to an inflection point from competing terms in Theorem 4.Source: §5.3.1 · p. 7, Figure 3 · p. 8
• When tail-class representation is below 5%, domain-adaptive training exceeds naive ERM by about 3.5% in average top-1 performance; as imbalance lessens, its average improvement is 1.8%.The parameter f controls the fraction of points available in tail classes relative to head classes.Source: §5.3.1 · p. 8, Figure 3 · p. 8

5.3.2 Algorithm Ablations

• On Geo-YFCC, prototype performance declines only when fewer than 50 points construct the embedding; it remains virtually identical from 50 through 2K points.This ablation tests consistency of the domain embedding algorithm.Source: §5.3.2 · p. 8
• Prototype embeddings are more expressive than mean features; softmax embeddings perform worse on LT-ImageNet but slightly better on Geo-YFCC, and random embeddings significantly degrade performance.Sampling 3–8 domains per prototypical-training round has no significant effect.Source: §5.3.2 · p. 8, Table 3 · p. 8

6. Conclusion

• The paper presents an adaptive domain-generalization framework and reports the first experiments at scale, aiming to bring evaluation closer to real-world applications.The authors say their results suggest significant effort is needed to scale to real-world problem settings.Source: §6 · p. 8

A.1. Motivation

• The proposed classifier uses a function over domains and inputs, rather than controlling empirical risk through an inter-domain variance penalty.The modeled prediction is F(D_X, x), where D_X is the input marginal under domain D.Source: §A.1 · p. 9

A.2. Background

• The theoretical setup treats domains as i.i.d. samples from a measure over input-label distributions and models test predictions using an empirical input distribution.The passage also describes the empirical distribution approaching the true input distribution as the sample count grows.Source: §A.2 · pp. 9–10
• The approach is compatible with invariant methods, but the paper does not investigate that combination in detail.The paper frames adaptive and invariant approaches as addressing separate aspects of the problem.Source: §A.2 · p. 9

A.3. Kernel Assumptions

• The theoretical setup uses a Lipschitz kernel on distributions and inputs, with boundedness and a Hölder condition on the distribution embedding.The distribution kernel uses a nonlinear kernel and a kernel mean embedding.Source: §A.2 · p. 10, §A.3 · p. 10

A.4. Consistency Bound (Theorem 1 of the Main Paper)

• Theorem 3 gives a high-probability result for an empirical distribution formed from n samples, under bounds on feature norms and expected squared norms.The supplied theorem passage does not include the bound’s expression.Source: §A.4 · p. 10
• The proof treats the embedding feature as a finite-dimensional random vector and applies a vector-valued Bernstein inequality.The proof passage identifies boundedness of the kernel as the basis for bounding feature norms.Source: §A.4 · p. 10
• The proof applies the concentration result to the empirical average of feature vectors and rearranges the resulting inequality.The passages define the population embedding as the expected feature vector.Source: §A.4 · p. 10

A.5. Generalization Bound (Theorem 2 of Main Paper)

• Theorem 4 gives a uniform risk bound for N i.i.d. training domains with n-sample training sets and a bounded, Lipschitz loss.The result is stated with probability at least 1 − δ.Source: §A.5 · p. 11
• The proof decomposes the bound, controls its terms, and combines them with a union bound over the N domains.One term is controlled using the loss’s Lipschitz property; another adapts the argument of Blanchard et al. to a Lipschitz kernel.Source: §A.5 · p. 11
• The analysis uses Bernstein concentration to obtain a variance-dependent result that can be tighter when variance is low, and extends product-kernel results to general Lipschitz kernels.The authors describe the latter kernels as more suitable for deep-learning systems.Source: §A.5 · p. 11

B. Two-Step Training Procedure

• The domain-aware network predicts a class label from an image augmented with its domain prototype.The prototype is obtained from the preceding domain-embedding step.Source: B. Two-Step Training Procedure · p. 12
• The network concatenates image features with the domain prototype and passes them through nonlinear layers to produce its output.The classifier is parameterized by both the network weights and the prototype-construction parameters.Source: B. Two-Step Training Procedure · p. 12
• At test time, prototype construction can be decoupled from inference by using a distinct set of n_p unsupervised points for each test domain.The cited remark says the resulting disparity from using the optimal prototype is small when σ_P is small.Source: B. Two-Step Training Procedure · p. 12

C.1. Small-Scale Hyperparameters

• The small-scale setup follows Gulrajani and Lopez-Paz [19] and uses their default values, with hyperparameter search distributions summarized in Table 4.Source: §C.1 · p. 12, Table 4 · p. 13

C.1.1 Prototypical Network

• The prototypical network uses an ILSVRC12-initialized ResNet-50 and 1,000 iterations, with four domains per batch and half the points reserved for prototype creation.Its batch size is 32 per domain; the learning rate is 1e−6 and weight decay is 1e−5.Source: §C.1.1 · p. 13

C.1.2 DA-CORAL and DA-MMD

• DA-CORAL and DA-MMD use the DA-ERM architecture, adding CORAL or MMD regularization with γ = 1 at the final penultimate layer.The regularizer is placed after the bottleneck and before the logits.Source: §C.1.2 · p. 13

C.2.1 LT-ImageNet

• LT-ImageNet DA-ERM prototype training uses 1,300 iterations, batch size 100 per domain, and a 50% support set, while main training runs five epochs across all methods.Prototype training uses learning rate 0.001 and weight decay 1e−5; main training uses learning rate 0.005 and weight decay 1e−5, with γ = 1 for MMD and CORAL.Source: §C.2.1 · p. 13

C.2.2 Geo-YFCC

• Geo-YFCC DA-ERM prototype training uses 1,300 iterations, batch size 80 per domain, and an 80% support set; all methods use six main-training epochs.Prototype training uses learning rate 2e−2 and weight decay 1e−5; main training uses learning rate 0.04 and weight decay 1e−5, with γ = 1 for MMD and CORAL.Source: §C.2.2 · p. 14

• Performance remains similar through 50 prototype points on LT-ImageNet and Geo-YFCC.The authors note this is desirable when few samples from new domains are available.Source: Table 5 · p. 14, D. Consistency Experiment · p. 14