PaperlayerView exact revision on arXiv

Source-linked AI summary

NICO++: Towards Better Benchmarking for Domain Generalization

Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, Peng Cui

arXiv:2204.08040v2cs.CVcs.LG

TL;DR

Distribution shifts challenge machine-learning methods that rely on the I.I.D. assumption. This paper introduces NICO++ and evaluation methods for domain generalization, reporting stronger evaluation capability than current datasets and a contribution to fairer benchmarking.

  • Problem

    Most machine-learning algorithms rely on an I.I.D. assumption that is rarely satisfied in real applications, where training and test distributions can shift.

    Source: §1 · p. 1

  • Method

    The paper constructs NICO++ with extensive domains and proposes covariate-shift and concept-shift metrics plus generalization bounds to evaluate DG datasets.

    Source: §1 · p. 2

  • Results

    Experiments show NICO++ has superior evaluation capability to current DG datasets, while benchmarking finds substantial room to improve generalization methods and helps alleviate oracle-leakage unfairness.

    Source: §1 · pp. 2–3, §6 · p. 13

  • Takeaways & Limitations

    NICO++ contributes a fairer benchmark for evaluating domain-generalization methods and supports comprehensive evaluation across multiple target domains.

    Source: §1 · p. 2

    The analysis represents each domain by an input distribution and labeling function, with the formulation covering potentially nondeterministic as well as deterministic labeling.

    Source: §4 · p. 5

Abstract

from arXiv · show

Despite the remarkable performance that modern deep neural networks have achieved on independent and identically distributed (I.I.D.) data, they can crash under distribution shifts. Most current evaluation methods for domain generalization (DG) adopt the leave-one-out strategy as a compromise on the limited number of domains. We propose a large-scale benchmark with extensive labeled domains named NICO++ along with more rational evaluation methods for comprehensively evaluating DG algorithms. To evaluate DG datasets, we propose two metrics to quantify covariate shift and concept shift, respectively. Two novel generalization bounds from the perspective of data construction are proposed to prove that limited concept shift and significant covariate shift favor the evaluation capability for generalization. Through extensive experiments, NICO++ shows its superior evaluation capability compared with current DG datasets and its contribution in alleviating unfairness caused by the leak of oracle knowledge in model selection.

1 Introduction

Distribution shift can make models unreliable in real applications, while limited domains force DG evaluations to test on only one unseen distribution at a time. NICO++ combines extensive domains with evaluation protocols and shift metrics; experiments report stronger evaluation capability and reduced oracle-leak improvement space.

1 Introduction

  • Distribution shift can make current approaches unreliable in real applications where the I.I.D. assumption is rarely satisfied.The passage highlights healthcare, autonomous driving, and security systems as high-stakes settings.Source: §1 · p. 1
  • Current DG benchmarks have few domains or limited distribution shifts, so leave-one-out evaluation tests only one unseen distribution after training.The paper describes leave-one-out as a compromise caused by limited domain availability.Source: §1 · p. 2
  • NICO++ provides 80 categories, 10 aligned common domains, 10 category-specific domains, and more than 200,000 images for flexible training and multi-target evaluation.Real-world photos and consistency within category concepts provide common knowledge for recognition across domains.Source: §1 · p. 2
  • The paper proposes metrics for covariate and concept shift and generalization bounds showing that significant covariate shift with limited concept shift favors evaluation capability.The analysis evaluates dataset construction rather than model design.Source: §1 · p. 2
  • DG model selection can become unfair when target-data knowledge is used to choose hyperparameters that favor test performance.The paper notes that test performance also varies substantially with algorithm-irrelevant hyperparameters on current datasets.Source: §1 · p. 2
  • NICO++ experiments show superior evaluation capability to current DG datasets and help alleviate oracle-leak unfairness by squeezing its possible improvement space.The experiments also report substantial room for improving generalization methods on NICO++.Source: §1 · pp. 2–3

2 Related Works

Related work spans synthetic-shift datasets and datasets collected from multiple source domains, each with limitations in realism, diversity, or controllability. DG methods are commonly grouped into representation learning, training strategies, and data augmentation.

Benchmark Datasets

  • One dataset branch simulates shifts through transformations, while another collects images from distinct source domains.Examples include synthetic variants and datasets such as PACS, Office-Home, WILDS, DomainNet, Terra Incognita, NICO, and VLCS.Source: §2 · p. 3
  • Existing datasets may use simple distribution criteria, lack domain diversity, lack controllable shift strength, or have domains unaligned across categories.The passage contrasts these limitations across several datasets, including iWildCam and the previous NICO version.Source: §2 · p. 3

Domain Generalization

  • DG methods are grouped into representation learning, training strategies, and data augmentation, whereas domain adaptation uses extra information about test domains.The passage also points to broader surveys of DG methods.Source: §2 · p. 3

3 NICO++: Domain-Extensive Large Scale Domain Generalization Benchmark

NICO++ is a large-scale DG benchmark designed to increase domain and category heterogeneity while supporting varied evaluation settings. It combines hierarchical categories, aligned common domains, category-specific unique domains, and a large annotated image collection.

3 NICO++: Domain-Extensive Large Scale Domain Generalization Benchmark

  • NICO++ enriches the types of categories and domains and collects more images to support thorough evaluation of domain generalization.Each image has a category label for its objective concept and a domain label for other visual information.Source: §3 · p. 4

3.1 Constructions of the Category / Domain Labels

  • The dataset has 80 concrete categories arranged under four broad categories, with 10 common domains aligned across categories and 10 unique domains for each category.Common domains represent semantically similar context families; unique domains support flexible settings where training domains are not category-aligned.Source: §3.1 · p. 4

3.2 Data Collection and Statistics

  • Two annotators reviewed each downloaded image, and the selected dataset contains 232.4k images from more than 1.0 million downloaded images.The data collection used over 32,000 category-domain search combinations; most common and unique domains contain at least 200 and 50 images, respectively.Source: §3.2 · pp. 4–5
  • NICO++ is large enough to support training deep convolutional networks such as ResNet-50 from scratch across DG scenarios.Figure 1 encodes instance counts by category and domain; its lower domains are common across categories, while its upper domains are category-specific.Source: §3.2 · p. 5, Figure 1 · p. 5

4 Covariate Shift and Concept Shift

This section defines metrics for covariate and concept shift and uses generalization bounds to characterize which shifts make a domain-generalization benchmark discriminative. Empirically, NICO++ has lower concept shift than the compared datasets, while its covariate shift is comparable to PACS and DomainNet.

4.1 Metrics for Covariate shift and Concept shift

  • Covariate shift is measured by discrepancy between domain input distributions, while concept shift uses the minimum and maximum cross-domain labeling-function losses.For 0-1 loss, concept shift reflects the probability of inconsistent labeling functions under training or test input densities.Source: §4.1 · pp. 6–7

4.2 Dataset Evaluation with the Metrics

  • The bounds indicate that significant covariate shift and limited concept shift favor a discriminative DG benchmark.Theorem 4.3 relates the test-loss bound to concept and covariate shift; the section argues that dataset properties, rather than model choice, can modify this bound.Source: §4.2 · pp. 7–9
  • Theorem 4.2 bounds the training–test performance gap, estimating how much dataset shift contributes to performance variation or drop.The stated results assume a symmetric loss satisfying the triangle inequality and that the training and test labeling functions belong to H.Source: §4.2 · p. 8

4.3 Empirical Evaluation

  • NICO++ has significantly lower concept shift than other datasets, while its covariate shift is comparable to PACS and DomainNet.The difference Mcpt − Mcov is lower than 0 on NICO++ but higher than 0 on current DG datasets.Source: Table 1 · p. 9, §4.3 · p. 9

5 Experiments

The experiments benchmark domain generalization with aligned and flexible domain settings, introduce metrics for accuracy and cross-domain variability, and compare NICO++ with existing datasets. Results indicate that NICO++ yields lower test variance and reduces the potential advantage from oracle-guided model selection.

5 Experiments

  • The experiments assess classic and flexible DG using NICO++ protocols designed for aligned and flexible domains.The paper reports major results in the main text and provides additional experiments and implementation details in the appendix.Source: §5 · p. 10

5.1 Evaluation Metrics for Algorithms

  • Average accuracy, overall accuracy, and cross-domain standard deviation capture different evaluation properties, with overall accuracy suited to mixed or category-varying domains.The authors note that standard deviation is informative when models are tested across multiple domains with fixed source domains, unlike leave-one-out evaluation.Source: §5.1 · p. 10

5.2 Benchmark for Classic Domain Generalization

  • Models generally generalize better when tested on common domains from a single cluster than across clusters, while a significant gap from Oracle remains.The clusters reflect location, background, and time shifts; Table 2 reports results for the classic DG setting.Source: §5.2 · pp. 10–11, Table 2 · p. 11

5.3 Benchmark for Flexible Domain Generalization

  • In flexible DG, current state-of-the-art methods outperform ERM, yet remain significantly below Oracle performance.The experiments include random and compositional settings with flexible category-domain combinations.Source: §5.3 · p. 12, Table 3 · p. 12

5.4 Test Variance and Model Selection

  • NICO++ shows lower test variance across random seeds and training epochs than other datasets, squeezing the improvement space from oracle knowledge and supporting fairer DG comparisons.The authors interpret the lower variance as a more stable estimate of generalization, robust to algorithm-irrelevant hyperparameters.Source: Table 4 · p. 12, §5.4 · p. 13

6 Conclusion

The conclusion presents NICO++ as a large-scale, context-extensive benchmark with evaluation methods for domain generalization. It reports that experiments show advantages over current datasets and comprehensive benchmarking of DG algorithms.

6 Conclusion

  • The paper concludes that NICO++ and its evaluation methods support comprehensive DG evaluation, with experiments showing advantages over current datasets.The proposed dataset is paired with metrics for covariate and concept shift based on two generalization bounds.Source: §6 · p. 13

A.1 Empirical version of Theorem 4.2 and Theorem 4.3

The appendix relates empirical versions of two generalization bounds to finite-sample size and explains the proposed covariate-shift metric through domain classification.

A.1 Empirical version of Theorem 4.2 and Theorem 4.3

  • The appendix introduces empirical Rademacher complexity to derive data-dependent bounds from empirical samples.The stated theorem conditions include a symmetric, bounded loss satisfying the triangle inequality and source and target functions in H.Source: §A.1 · p. 21
  • The empirical bounds become tighter as sample size increases and converge to the bounds in Theorems 4.2 and 4.3 as sample size tends to infinity.The appendix states that these bounds quantify finite-sample effects.Source: §A.1 · p. 21

A.2 An Intuitively Explanation of Proposed Metrics

  • The proposed covariate-shift metric connects to domain classification: more distinguishable training and test subsets indicate stronger covariate shift.The explanation relates discrepancy distance to dA-distance and optimal domain-classification accuracy.Source: §A.2 · p. 22

A.3 Comparison between the Proposed Metrics and Kullback-Leibler Divergence

The proposed shift metrics are presented as easier to estimate than KL divergence and more closely connected to model error. The appendix also compares them with other shift metrics and supplies lemmas and proofs for subsequent bounds.

A.3 Comparison between the Proposed Metrics and Kullback-Leibler Divergence

  • The proposed metrics are easier to estimate with finite samples than KL divergence and connect closely to model error, making them more suitable for evaluating DG datasets.The paper therefore adopts Mcov and Mcpt to measure covariate shift and concept shift.Source: §A.3 · p. 22

A.4 Comparison with Other Metrics

  • Unlike the metrics in Ye et al. [Ye et al., 2021], the proposed metrics do not assume shared labeling rules or equal label distributions and are defined directly from X.The comparison also notes that Ye et al.'s approach requires identifying variables related to X but irrelevant to Y, which can be challenging or unsolvable.Source: §A.4 · p. 23

B.1 Important lemmas

  • Lemmas B.1 and B.2 provide Rademacher and discrepancy-distance bounds, and L_H is defined using losses between pairs of hypotheses.The supplied passages identify the lemmas and define L_H but do not include their full bound statements.Source: §B.1 · p. 23

B.2 Proof of Proposition 4.1

  • For the proof of Proposition 4.1, when H contains all possible functions, the discrepancy is related to L1 distance by a factor of M/2.The proof passage gives the corresponding supremum equality in fragments.Source: §B.2 · p. 24

B.3 Proof of Theorem 4.2

  • Theorem 4.2 bounds test error above by training error, discrepancy distance, and the training-distribution loss between ftr and fte.The derivation uses the assumption fte ∈ H and the triangle inequality for ℓ.Source: §B.3 · p. 24

B.4 Proof of Theorem 4.3

  • Theorem 4.3 bounds test error below using either the training- or test-distribution loss between ftr and fte, minus discrepancy distance and training error.The proof derives both lower-bound forms using fte ∈ H and the triangle inequality for ℓ.Source: §B.4 · p. 24

B.5 Proof of Theorem A.1

The proof of Theorem A.1 invokes Theorem 4.2 and the triangle inequality for discrepancy distance, then combines probabilistic bounds.

B.5 Proof of Theorem A.1

  • The proof starts from Theorem 4.2 and the triangle inequality for discrepancy distance.The cited proof passage introduces a bound containing training error, discrepancy distance, and a minimum of training- and test-distribution losses between ftr and fte.Source: §B.5 · p. 25
  • The proof also invokes Lemma B.1 and Lemma B.2, each with probability at least 1 − δ/3.The supplied passage does not state the complete resulting inequality.Source: §B.5 · p. 25

B.6 Proof of Theorem A.2

Theorem A.2's proof builds on Theorem 4.3 and the triangle inequality for discrepancy distance. The appendix also identifies the scope of additional experiments and their table's reported measures.

B.6 Proof of Theorem A.2

  • Theorem A.2's proof invokes Theorem 4.3 and the triangle inequality for discrepancy distance to derive a lower bound involving the maximum of two losses.The displayed proof expression subtracts discrepancy distance and training error from that maximum.Source: §B.6 · p. 25

C More Experiments and Discussions

  • The additional ResNet-18 DG results report per-target-domain accuracy, overall and mean accuracy, and variance across target domains.The overall accuracy and arithmetic mean can differ because target domains have unequal capacities; results average three repetitions.Source: Table 5 · p. 26

C.1 Benchmark with ResNet-18 as Backbone

The appendix benchmarks DG methods with ResNet-18 using the same settings and data splits as the main paper. It reports that EoA, CORAL, and StableNet remain strong, consistent with the ResNet-50 results.

C.1 Benchmark with ResNet-18 as Backbone

  • The ResNet-18 benchmark keeps the main paper's settings and data splits and includes both classic and flexible DG results.Table 6 reports flexible DG results; the supplied passage does not state its numerical outcomes.Source: §C.1 · p. 26, Table 6 · p. 27, Table 7 · p. 27
  • With ResNet-18, EoA, CORAL, and StableNet still show outstanding performance, consistent with the main-paper results and supporting benchmark consistency across backbones.The appendix presents classic and flexible DG results in Tables 5 and 6, respectively.Source: §C.1 · p. 26

C.2 Pretraining Methods

The appendix examines ImageNet pretraining and an additional NICO++ domain split, then records implementation settings and dataset examples and statistics.

C.2 Pretraining Methods

  • ImageNet pretraining may bias visual-to-label mappings toward domains represented in ImageNet, so the authors benchmark DG methods with randomly initialized ResNet-50.Without pretraining, ERM and most current DG methods remain valid, while IRM and MLDG do not; the authors suggest careful tuning and hyperparameter choice may be required.Source: §C.2 · p. 27

C.3 Other Splits of Domains

  • Across another domain split, EoA, CORAL, and StableNet remain outstanding; the authors associate this consistency with comparable, small concept shifts while noting a considerable oracle*–oracle gap.They also say DG methods have significant room for improvement on NICO++.Source: §C.3 · pp. 27–28, Table 9 · p. 28

C.4 Implementation Details

  • MNIST-M backgrounds are cropped from 200 BSDS500 images, and the appendix specifies CNN and ResNet-50 training settings for dataset evaluation and DG benchmarks.Backgrounds cropped from the same image may differ; DG experiments also report separate settings for ImageNet-pretrained and randomly initialized models.Source: §C.4 · pp. 28–29

D More Statics and Example Images of NICO++

Related papers

Loading 2204.08040v2…