Towards Non-I.I.D. Image Classification: A Dataset and Baselines

Motivation

• Training and testing distributions can differ in practice, so minimizing training error does not necessarily ensure good testing performance.This challenge is especially relevant when testing distributions cannot be constrained, including risk-sensitive applications.Source: §1 · pp. 2–3

Problem

• Existing benchmark datasets support I.I.D. evaluation but provide no explicit option to simulate Non-I.I.D. image classification.The absence of a well-structured, reasonably scaled dataset has also limited development of dedicated models.Source: §1 · pp. 2–3

NICO dataset

• NICO labels images by both main concept and context, allowing models to train on some contexts and test on unseen contexts.For example, dog images can be separated into grass, car, and beach contexts.Source: §1 · p. 2
• The context proportions in training and testing data can be adjusted to control the degree of distribution shift.NICO supports Targeted Non-I.I.D. settings with known shifts and General Non-I.I.D. settings with unknown testing distributions.Source: §1 · pp. 2–3
• NICO contains 19 classes, 188 contexts, and nearly 25,000 images, with sufficient scale to train deep convolutional networks from scratch.The dataset scale was described as increasing and already sufficient for ConvNet training from scratch.Source: §1 · p. 3

Baseline model

• CNBB is proposed as a ConvNet baseline for General Non-I.I.D. image classification, and its batch-balancing mechanism helps reduce distribution-shift effects.The experiments show that batch balancing helps ConvNets resist the negative effect of distribution shift to some extent.Source: §1 · p. 3

2.1. Problem Definition

• Non-I.I.D. classification learns a feature extractor and classifier when testing data differ from training data, with testing-distribution knowledge distinguishing targeted from general settings.General Non-I.I.D. classification assumes the testing distribution is unknown, whereas targeted classification assumes it is known.Source: §2.1 · pp. 3–4
• NI measures distribution shift between training and testing data using feature-extractor statistics, although its numerical value depends on the chosen extractor.The paper uses NI to quantify shift and analyze trends when the extractor is fixed.Source: §2.1 · p. 4, §2.2 · p. 5

2.2. Existence of Non-IIDness

• NI is above zero for every class, showing that training–testing Non-IIDness occurs even in ImageNet and other large-scale datasets.Different classes have different NI values, and higher NI corresponds to higher testing error.Source: §2.2 · p. 5
• The NI–testing-error correlation is r = 0.95 with p = 2e-15, supporting NI as a key factor influencing classification performance.The analysis uses Figure 1 to relate class-level NI to testing error.Source: §2.2 · p. 5, Figure 1 · p. 5

2.3. Limitations of Existing Datasets

• NICO is designed to support various Non-I.I.D. situations flexibly and consciously, addressing the need for a dedicated dataset.Its construction uses contextual variation to create controllable distribution shifts.Source: §2.2 · p. 5, §2.3 · p. 6
• Existing benchmark datasets provide relatively small shifts and no explicit control over their degree, preventing fair and systematic Non-I.I.D. evaluation.Across three identically structured ImageNet-derived datasets, the NI of a given class is unstable.Source: §2.3 · p. 6

3. The NICO Dataset

• The NICO section presents the dataset’s properties, collection procedure, and preliminary empirical results across different Non-I.I.D. settings.The section is organized around dataset construction and initial evaluation.Source: §3 · p. 7

3.1. Context for Non-I.I.D. Images

• NICO labels each image with both a main concept and its concrete context, enabling training and testing on different contexts.Examples include dog on grass, cat on snow, horse beside people, and airplane in sunrise.Source: §3.1 · p. 7
• A model for Non-I.I.D. classification is expected to perform well in both training contexts and unseen testing contexts.The benchmark therefore evaluates contextual generalization rather than only concept recognition within observed contexts.Source: §3.1 · p. 7
• The dataset includes object attributes such as color, action, and shape, along with backgrounds or scenes as context types.These context types form concept-plus-context pairs such as white bear, climbing monkey, and double decker.Source: §3.1 · p. 7
• Selecting contexts that occur across multiple concepts lets a context appear in positive samples and optionally in negative samples, producing different Non-I.I.D. conditions.This design adds flexibility beyond simply changing contexts between training and testing.Source: §3.1 · p. 7

3.2. Data Collection and Statistics

• The current NICO release has 10 Animal classes and 9 Vehicle classes, with 9 or 10 contexts per class.The dataset is organized into two superclasses.Source: §3.2 · pp. 7, 9
• NICO selects frequently co-occurring tags from YFCC100m, filters them across concepts, and manually retains tags matching object attributes or backgrounds and scenes.The process is guided by cross-concept occurrence and manual consistency checks.Source: §3.2 · p. 8
• Images are collected by querying Google and Bing with concatenated concept and context tags, then screened for correct content and balanced context counts.Screening requires adequate, approximately balanced images for each context within a class.Source: §3.2 · p. 8
• The collection avoids object-centralization filtering, making selected images more realistic and in-the-wild than ImageNet images.The paper explicitly notes that no image registration or object-centralization filtering is performed.Source: §3.2 · p. 8
• NICO contains 25,000 images, averages about 1,300 images per class, and has 83–215 images per context on average.Its hierarchical structure supports future expansion.Source: §3.2 · p. 9

3.3. Supported Non-I.I.D. Settings

• NICO defines four Non-I.I.D. settings—minimum, proportional, compositional, and adversarial bias—with different levels of distribution shift.The settings are designed to generate Non-I.I.D. training and testing subsets, with distribution-shift severity increasing toward adversarial bias.Source: §3.3 · pp. 9–10

Setting 1. Minimum Bias

• Random sampling creates a minimum-bias setting that approximates an i.i.d. scenario by minimizing distribution shift between training and testing data.Positive and negative samples are randomly split into training and testing subsets while ignoring contexts.Source: §3.3 · pp. 9–10

Setting 2. Proportional Bias

• Proportional bias changes context proportions across training and testing, allowing distribution-shift severity to be tuned by adjusting their proportion difference.The dominant ratio compares the dominant context size with the average size of other contexts, and experiments vary either the training or testing ratio.Source: §3.3 · p. 10

Setting 3. Compositional Bias

• Compositional bias withholds some contexts from training while testing across all contexts, so fewer observed contexts produce greater information loss and distribution shift.Combining compositional and proportional bias produces a more radical shift; the setting reflects datasets that do not contain every potential context.Source: §3.3 · p. 10

Setting 4. Adversarial Bias

• Adversarial bias selects negative samples from contexts excluded from or included in positive training samples, creating stronger shifts and making i.i.d.-developed models more prone to failure.The number of confounding contexts can be varied, and adding more confounding contexts further increases distribution-shift severity.Source: §3.3 · p. 10

3.3. Supported Non-I.I.D. Settings

• Each setting supports targeted or general Non-I.I.D. classification according to whether the testing-subset distribution is known or unknown.NICO therefore varies both the kind of shift and the assumption about access to the testing distribution.Source: §3.3 · pp. 9, 11

3.4.1. Minimum Bias Setting

• 49.6% accuracy and 3.85 NI were obtained for Animal, versus 63.0% accuracy and 3.20 NI for Vehicle, even under random sampling.NICO’s NI was reported as much higher than ImageNet without explicit sampling bias.Source: §3.4.1 · p. 11

3.4.2. Proportional Bias Setting

• NI increases consistently as the discrepancy between dominant training and testing ratios becomes larger, enabling controlled simulation of different distribution-shift extents.The dominant ratio of either training or testing data is varied while the other is fixed.Source: §3.4.2 · p. 13

3.4.3. Compositional Bias Setting

• NI consistently decreases as more contexts are observed during training, while combining compositional and proportional bias creates a more severe Non-I.I.D. condition.The combined setting uses restricted context observations and dominance choices to intensify the shift.Source: §3.4.3 · pp. 13–14

3.4.4. Adversarial Bias Setting

• The number of confounding contexts consistently influences NI across target classes, and adding confounding contexts can further confuse the ConvNet.The adversarial experiment varies confounding-context counts for four Animal classes.Source: §3.4.4 · p. 14
• Across supported settings, NICO shows significantly higher NI than ImageNet, with an ascending trend from minimum to adversarial bias.Figure 8 summarizes the range of average NI over the Animal superclass across settings.Source: §3.4.4 · p. 14, Figure 8 · p. 14

4. General Non-I.I.D. Image Classification

• The method addresses prior approaches’ reliance on testing-distribution knowledge, multiple labeled domains, or sufficiently diverse training data for Non-I.I.D. image classification.These constraints limit the feasibility or scalability of targeted methods and prevent leveraging ConvNet representation learning on large-scale images.Source: §4 · pp. 15–16

4.1. ConvNet with Batch Balancing

• CNBB extends global confounder balancing into ConvNets by learning batch sample weights and combining weighted softmax loss with quantization loss.Unlike CRLR, CNBB balances confounders within batches, first learning sample weights unsupervised with fixed representations before optimizing the model.Source: §4 · p. 16, §4.1 · p. 18
• To adapt confounder balancing to ConvNets, CNBB uses quantization loss to binarize continuous features and performs balancing over each training batch rather than all samples.The batch formulation makes the procedure feasible when the full training set cannot be processed in one iteration.Source: §4.1 · p. 17

4.2. Experiments on NICO

• Experiments on NICO evaluate CNBB against CNN and CNN+BN across minimum, proportional, compositional, and combined proportional-compositional bias settings.The settings vary training/testing dominant ratios and the contexts observed during training versus testing.Source: §4.2 · p. 18, §4.2.1 · p. 18

4.2.2. Experimental Results

• Under explicit distribution shift, CNBB outperforms the other baselines at almost every setting; in minimum bias, it reaches 49.94% versus CNN’s 49.60%, while CNN+BN records 46.48%.Source: §4.2.2 · pp. 19–20
• The batch-balancing module enables CNBB to identify more stable features and partially resist negative effects caused by distribution shift.The comparison indicates that features normalized only from training data are less stable than those learned with batch balancing.Source: §4.2.2 · p. 20
• CNBB gains larger improvements over baselines as the training–testing discrepancy, measured by higher NI, increases.Table 5 summarizes CNBB’s average improvement over the best baseline across experiments, demonstrating an advantage in more challenging Non-I.I.D. settings.Source: Table 5 · p. 20, §4.2.2 · p. 20
• For training data with a dominant ratio of 3:1, Figure 10 analyzes testing accuracy and shows a clear convex hull as α increases, with CNBB improving over CNN.Source: Figure 10 · p. 21, §4.2.2 · p. 21
• Both valid sample size and batch-balancing degree are critical to Non-I.I.D. image-classification performance.When α is large, sample weights become more uniform and valid sample size increases; when α is zero, weights concentrate on fewer images, producing stronger batch balancing.Source: §4.2.2 · p. 21

• NICO is presented as a well-structured Non-I.I.D. image dataset with reasonable scale for training ConvNets.Source: §5 · p. 21
• Context labels let NICO provide varied Non-I.I.D. settings and consciously create different levels of Non-IIDness.Source: §5 · p. 21
• The proposed ConvNet baseline targets General Non-I.I.D. classification when testing data have an unknown distribution shift from training data.Source: §5 · p. 22
• Empirical results demonstrate NICO's ability to train ConvNets and the proposed model's superiority across various Non-I.I.D. settings.Source: §5 · p. 22
• Future work includes improving NICO's quality, quantity, controllability, scale, and coverage of Non-I.I.D. settings.Source: §5 · p. 22
• The authors also plan to design more effective models for different Non-I.I.D. settings.Source: §5 · p. 22

• Table 1 describes the basic structure of the ConvNet used in the paper.Source: Table 1 · p. 23
• Table 2 reports data sizes for each context and class in the Animal superclass.Source: Table 2 · p. 24
• Table 3 reports data sizes for each context and class in the Vehicle superclass.Source: Table 3 · p. 25