PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Importance Weighting for Deep Learning under Distribution Shift

Tongtong Fang, Nan Lu, Gang Niu, Masashi Sugiyama

arXiv:2006.04662v2cs.LGstat.ML

TL;DR

Importance weighting under distribution shift is difficult for deep learning because weight estimation and weighted classification depend on one another, while static pretraining yields biased features. The paper proposes dynamic importance weighting, which iterates between both stages; experiments report favorable comparisons with state-of-the-art methods and effectiveness across label-noise and class-prior-shift settings.

  • Problem

    Importance weighting for deep learning under distribution shift is limited because weight estimation is difficult with complex data and is circularly dependent on classifier training.

  • Method

    Dynamic importance weighting iterates between weight estimation on transformed data and weighted classification, updating the classifier and feature extractor end to end.

  • Results

    Dynamic importance weighting compares favorably with state-of-the-art methods across experiments involving label noise and class-prior shift.

  • Takeaways & Limitations

    The experiments demonstrate the effectiveness of dynamic importance weighting for handling label-noise and class-prior-shift distribution changes.

  • Takeaways & Limitations

    Static importance weighting can produce a feature extractor biased toward training data, while small mini-batches can make repeated weight estimation unreliable.

Abstract

from arXiv · show

Under distribution shift (DS) where the training data distribution differs from the test one, a powerful technique is importance weighting (IW) which handles DS in two separate steps: weight estimation (WE) estimates the test-over-training density ratio and weighted classification (WC) trains the classifier from weighted training data. However, IW cannot work well on complex data, since WE is incompatible with deep learning. In this paper, we rethink IW and theoretically show it suffers from a circular dependency: we need not only WE for WC, but also WC for WE where a trained deep classifier is used as the feature extractor (FE). To cut off the dependency, we try to pretrain FE from unweighted training data, which leads to biased FE. To overcome the bias, we propose an end-to-end solution dynamic IW that iterates between WE and WC and combines them in a seamless manner, and hence our WE can also enjoy deep networks and stochastic optimizers indirectly. Experiments with two representative types of DS on three popular datasets show that our dynamic IW compares favorably with state-of-the-art methods.

1 Introduction

Under distribution shift, importance weighting addresses differing training and test distributions through weight estimation and weighted classification, but deep learning exposes a circular dependency between these stages. Static pretraining can leave the feature extractor biased, motivating dynamic importance weighting, which iterates between both stages.

  • Distribution shift occurs when training and test data come from different distributions, potentially biasing deep classifiers toward training data and reducing test performance.
  • Importance weighting handles distribution shift through weight estimation followed by weighted classification using the estimated weights.
  • Deep importance weighting faces a circular dependency: weighted classification needs estimated weights, while weight estimation benefits from a trained classifier as feature extractor.
  • Static importance weighting pretrains a classifier on unweighted training data, but its resulting feature extractor remains biased toward the training distribution.
  • Dynamic importance weighting iterates between weight estimation and weighted classification, updating weights and classifier features in a seamless end-to-end procedure.

2 Dynamic importance weighting

Dynamic importance weighting addresses the circular dependency between feature extraction, weight estimation, and weighted classification by updating weights and the classifier together. It estimates weights after transforming data and repeatedly matches transformed training and validation distributions during classifier training.

  • Motivation: Importance weighting separates weight estimation from weighted classification, but deep-data settings require expressive transformations because direct estimation in the original d-dimensional space is difficult.The method applies a nonlinear transformation z = π(x, y) into a reduced-dimensional space before weight estimation.
  • Theoretical basis: A fixed, deterministic, invertible transformation preserves the density-ratio relationship between training and test distributions after transformation.The theoretical justification depends on π being fixed, deterministic, and invertible.
  • Circular dependency: The feature extractor creates a circular dependency: weight estimation treats the classifier transformation as fixed, while invertibility requires the classifier to be reasonably good.This is identified as the paper’s major theoretical contribution.
  • Dynamic importance weighting: Dynamic importance weighting updates weights and classifier parameters iteratively, allowing each new weight set to correspond to the classifier at its current training state.Unlike the general goal of recovering the exact density ratio, DIW seeks weights sufficient for the current classifier objective.
  • Weight estimation: DIW estimates weights by matching transformed training and validation distributions with kernel mean matching, subject to bounded-weight and normalization constraints.The objective minimizes a distribution discrepancy represented through maximum mean discrepancy.
  • Algorithm: In each mini-batch, DIW transforms data, estimates weights, weights empirical classification risk, and updates the deep classifier through backpropagation.The algorithm supports hidden-layer-output and loss-value transformation variants.

3 Applications

The paper applies dynamic importance weighting to covariate shift, class-prior shift, and label noise, emphasizing how each shift changes the relationship between training and test distributions. These shifts can bias classifiers toward training-frequency regions, over-represented classes, or corrupted labels.

  • Distribution shift includes covariate shift, class-prior shift, and label noise, each altering training–test distribution relationships differently.
  • Covariate shift: Under covariate shift, unchanged p(y | x) can still harm performance because limited models focus on regions more prevalent in training data.
  • Class-prior shift: Under class-prior shift, only p(y) changes, and the optimal weight is pte(y)/ptr(y), making counting sufficient instead of density-ratio estimation.
  • Label noise: Label noise changes p(y | x) while preserving p(x), so classifiers may fit corrupted labels and learn an improper map from inputs to true labels.
  • Label noise: DIW can estimate p(˜y | y, x) because its validation data carry information about the clean-label distribution.

4 Discussions

The discussion distinguishes importance weighting from learning to reweight, distributionally robust learning, and domain adaptation by their targets, assumptions, and treatment of shifted distributions. These differences determine whether methods estimate a fixed ratio, optimize validation risk, guard against worst-case shifts, or transfer representations across domains.

  • Learning to reweight: Learning to reweight alternates weighted training classification with unweighted validation classification, optimizing limited validation risk rather than estimating a specific target weight.
  • Distributionally robust learning: Distributionally robust learning assumes no target validation data and instead considers worst-case shifts within a prespecified uncertainty set.
  • Importance weighting versus robust learning: Importance weighting treats the test distribution as fixed and the training distribution as shifted, whereas distributionally robust learning reverses that viewpoint.
  • Domain adaptation: Domain adaptation calls training and test distributions source and target domains, with unsupervised domain adaptation using target features rather than target labels.

5 Experiments

Experiments on Fashion-MNIST, CIFAR-10, and CIFAR-100 evaluate DIW under label noise and class-prior shift using several baselines and ablations. DIW outperforms baselines, learns more accurate weights, and shows denoising and representation-clustering effects, with a CIFAR-100 applicability caveat for “-F” variants.

  • Setup: Experiments use Fashion-MNIST, CIFAR-10, and CIFAR-100, comparing DIW with Clean, Uniform, Random, IW, and Reweight baselines.
  • Label-noise experiments: Under label noise, DIW outperforms baselines and remains reasonably robust as noise increases, while baselines tend to overfit noisy labels.
  • Weight analysis: DIW identifies intact and mislabeled training data and automatically up-weights or down-weights them, unlike competing methods that do not effectively separate them.
  • Class-prior-shift experiments: Under class-prior shift, DIW outperforms baselines, and its learned weights are closer to the true weights according to MAE and RMSE comparisons.
  • Ablation study: Ablations show that dynamic weights improve over static weighting, and updating pretrained feature extractors is usually better than fixing them.
  • Representation visualization: DIW methods produce more concentrated embedded-data clusters on noisy CIFAR-10 than IW and SIW.
  • Denoising analysis: DIW2-L simultaneously fits intact data and denoises mislabeled data, yielding lower accuracy on flipped labels than on withheld ground-truth labels.
  • Limitation: “-F” DIW methods are not applicable on CIFAR-100 because mini-batches contain too few data from a class.

6 Conclusions

The paper reframes importance weighting for deep learning as a circular dependency and proposes dynamic importance weighting to resolve it. Experiments on label-noise and class-prior-shift settings demonstrate its effectiveness.

  • Importance weighting under distribution shift suffers from a conceptual and theoretical circular dependency.
  • Dynamic importance weighting iterates between weight estimation and weighted classification.Features for weight estimation can come from hidden-layer outputs or loss values.
  • Experiments on label-noise and class-prior-shift shifts demonstrate the effectiveness of dynamic importance weighting.

7 Broader impact

Distribution shift is widespread and can threaten machine-learning applications, including autonomous driving and fairness-sensitive social analysis. The work aims to mitigate this shift with dynamic importance weighting and reports robust classifiers across different shift forms.

  • Distribution shift arises from causes including subjective data-collection bias and non-stationary environments.
  • Distribution shift can create safety threats in autonomous driving and fairness issues in social-science applications.
  • The proposed dynamic importance weighting framework aims to learn robust classifiers under different forms of distribution shift.The paper also studies the framework theoretically and algorithmically.

Supplementary Material

The supplementary material documents experimental setup, datasets, and base models. It covers PyTorch implementation details for Fashion-MNIST, CIFAR-10, and CIFAR-100 experiments.

  • All supplementary experiments are implemented using PyTorch 1.6.0.
  • Datasets: Fashion-MNIST contains 60,000 training and 10,000 test 28*28 grayscale images across 10 classes.
  • Base models: Fashion-MNIST uses a LeNet-5 base model with convolutional, pooling, and fully connected layers.
  • Datasets: CIFAR-10 has 50,000 training and 10,000 test RGB images in 10 classes, while CIFAR-100 has 100 classes with 600 images per class.
  • Base models: CIFAR-10 and CIFAR-100 use ResNet-32 as their base model.

A.2 Label-noise experiments

The label-noise experiments define pair-flip and symmetric-flip corruption through transition matrices and specify optimization and kernel settings. The transition matrix and noise rate are unknown to the model.

  • Noise construction: Noisy labels are generated from a transition matrix T, where T_ij = P(ỹ = j|y = i).
  • Noise construction: Pair-flip noise changes label j to class (j mod k + 1) with probability η, whereas symmetric noise distributes flips across other classes.
  • Assumptions: The label-noise transition matrix and noise rate are unknown to the model.
  • Noise construction: Figure 7 presents the transition matrices for pair-flip and symmetric-flip label noise.
  • Optimization: Fashion-MNIST uses SGD, while CIFAR-10 and CIFAR-100 use Adam with dataset-specific weight decay and learning-rate schedules.The distribution-matching step uses an RBF kernel, with weight upper bounds of 50 for Fashion-MNIST and 10 for CIFAR-10/100.

A.3 Class-prior-shift experiments

The class-prior-shift experiments specify dataset construction, optimization settings, and several static or dynamic importance-weighting variants. These variants differ in whether weight estimation uses original data, pretrained features, loss values, or the jointly trained classifier.

  • Experimental setup: The Fashion-MNIST setup samples 10 validation examples per class, uses 4,000 training examples per majority class, and determines minority-class counts from ρ.Majority and minority classes are randomly selected, with classes 8 and 9 used in the described setup.
  • Experimental setup: Class-prior-shift training uses SGD with weight decay 1e-5, an initial learning rate of 0.0005, and batch sizes 256 for training and 100 for validation.The learning rate decays each epoch by multiplying by 0.993; baseline Clean and IW use different initial learning rates.
  • Implementation: The implementation uses an RBF kernel with γ set to the 99th distance quantile, regularizes the kernel matrix by 1e-05, and caps weights at 100.These details apply to the distribution-matching step.
  • Static importance weighting: SIW estimates fixed weights once, either on original data or on transformations from a pretrained feature extractor, before weighted classification.SIW-F uses hidden-layer outputs, while SIW-L uses loss-value transformations and does not require label-based partitioning.
  • Dynamic importance weighting: DIW1 estimates weights for each mini-batch using a pretrained, fixed feature extractor, then trains another classifier with those weights.DIW1-F uses hidden-layer outputs and DIW1-L uses loss values; the weights remain dynamic because mini-batches are randomly selected.
  • Dynamic importance weighting: DIW2 estimates weights within each mini-batch using a randomly initialized feature extractor in the same deep classifier that receives weighted training.This couples weight estimation and weighted classification seamlessly; DIW3 adds limited pretraining to the DIW2 feature extractor.

B Supplementary experimental results

The supplementary section reports classification accuracy under label noise and examines learned importance-weight distributions on CIFAR-10. Its figures and table compare the behavior of IW-related methods across datasets and noise settings.

  • Classification results: Supplementary results include mean accuracy and standard deviation on Fashion-MNIST, CIFAR-10, and CIFAR-100 under label noise.Table 4 reports five trials and corresponds to Figure 3.
  • Weight distributions: DIW successfully identifies intact and mislabeled CIFAR-10 training data under 0.3 pair flips and 0.5 symmetric flips, up- or down-weighting them accordingly.Figure 9 compares DIW with reweight and IW through the distributions of learned importance weights.
  • Method illustrations: Figure 8 illustrates IW, SIW, and DIW, including DIW1’s seamless combination of weight estimation and weighted classification.The figure provides visual comparison of the method designs rather than an additional quantitative result.
Loading 2006.04662v2…