Uncertainty Modeling for Out-of-Distribution Generalization

Xiaotong Li; Yongxing Dai; Yixiao Ge; Jun Liu; Ying Shan; Ling-Yu Duan

Source-linked AI summary

Uncertainty Modeling for Out-of-Distribution Generalization

Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, Ling-Yu Duan

arXiv:2202.03958v2Feb 8, 2022cs.CVcs.LG

TL;DR

Out-of-distribution shifts expose deep networks’ reliance on stable feature statistics and can reduce performance on unseen domains. DSU models feature-statistic uncertainty with Gaussian sampling during training, and experiments report improved generalization across multiple vision tasks without additional parameters. The paper concludes that synthesized uncertain statistics improve robustness to diverse domain shifts.

Problem
Deep networks often treat domain-characterizing feature means and standard deviations deterministically, leaving uncertain testing-time statistic discrepancies insufficiently modeled.
Method
DSU models each feature statistic as a multivariate Gaussian and randomly samples synthesized statistics during training to represent diverse domain shifts.
Results
DSU consistently improves network generalization across image classification, semantic segmentation, and instance retrieval, while requiring no additional model parameters or loss constraints.
Takeaways & Limitations
Introducing uncertainty into feature statistics improves model robustness against diverse domain shifts within the evaluated vision tasks.
Takeaways & Limitations
The method assumes that uncertain feature statistics follow multivariate Gaussian distributions.

Abstract

from arXiv · show

Though remarkable progress has been achieved in various vision tasks, deep neural networks still suffer obvious performance degradation when tested in out-of-distribution scenarios. We argue that the feature statistics (mean and standard deviation), which carry the domain characteristics of the training data, can be properly manipulated to improve the generalization ability of deep learning models. Common methods often consider the feature statistics as deterministic values measured from the learned features and do not explicitly consider the uncertain statistics discrepancy caused by potential domain shifts during testing. In this paper, we improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training. Specifically, we hypothesize that the feature statistic, after considering the potential uncertainties, follows a multivariate Gaussian distribution. Hence, each feature statistic is no longer a deterministic value, but a probabilistic point with diverse distribution possibilities. With the uncertain feature statistics, the models can be trained to alleviate the domain perturbations and achieve better robustness against potential domain shifts. Our method can be readily integrated into networks without additional parameters. Extensive experiments demonstrate that our proposed method consistently improves the network generalization ability on multiple vision tasks, including image classification, semantic segmentation, and instance retrieval. The code can be available at https://github.com/lixiaotong97/DSU.

1 INTRODUCTION

Out-of-distribution deployment violates the shared-distribution assumption behind deep networks, exposing their vulnerability to domain-specific feature-statistic shifts. DSU models those shifts probabilistically during training to improve robustness and generalization without adding parameters or loss constraints.

Motivation: Out-of-distribution deployment can cause substantial performance drops because training and testing domains often do not follow the same distribution.Examples include segmentation from sunny to rainy or foggy environments and recognition from photographs to art paintings.
Motivation: Feature statistics, specifically means and standard deviations, carry domain characteristics such as photo style and capturing environment information.These statistics are moments of learned features and are generally inconsistent across domains with different data distributions.
Motivation: Models that treat feature statistics deterministically do not explicitly account for uncertain testing-time discrepancies and can overfit the training domain.This leaves learned representations vulnerable to statistic changes and limits their generalization ability.
Approach: DSU models domain shifts with uncertainty by synthesizing feature-statistic variants whose shifts can differ in direction and intensity.The method hypothesizes that feature statistics follow multivariate Gaussian distributions rather than being deterministic points.
Outcome: The method consistently improves generalization across image classification, semantic segmentation, and instance retrieval without additional model parameters or loss constraints.The authors report that uncertain feature statistics alleviate performance drops caused by domain shifts.

2 RELATED WORK

Related work improves domain generalization through data augmentation, invariant representation learning, and learning strategies. DSU belongs to feature-statistics augmentation while explicitly modeling uncertainty in domain shifts.

Domain Generalization: Domain generalization seeks out-of-distribution performance on unseen target domains using single or multiple source domains for training.The literature includes many approaches addressing this setting.
Data Augmentation: Image- and feature-level augmentation methods improve generalization by transforming inputs, labels, features, or feature statistics.Examples include AugMix, CutMix, mixup, Manifold Mixup, MixStyle, and pAdaIn.
Invariant Representation Learning: Invariant representation learning targets features that remain invariant to domain shifts through distribution alignment or disentanglement of domain-specific and domain-invariant components.These approaches differ in whether they align entire distributions or separate feature roles.
Learning Strategies: Ensemble learning and meta-learning are learning strategies used to improve generalization and robustness by combining diverse predictors or simulating domain shifts.The cited methods operate through model or module diversity and learned domain-shift simulation.
Uncertainty Modeling: Uncertainty modeling has been studied in representation learning through generative models such as variational auto-encoders and Bayesian interpretations of dropout.These methods address uncertainty in hidden representations or model parameters, providing context for DSU’s uncertainty focus.

3 METHOD

DSU represents feature means and standard deviations as uncertain distributions, estimates channel-wise variation from mini-batch statistics, and samples replacements during training. The resulting plug-and-play module models diverse domain shifts while remaining inactive at test time.

Preliminaries: For intermediate features x, DSU computes channel-wise means and standard deviations for each instance in a mini-batch.These statistics provide the quantities whose domain shifts are modeled.
Problem Formulation: Feature statistics encode domain characteristics, but their deterministic treatment leaves models exposed to uncertain shifts in direction and intensity.Such inconsistencies can be ill-suited to network nonlinearities and normalization layers.
Probabilistic Modeling: DSU hypothesizes that each feature statistic follows a multivariate Gaussian distribution centered on its original value.The distribution’s standard deviation represents the uncertainty scope for potential shifts.
Uncertainty Estimation: Uncertainty scope is estimated non-parametrically from mini-batch feature-statistic variances to provide a meaningful variation range for each channel.The estimated magnitudes indicate which channels may potentially change.
Probabilistic Sampling: Random sampling generates feature-statistic variants with different directions and intensities, using a differentiable re-parameterization operation.These variants are then used to replace the original statistics in the feature transformation.
Implementation: The AdaIN-based module can be inserted at multiple network positions, applied during training with probability p, and discarded during testing.It is designed as a flexible training-time operation rather than a test-time component.

4 EXPERIMENTS

Experiments evaluate DSU across classification, semantic segmentation, instance retrieval, and corruption robustness under varied distribution shifts. The method improves generalization across these tasks, including substantial gains on PACS and synthetic-to-real segmentation.

Experiments cover image classification, semantic segmentation, instance retrieval, and robustness to corruptions under style, synthetic-to-real, scene, and pixel-level shifts.The evaluation uses PACS, GTA5-to-Cityscapes, DukeMTMC and Market1501, and ImageNet-C settings.
Nearly 10% average-accuracy improvement occurs over the baseline in PACS Art and Sketch domains.The method also outperforms competing methods, while Photo shows a slight drop attributed to ImageNet pretraining.
6.1% mIOU and 5.5% mAcc improvements are reported for semantic segmentation from synthetic GTA5 to real Cityscapes.The synthesized statistics remain centered on the originals with mild perturbations to preserve detailed information in the dense prediction task.
DSU achieves large gains over the baseline and substantial margins over MixStyle and pAdaIN on fine-grained person re-identification.The method is described as preserving original person information while offering more feature-statistic variants.

5 ABLATION STUDY

The ablation study examines module placement, augmentation probability, and uncertainty distributions. Results support using positions 0–5, probability p=0.5, and uncertainty estimation with a Gaussian distribution.

5 ABLATION STUDY: The ablation study evaluates insertion positions, augmentation probability, and alternative uncertainty distributions on PACS and GTA5-to-Cityscapes segmentation.The experiments use ResNet models.
Effects of Different Inserted Positions: DSU performance is consistently higher than baseline across insertion positions 0–5, with all-stage insertion producing better performance.The authors use positions 0–5 in all experiments.
Effects of Hyper-parameter: The probability p controls feature-statistics augmentation strength, with accuracy reaching its best result at p=0.5.Results are not sensitive to the probability setting, and p=0.5 is used by default.
Choices of Uncertainty Distribution: Gaussian uncertainty estimation outperforms the compared random and uniform distributions for synthesizing feature-statistics shifts.Random uses shifts from N(0, 1), while Uniform samples from U(−Σ, Σ); improper variant scopes can harm performance.

6 FURTHER ANALYSIS

Further analysis examines feature-statistics shifts and the diverse changes produced by the probabilistic method. The analyses focus on unseen-domain behavior and synthetic reconstructions.

Feature-Statistics Shifts: Feature statistics from the baseline model show an obvious shift between training and unseen testing domains under different data distributions.The analysis measures average statistics from intermediate ResNet18 features, using Art Painting as the unseen PACS domain.
Synthetic Changes: The method produces diverse synthetic changes in reconstructed images when inserted into an autoencoder encoder.A predefined autoencoder reconstructs synthetic images from the altered feature representations.

7 CONCLUSIONS

The paper proposes modeling uncertain domain shifts with synthesized feature statistics during training. It hypothesizes multivariate Gaussian statistics and reports improved generalization and robustness to diverse shifts.

7 CONCLUSIONS: The method models uncertainty in domain shifts by synthesizing feature statistics during training, with each statistic following a multivariate Gaussian distribution.The generated statistics provide diverse distribution possibilities for training.
7 CONCLUSIONS: Experiments demonstrate that the probabilistic approach improves network generalization ability.

A.1 ALGORITHM

Algorithm 1 applies the module probabilistically during training to intermediate features. It computes feature statistics, estimates uncertainty, samples synthetic statistics, and returns an altered feature or skips the module.

Inputs and Control: The algorithm takes an intermediate feature x and probability p, and outputs an intermediate feature bx after considering potential statistics shifts.The module is conditionally forwarded during training.
Statistics Processing: The module computes channel-wise mean and standard deviation for each mini-batch instance before estimating uncertainty in the feature statistics.
Statistics Synthesis: Synthetic feature statistics are generated by randomly sampling from the specified Gaussian distributions.The algorithm then obtains the feature after considering the potential statistics shifts.
Output: The algorithm returns the feature bx with uncertain feature statistics, or retains the original feature x when the module is skipped.

A.2 MULTI-DOMAIN GENERALIZATION ON OFFICE HOME.

On Office-Home, the proposed method improves over the baseline and competing methods, with especially large gains on the stylistically distinct Clipart domain.

The proposed method improves Office-Home performance over the baseline and competing methods.The experiment uses a ResNet18 backbone on 15,500 images across 65 classes.

A.3 CHOICE OF UNCERTAINTY DISTRIBUTION

Fixed Gaussian uncertainty choices underperform the proposed adaptive estimation, while the method also improves unseen-domain feature compactness and preserves or improves in-domain performance. Its Gaussian sampling generates more diverse statistic variants than pairwise methods, and gains remain consistent across batch sizes.

Choice of uncertainty distribution: Fixed Gaussian distributions perform substantially worse than the proposed uncertainty estimation.Excessive uncertainty can harm training, and the best fixed value varies across tasks; the proposed method adapts without manual adjustment.
Feature representations: The proposed method makes same-category features more compact in an unseen domain, supporting more invariant representations.The paper attributes this to alleviating domain perturbations during training and focusing on content information.
Comparisons to related methods: Unlike pairwise methods, the Gaussian-based approach generates statistic variants with diverse directions and intensities, including variants beyond the training-domain scope.Pairwise methods are constrained by the selected reference sample and inner interpolation between training samples.
Within-dataset performance: 23.4% top-1 error is achieved on ImageNet after DSU training, down from 23.8% on the Clean setting.The reported result indicates no sacrifice in within-dataset performance.
Within-dataset performance: Within-dataset performance on PACS slightly beats the baseline, consistent with the ImageNet result.The PACS average follows the multi-source training protocol across Art, Cartoon, Photo, and Sketch domains.
Batch-size ablation: Consistent performance gains are observed across various batch sizes on PACS, with batch size 64 used for fair comparison.The batch-size ablation follows the original PACS setting.