PaperlayerView exact revision on arXiv

Source-linked AI summary

Style Normalization and Restitution for Domain Generalization and Adaptation

Xin Jin, Cuiling Lan, Wenjun Zeng, Zhibo Chen

arXiv:2101.00588v3cs.CV

TL;DR

Models can lose performance in unseen environments when image styles differ from training data. SNR combines instance normalization with task-relevant feature restitution and a dual loss; experiments report improvements for domain generalization and adaptation across vision tasks.

  • Problem

    Models that perform well on training-like data can degrade in unseen environments where image styles differ from the training data.

    Source: §I · p. 1

  • Method

    SNR uses Instance Normalization to reduce style variation, restores task-relevant features from the removed residual, and applies a dual restitution loss.

    Source: §I · p. 2

  • Results

    Experiments report improved domain generalization and adaptation, with SNR-powered schemes achieving state-of-the-art performance across classification, semantic segmentation, and object detection.

    Source: §I · p. 2, §V · p. 12

  • Takeaways & Limitations

    SNR is presented as a generic plug-and-play module for existing networks and multiple computer vision tasks.

    Source: §V · p. 12

Abstract

from arXiv · show

For many practical computer vision applications, the learned models usually have high performance on the datasets used for training but suffer from significant performance degradation when deployed in new environments, where there are usually style differences between the training images and the testing images. An effective domain generalizable model is expected to be able to learn feature representations that are both generalizable and discriminative. In this paper, we design a novel Style Normalization and Restitution module (SNR) to simultaneously ensure both high generalization and discrimination capability of the networks. In the SNR module, particularly, we filter out the style variations (e.g, illumination, color contrast) by performing Instance Normalization (IN) to obtain style normalized features, where the discrepancy among different samples and domains is reduced. However, such a process is task-ignorant and inevitably removes some task-relevant discriminative information, which could hurt the performance. To remedy this, we propose to distill task-relevant discriminative features from the residual (i.e, the difference between the original feature and the style normalized feature) and add them back to the network to ensure high discrimination. Moreover, for better disentanglement, we enforce a dual causality loss constraint in the restitution step to encourage the better separation of task-relevant and task-irrelevant features. We validate the effectiveness of our SNR on different computer vision tasks, including classification, semantic segmentation, and object detection. Experiments demonstrate that our SNR module is capable of improving the performance of networks for domain generalization (DG) and unsupervised domain adaptation (UDA) on many tasks. Code are available at https://github.com/microsoft/SNR.

I. INTRODUCTION

Models trained on one dataset can lose performance in unseen environments, where style differences create train–test domain gaps. The paper proposes SNR to reduce style variation while restoring task-relevant discrimination, and evaluates it for domain generalization and adaptation.

I. INTRODUCTION

  • Style differences between training and testing images create domain gaps, and models can degrade when applied in unseen environments.The paper describes variations in illumination, color contrast, saturation, quality, and imaging style.Source: §I · p. 1
  • SNR applies Instance Normalization to reduce style variation, then distills task-relevant discriminative information from the removed residual and adds it back.The residual is the difference between the original feature and its instance-normalized version.Source: §I · p. 2
  • A dual restitution loss constraint encourages better separation of task-relevant features from residual information.The constraint is designed to support both generalization and discrimination.Source: §I · p. 2
  • Experiments show that SNR improves network generalization and existing unsupervised domain adaptation methods across the paper’s evaluated vision tasks.The work extends an earlier person re-identification design to classification, detection, and semantic segmentation.Source: §I · p. 2

II. RELATED WORK

Prior work addresses domain gaps through feature alignment, normalization, or disentanglement, while the paper presents SNR as a module for preserving both generalization and discrimination. Its restitution step restores task-relevant information removed by normalization.

A. Domain Generalization (DG)

  • DG work explores domain alignment, optimization strategies, and style regularization with Instance Normalization, which can remove task-relevant discriminative information.The paper contrasts SNR’s use of IN across channels with approaches that mix or select IN and Batch Normalization.Source: A. Domain Generalization (DG) · p. 3

B. Unsupervised Domain Adaptation (UDA)

  • UDA methods align source and target features using distribution distances or adversarial learning; SNR is proposed to enhance existing UDA approaches while preserving discrimination.The paper notes that domain alignment can reduce domain-specific variation while also losing discriminative information.Source: B. Unsupervised Domain Adaptation (UDA) · p. 3

C. Feature Disentanglement

  • SNR separates task-specific features from removed residual information, using a dual restitution loss to encourage sharper class likelihood after restitution.The stated aim is less ambiguity for a sample.Source: C. Feature Disentanglement · p. 3

III. STYLE NORMALIZATION AND RESTITUTION

  • SNR is a plug-and-play module that applies IN, restores task-relevant features from the residual, and uses a dual loss to separate relevant from irrelevant information.It can be inserted into existing classification, detection, and segmentation networks; the paper describes task-specific variations to the loss forms.Source: §III · pp. 3–4

A. Style Normalization and Restitution Module

SNR first applies Instance Normalization to reduce style discrepancy, then restores task-relevant features from the information normalization removes. Channel attention separates the residual into task-relevant and task-irrelevant components.

1) Style Normalization to Reduce Domain Discrepancy:

  • Instance Normalization normalizes spatial statistics independently for each sample and channel, filtering some instance-specific style information while preserving feature content.The normalization uses learned γ and β parameters.Source: 1) Style Normalization to Reduce… · p. 4

2) Feature Restitution to Preserve Discrimination:

  • The residual between the original and normalized features is split into task-relevant and task-irrelevant components, and the task-relevant component is added back to the normalized feature.A learned channel-attention vector adaptively masks the residual; its complement produces the task-irrelevant component.Source: 2) Feature Restitution to Preserve… · p. 4
  • Using one channel-attention vector and its complement, rather than two independent attention modules, facilitates disentanglement.The channel-level design is consistent with Instance Normalization, which normalizes independently across spatial positions for each channel.Source: 2) Feature Restitution to Preserve… · p. 4

3) Dual Restitution Loss Constraint:

The dual restitution loss encourages task-relevant features to increase discrimination and task-irrelevant features to reduce it. Its feature-vector calculation varies with the task.

3) Dual Restitution Loss Constraint:

  • The task-specific feature vectors use whole-map pooling for classification, per-pixel entropy for segmentation, and ground-truth-box pooling for detection.The figure illustrates these alternatives for restitution-loss optimization.Source: Figure 3 · p. 5
  • For classification, pooled enhanced, contaminated, and normalized features pass through a classifier and softmax, and their prediction entropies are compared.Softplus avoids negative loss values and is used to reduce optimization difficulty.Source: 3) Dual Restitution Loss Constraint · p. 5
  • The dual restitution loss promotes better disentanglement by encouraging enhanced features to be more discriminative and contaminated features to be less discriminative than normalized features.The respective changes correspond to lower and higher entropy of predicted class likelihoods.Source: 3) Dual Restitution Loss Constraint · pp. 4–5

B. Applications, Extensions, and Variants

SNR is presented as a plug-and-play module for different networks and computer-vision tasks. Its restitution-loss feature-vector calculation adapts to classification and segmentation.

B. Applications, Extensions, and Variants

  • SNR is a plug-and-play module intended to improve network generalization and discrimination for domain generalization and adaptation across classification, segmentation, and detection.Source: B. Applications, Extensions, and… · p. 5

1) Classification:

  • For classification, SNR can be inserted after convolution blocks, and spatially averaged enhanced, contaminated, and normalized features provide vectors for the dual restitution loss.Source: 1) Classification · p. 5

2) Segmentation:

  • For segmentation, restitution loss uses entropy at each spatial position and averages entropy across pixels, rather than pooling features spatially first.The authors report that averaging pixel entropies is slightly better in performance and uses less computation than calculating restitution loss for each pixel.Source: 2) Segmentation · p. 5

3) Detection:

The supplied passages describe detection as region-wise classification for restitution-loss calculation and report classification results on PACS and Office-Home. SNR is compared with a source-domain aggregate baseline and other DG methods.

3) Detection:

  • For detection, entropy is calculated for each ground-truth bounding-box region using features spatially averaged within that box, then averaged across object regions.Source: 3) Detection · pp. 5–6

IV. EXPERIMENT

  • The experiments assess SNR under domain generalization and adaptation for classification, segmentation, and detection.Source: §IV · p. 6

1) Datasets and Implementation Details:

2) Results on Domain Generalization:

3) Results on Unsupervised Domain Adaptation:

SNR improves UDA classification over M3SDA across Digit-Five and DomainNet, while DG ablations show that restitution and the dual restitution loss contribute to performance. SNR reduces style discrepancy but does not replace explicit source–target alignment.

3) Results on Unsupervised Domain Adaptation:

  • Compared with Baseline (M3SDA), which uses ’s aligning technique for domain adaptation, adding SNR (SNR-M3SDA) yields significant accuracy gains of 7.99% on Digit-Five, 3.04% on mini-DomainNet, and 4.0% on full DomainNet.Source: 3) Results on Unsupervised Domain… · p. 6

4) Ablation Study:

  • SNR outperforms AGG by 2.3% on PACS and 1.4% on Office-Home, and outperforms AGG-IN by 1.7% and 0.9%, respectively.It also exceeds AGG-BIN* by 1.2% on PACS and 0.5% on Office-Home; the authors conclude style normalization alone is insufficient and restitution is critical.Source: 4) Ablation Study · p. 7, Table IV · p. 7
  • The dual restitution loss improves SNR over SNR without L_SNR by 1.0% on PACS and 0.8% on Office-Home.The comparison-based entropy strategy also exceeds SNR without comparison by 0.6% on PACS and 0.7% on Office-Home.Source: 4) Ablation Study · p. 8, Table V · p. 8
  • SNR-UDA exceeds SNR by 0.94% in average accuracy on Digital-Five, while adding SNR to M3SDA brings a 7.99% gain.SNR-UDA uses unlabeled target-domain data; SNR does not, and the text says explicit alignment such as M3SDA remains useful.Source: 4) Ablation Study · p. 8, Table VI · p. 8
  • SNR reduces source–target style discrepancy but does not explicitly interact across domains after restitution, so explicit alignment remains useful for UDA.The authors identify M3SDA as an example of such alignment.Source: 4) Ablation Study · p. 8

5) Design Choices of SNR:

SNR performs best when inserted at all four ResNet18 stages, and its channel-attention design separates task-relevant from task-irrelevant residual features. Visualizations show more consistent activations across styles and better class separation in UDA.

5) Design Choices of SNR:

  • Adding SNR at all four ResNet18 stages achieves the best performance, while insertion at any individual stage improves on Baseline (AGG).The reported gains are not sensitive to the insertion position.Source: 5) Design Choices of SNR · p. 8, Table VII · p. 8
  • SNR outperforms SNR_conv by 1.3% and SNR_g(·)2 by 0.9% on average on PACS; SNR-SC exceeds channel-only SNR by 0.3%.The authors use channel attention by default for simplicity and to align with distilling removed task-relevant information.Source: 5) Design Choices of SNR · pp. 8–9, Table VIII · p. 9

6) Visualization:

  • SNR’s enhanced maps respond strongly to object regions, while its contaminated maps respond mainly to background.Across varied input styles, SNR activation maps are more consistent than Baseline (AGG) maps.Source: 6) Visualization · p. 9, Figure 4 · p. 9
  • On Digit-Five UDA classification, SNR features are better separated by class than Baseline (M3SDA) features in the t-SNE visualization.The visualization compares SNR-M3SDA with Baseline (M3SDA).Source: 6) Visualization · p. 9, Figure 5 · p. 9

B. Semantic Segmentation

For semantic segmentation, SNR improves domain generalization and unsupervised domain adaptation on transfers to Cityscapes. Qualitative comparisons also show gains from adding SNR and better segmentation with adaptation than with domain generalization.

1) Datasets and Implementation Details:

2) Results on Domain Generalization:

3) Results on Unsupervised Domain Adaptation:

4) Visualization of DG and UDA Results:

C. Object Detection

Object detection experiments evaluate SNR for domain generalization and unsupervised domain adaptation across weather and cross-dataset shifts. SNR improves detection performance in the reported settings.

1) Datasets and Implementation Details:

  • The experiments use Cityscapes, Foggy Cityscapes, and KITTI; DG trains Faster R-CNN on source data, while UDA uses DA Faster R-CNN.For DG, SNR modules are inserted after the first four ResNet-50 convolutional blocks; evaluation reports mAP at a 0.5 threshold.Source: 1) Datasets and Implementation Details · p. 11

2) Results on DG and UDA:

3) Qualitative Results:

In qualitative UDA detection results, SNR corrects false positives and detects cars missed by DA Faster R-CNN. The paper also describes SNR as a plug-and-play approach for learning generalizable and discriminative representations across computer-vision tasks.

3) Qualitative Results:

D. Complexity Analysis

  • For ResNet-50, SNR increases model size by 2.2% (24.56M vs. 25.12M) and computational complexity by 5.1% (3.87G vs. 4.08G FLOPs).The analysis inserts SNR after each convolutional block in the first four backbone blocks.Source: D. Complexity Analysis · p. 12

V. CONCLUSION

  • SNR uses Instance Normalization to reduce style variation, then restitutes task-relevant discriminative features from the residual and applies a dual restitution loss to promote feature disentanglement.The conclusion reports effectiveness for DG and adaptation across classification, semantic segmentation, and object detection.Source: §V · p. 12

Appendix

The appendix describes datasets and evaluation setups for object classification, semantic segmentation, and object detection. Classification experiments use leave-one-domain-out protocols, while the segmentation and detection sections specify their datasets and transfer settings.

A. Object Classification

  • Classification evaluation covers DG on PACS and Office-Home and adaptation on Digit-5 and DomainNet, using leave-one-domain-out protocols.A mini-DomainNet subset is used for ablations, with full DomainNet used for state-of-the-art comparisons.Source: A. Object Classification · p. 13

B. Semantic Segmentation

  • Segmentation uses GTA5 or SYNTHIA as source and Cityscapes validation as the test set, evaluating mIoU across 19 or 16 shared classes, respectively.The datasets include Cityscapes, GTA5, and the 9,400-image SYNTHIA-RAND-CITYSCAPES subset.Source: B. Semantic Segmentation · p. 13

C. Object Detection

  • Detection uses Cityscapes, Foggy Cityscapes at the highest fog intensity, and KITTI, with KITTI serving as a source or DG target in the stated settings.When KITTI is the target test set for DG, the entire dataset is used.Source: C. Object Detection · pp. 13–14
Loading 2101.00588v3…