PaperlayerView exact revision on arXiv

Source-linked AI summary

RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening

Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne Kim, Seungryong Kim, Jaegul Choo

arXiv:2103.15597v2cs.CV

TL;DR

Deep networks often generalize poorly from a source dataset to unseen domains, while domain adaptation requires target-domain data and many DG methods require costly multi-domain datasets. RobustNet introduces an instance selective whitening loss that suppresses style-sensitive feature covariances while preserving discriminative content. Experiments in urban-scene segmentation show consistent DG improvements across backbone networks and superiority to existing approaches, with negligible computational cost.

  • Problem

    Deep neural networks can fail on unseen domains because limited training data cannot model diverse real-world conditions, while existing adaptation or multi-domain approaches impose data-access or collection limitations.

  • Method

    RobustNet uses an instance selective whitening loss to disentangle style- and content-related feature covariances and suppress only domain-specific style information.

  • Results

    RobustNet consistently boosts domain-generalization performance in urban-scene segmentation across several backbone networks and outperforms existing approaches qualitatively and quantitatively.

  • Takeaways & Limitations

    Selective covariance whitening provides a simple, broadly applicable way to improve robustness to unseen domains without additional computational cost.

  • Takeaways & Limitations

    Whitening all covariance elements can diminish feature discrimination and distort object boundaries because covariance encodes both domain-specific style and domain-invariant content.

Abstract

from arXiv · show

Enhancing the generalization capability of deep neural networks to unseen domains is crucial for safety-critical applications in the real world such as autonomous driving. To address this issue, this paper proposes a novel instance selective whitening loss to improve the robustness of the segmentation networks for unseen domains. Our approach disentangles the domain-specific style and domain-invariant content encoded in higher-order statistics (i.e., feature covariance) of the feature representations and selectively removes only the style information causing domain shift. As shown in Fig. 1, our method provides reasonable predictions for (a) low-illuminated, (b) rainy, and (c) unseen structures. These types of images are not included in the training dataset, where the baseline shows a significant performance drop, contrary to ours. Being simple yet effective, our approach improves the robustness of various backbone networks without additional computational cost. We conduct extensive experiments in urban-scene segmentation and show the superiority of our approach to existing work. Our code is available at https://github.com/shachoi/RobustNet.

1. Introduction

Domain generalization targets robust performance on arbitrary unseen domains without requiring target-domain data or costly multi-domain datasets. RobustNet addresses this challenge with an instance selective whitening loss that suppresses domain-specific covariance while preserving discriminative content.

  • Motivation: Domain shifts from illumination, weather, and location differences can cause DNNs trained on one dataset to fail on unseen real-world data.The paper frames this as especially important for safety-critical applications such as autonomous driving.
  • Motivation: Domain adaptation requires target-domain samples, limiting its applicability when the entire real world is treated as the target domain.Limited training data cannot fully model the target distribution.
  • Motivation: Domain generalization improves robustness to arbitrary unseen domains, but common methods rely on costly multi-domain datasets and depend on the number of source datasets.This motivates methods that can generalize without collecting many source domains.
  • Motivation: Instance normalization supports simple single-source domain generalization but ignores channel correlations, although feature covariance can encode domain-specific style.Whitening can remove style information, but applying it indiscriminately may also affect content.
  • Contribution: RobustNet proposes an instance selective whitening loss that removes style-sensitive covariances while retaining discriminative feature information.The method implicitly encourages whitening through a loss rather than using an explicit closed-form transformation, with negligible computational cost.
  • Contribution: The proposed loss disentangles domain-specific and domain-invariant properties in higher-order feature statistics and improves urban-scene segmentation generalization across several backbone networks.The paper reports superiority over existing approaches qualitatively and quantitatively.

2. Related Work

Related work contrasts domain adaptation, which accesses target domains, with domain generalization, which must handle unseen domains without target-domain access. In semantic segmentation, DG remains relatively underexplored despite practical variation in weather, illumination, and location.

  • Domain adaptation and generalization: Domain adaptation transfers knowledge from labeled source data to target domains with scarce labels, whereas domain generalization assumes no target-domain access during training.DG aims to improve performance on unseen domains under this restriction.
  • Semantic segmentation in DG: Semantic-segmentation DG has received less attention than image-classification DG, even though real-world segmentation encounters adverse weather, illumination, and location differences.The paper positions its work as addressing this practical gap.

3. Preliminaries

Whitening transforms intermediate features so channel variances become one and inter-channel covariances become zero, but standard whitening can remove both style and content. RobustNet therefore motivates selectively whitening covariance elements to preserve feature discrimination and object boundaries.

  • Whitening transformation: For an intermediate feature map X ∈ R^C×HW, whitening produces a feature map whose covariance-related product equals (HW)I.Here C is the channel count, while H and W are spatial dimensions.
  • Whitening transformation: The feature covariance matrix Σ_µ is defined from centered features using the mean vector µ.The covariance matrix captures relationships among feature channels.
  • Whitening transformation: Whitening matrices can be computed through eigendecomposition of the covariance matrix into eigenvectors Q and eigenvalues Λ.The inverse square root of the covariance matrix is then obtained from this decomposition.
  • Whitening transformation: Whitening can remove style information when applied instance-wise, as demonstrated in style transfer.This motivates using whitening for domain generalization.
  • Limitations of whitening: Analytic whitening is computationally expensive because eigendecomposition slows training and inference and prevents gradient back-propagation.Whitening losses and Newton iterations are alternatives that avoid eigendecomposition.
  • Limitations of whitening: Whitening all covariance elements can diminish feature discrimination and distort object boundaries because covariance simultaneously encodes style and content.This is the central limitation that motivates selective whitening.

4. Proposed Method

The method standardizes each instance, computes feature covariance, and selectively suppresses covariance elements associated with domain-specific style while preserving potentially discriminative content. It uses photometric transformations to identify style-sensitive covariances and applies the resulting loss during training without adding inference computation.

  • Instance Whitening Loss: Instance whitening first standardizes each feature map and computes its covariance so diagonal variances are fixed while off-diagonal correlations can be reduced.Standardization resolves the conflict between optimizing diagonal elements toward one and off-diagonal elements toward zero.
  • Network Architecture with Proposed ISW Loss: ISW is added to an instance-normalization-based architecture and uses training-only operations, so inference requires no additional computation.The approach follows the IBN-b architectural baseline and applies the proposed loss to instance normalization.
  • Margin-based Relaxation of Whitening Loss: The ordinary whitening loss can harm discriminative feature power by suppressing all covariance elements, motivating a margin-based relaxed loss.The relaxed loss permits a specified covariance margin and empirically performs better than the unmarginized instance whitening loss.
  • Separating Covariance Elements: The method estimates covariance sensitivity by comparing an image with a photometrically transformed version after an initial five-epoch training phase without whitening loss.Color jittering and Gaussian blurring simulate shifts in color and blurriness; covariance-element variances are then clustered to separate high- and low-sensitivity groups.
  • Separating Covariance Elements: The instance selective whitening loss masks and suppresses only covariances assumed to encode domain-specific style, rather than removing all covariance information.The method assumes high-variance covariance elements contain style and low-variance elements contain domain-invariant content.

5. Experiments

Experiments evaluate RobustNet for urban-scene segmentation under domain generalization, including single- and multi-source training, varied backbones, comparisons, qualitative analyses, and computational cost. ISW generally improves unseen-domain performance and preserves applicability without additional computational cost, though source-domain performance may decrease.

  • Experimental setup: Models are trained on source datasets and evaluated on other datasets, including Cityscapes, BDD-100K, Mapillary, GTAV, and SYNTHIA, to measure unseen-domain generalization.Experiments use DeepLabV3+ and evaluate several backbone networks, including ResNet, ShuffleNetV2, and MobileNetV2.
  • Effectiveness of ISW: ISW outperforms competing normalization methods on most target datasets, including BDD-100K, Mapillary, and SYNTHIA for models trained on Cityscapes.For models trained on GTAV, ISW outperforms other methods on all datasets except the GTAV source dataset.
  • Generalization behavior: Baseline, SW, and IBN-Net tend to overfit the source domain, whereas ISW may sacrifice source-domain performance while improving generalizability under large domain shifts.This trade-off is presented as relevant to deployment in environments where large domain shifts are expected.
  • Backbone and source-domain applicability: ISW improves performance across ShuffleNetV2 and MobileNetV2 backbones on real-world datasets compared with baseline and IBN-Net models.Additional experiments also evaluate training on multiple synthetic domains.
  • Comparison with existing DG methods: Compared with IBN-Net and DRPC, RobustNet achieves the largest average performance increase and particularly improves results on BDD-100K and Mapillary.These datasets contain more diverse driving scenes than Cityscapes.
  • Analysis and cost: The whitening transformation implemented by ISW adds no additional computational cost, while covariance visualizations and reconstructed images support selective suppression of style information.Reconstructed images retain segmentation-relevant content while suppressing illumination and color information.

6. Discussions

The discussion identifies two directions for improving ISW: recovering the original feature distribution after normalization and broadening the photometric transformations used to separate style from content.

  • Affine parameters: Adding affine parameters or a 1×1 convolution after normalization did not improve the method.The authors conjecture that these components lack sufficient complexity to recover the original distribution.
  • Photometric transformation: The method uses color transformation and Gaussian blur for photometric separation, while broader photometric augmentation remains a potential improvement.The authors report that these transformations did not harm content information.

7. Conclusions

The paper proposes ISW for urban-scene segmentation domain generalization by suppressing style-related covariance while retaining content-related information. It presents this focus as a practical contribution and a direction for further research.

  • Conclusion: ISW disentangles style- and content-related covariance in intermediate features and suppresses only the style-related components to learn domain-invariant representations.The conclusion frames this as the paper’s central contribution to domain generalization.
  • Conclusion: The work targets urban-scene segmentation domain generalization, which the authors describe as practically relevant for deployment in the wild but relatively understudied.The authors aim to encourage further research in this area.

A. Supplementary Material

The supplementary material adds quantitative, implementation, and qualitative analyses. On BDD-100K images with dust, water drops, night, and snow, the method makes reasonable predictions while the baseline completely fails.

  • The supplementary section examines hyper-parameter impacts, implementation details, and qualitative results.
  • On BDD-100K images, the method makes reasonable predictions while the baseline completely fails.The models are trained on Cityscapes.
  • The BDD-100K examples include windshield dust and water drops, plus night and snow conditions absent from Cityscapes.

A.1. Comparison with DA methods

The method is compared with domain adaptation approaches using mIoU gains from the baseline, while emphasizing their different access assumptions. Although not the top performer, it achieves comparable results without target-domain access.

  • The method shows comparable mIoU gains to several domain adaptation methods, although it is not the top performer.The comparison includes FCN Wild, CDA, DCAN, DTA, IBN-Net, and DRPC.
  • Unlike domain adaptation methods, the proposed approach targets arbitrary unseen domains without access to target-domain samples.The paper states that direct comparison under the same setting is impossible because domain adaptation requires target-domain access.

A.2. Hyper-parameter Impacts

The method separates covariance elements by their variance under photometric transformations and evaluates sensitivity, whitening scope, margin, and loss-weight choices through controlled experiments.

  • Criteria for separating covariance elements: Covariance elements are divided into domain-specific style and domain-invariant content groups using k-means on variance across photometric transformations.Color jittering and Gaussian blur provide the transformations used to measure covariance sensitivity.
  • Criteria for separating covariance elements: The threshold separating covariance groups depends on k, with the first m clusters treated as insensitive and remaining clusters as transformation-sensitive.The paper sets m to one and searches for the optimal k.
  • Hyper-parameter evaluation: The experiments report mIoU across Cityscapes, BDD-100K, Mapillary, SYNTHIA, and GTAV validation sets for different k, δ, and γ values.The models are trained on GTAV, use ResNet-50, and results are averaged over three runs.
  • Whitening scope: ISW applies whitening to only selected covariance elements according to k, whereas the baseline whitening loss applies it to all covariance elements.
  • Loss hyper-parameters: The proposed ISW loss weight γ is empirically set to 0.6, while the IRW margin δ is varied to assess its performance impact.

A.3. Further Implementation Details

The implementation applies selective whitening within an encoder-based segmentation architecture and examines qualitative behavior across seen and unseen urban-scene conditions. Additional analyses cover failure cases and covariance-associated image tendencies.

  • Architecture: The segmentation architecture uses ResNet with DeepLabV3+, auxiliary per-pixel cross-entropy, and low-level features concatenated with high-level features.The detailed architecture is shown in Fig. 11.
  • Selective whitening placement: ISW is applied to the input convolutional layer and ResNet stages 1 and 2 because earlier layers tend to encode style information.
  • Training details: The total loss combines the training objectives with weights γ1 = 0.4 and γ2 = 0.6, using batch sizes of 8 for Cityscapes and 16 for GTA.Photometric augmentation uses Gaussian blur and color jittering with brightness, contrast, saturation, and hue settings specified in the implementation.
  • Comparison of segmentation results: On seen Cityscapes images, the model has comparable qualitative performance to the baseline, while under unseen illumination changes it produces less damaged maps and better predicts roads and cars.The evaluated illumination conditions include dusk, night, and shadows in BDD-100K.
  • Comparison of segmentation results: Across adverse weather, unseen structures, and lush vegetation, the method gives fair or reasonable predictions where the baseline often fails or produces noisy segmentation.Examples include snowy sidewalks, foggy scenes, roads, sidewalks, parking lots, and vegetation.
  • Covariance effects in images: The covariance analysis associates smaller sensitive covariance values with darker images and smaller insensitive covariance values with simpler scenes.The paper conjectures that sensitive covariances represent illumination information, whereas insensitive covariances tend to represent scene complexity.
Loading 2103.15597v2…