PaperlayerView exact revision on arXiv

Source-linked AI summary

Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net

Xingang Pan, Ping Luo, Jianping Shi, Xiaoou Tang

arXiv:1807.09441v3cs.CV

TL;DR

CNN architectures often perform well within a single domain but generalize poorly across appearance changes. IBN-Net combines Instance Normalization and Batch Normalization to learn appearance invariance while preserving content information. It improves both within-domain performance and cross-domain generalization without target-domain data or increased computational cost.

  • Problem

    Existing CNN architectures are mainly designed for single tasks and domains, while performance drops when they are applied across domains with different appearances.

  • Method

    IBN-Net integrates Instance Normalization and Batch Normalization, using their complementary feature properties to build appearance invariance while preserving content discrimination.

  • Results

    IBN-Net consistently improves classic CNNs on ImageNet and improves cross-domain performance without target-domain data or finetuning.

  • Takeaways & Limitations

    Built-in appearance invariance enables CNNs to generalize to new image domains when target-domain data are unavailable.

  • Takeaways & Limitations

    Existing CNN architectures are vulnerable to appearance variations and have limited capacity to generalize beyond a single task and domain.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have achieved great successes in many computer vision problems. Unlike existing works that designed CNN architectures to improve performance on a single task of a single domain and not generalizable, we present IBN-Net, a novel convolutional architecture, which remarkably enhances a CNN's modeling ability on one domain (e.g. Cityscapes) as well as its generalization capacity on another domain (e.g. GTA5) without finetuning. IBN-Net carefully integrates Instance Normalization (IN) and Batch Normalization (BN) as building blocks, and can be wrapped into many advanced deep networks to improve their performances. This work has three key contributions. (1) By delving into IN and BN, we disclose that IN learns features that are invariant to appearance changes, such as colors, styles, and virtuality/reality, while BN is essential for preserving content related information. (2) IBN-Net can be applied to many advanced deep architectures, such as DenseNet, ResNet, ResNeXt, and SENet, and consistently improve their performance without increasing computational cost. (3) When applying the trained networks to new domains, e.g. from GTA5 to Cityscapes, IBN-Net achieves comparable improvements as domain adaptation methods, even without using data from the target domain. With IBN-Net, we won the 1st place on the WAD 2018 Challenge Drivable Area track, with an mIoU of 86.18%.

1 Introduction

IBN-Net addresses appearance gaps across domains by combining Instance Normalization and Batch Normalization to improve both CNN learning and generalization. Its design places normalization components according to where appearance and content information diverge across network depth.

  • Motivation: CNN performance drops when networks trained on one domain are applied to another because the images have different appearances.Cityscapes and GTA5 can share similar scene categories and configurations while differing between reality and virtuality.
  • Motivation: Complex, unknown appearance variations require architectural components that learn to reduce them rather than relying only on explicit data augmentation.The paper identifies arbitrary styles and virtuality as examples of such variations.
  • IBN-Net: IBN-Net integrates Instance Normalization and Batch Normalization so CNNs learn appearance-invariant features while preserving content discrimination.The architecture unifies normalization types instead of treating them as isolated components.
  • IBN-Net: Feature divergence decreases with depth for appearance differences but increases mainly in deeper layers for content differences, motivating depth-specific normalization.The design uses IN in shallow layers and retains BN features to preserve content information.
  • Results: 22.54%/6.32% top1/top5 errors are achieved by IBN-Net50 on the original ImageNet validation set, outperforming ResNet50 by 1.73%/0.76%.The networks have similar numbers of parameters and computational cost.
  • Results: 8.5% and 7.5% improvements are achieved by IBN-Net-integrated ResNet50 in the two cross-domain evaluation directions without finetuning.With 30% of Cityscapes training data, IBN-Net reaches 65.5% segmentation accuracy versus 63.8% for ResNet50 finetuned on all training data.

2 Related Works

Prior CNN architectures primarily target modeling performance within a single domain, while appearance variation limits cross-domain generalization. Related approaches use architectural invariance, transfer learning, domain adaptation, or domain generalization, but often require additional data or models.

  • Invariance in CNNs: Spatial invariance modules improve robustness to pose, viewpoint, scale, and distortion, but do not address image-appearance variation.Appearance changes can be encoded in hidden-feature means and variances, motivating instance normalization.
  • CNN Architectures: Existing CNN architectures mainly optimize performance for a single task and domain, leaving generalization to new domains limited.Their basic modules remain vulnerable to appearance variations.
  • Invariance in CNNs: Instance normalization helps remove appearance differences but can discard useful content information, hindering high-level vision performance.Prior image-classification experiments with IN performed worse than CNNs using BN.
  • Improve Performances across Domains: Transfer learning and domain adaptation reduce cross-domain performance drops but commonly require target-domain data, annotations, parameter adjustment, or separate model weights.Domain adaptation methods include discrepancy losses, adversarial losses, image translation, and BN-statistics adjustment.
  • Improve Performances across Domains: Domain generalization seeks representations that transfer to unknown target domains, yet acquiring multiple related source domains is often difficult.IBN-Net instead designs built-in appearance invariance without requiring target-domain data or related source domains.

3 Method

The method combines batch and instance normalization to balance content discrimination with appearance invariance. It places IN selectively in shallower network regions, preserves BN in deeper layers, and instantiates several IBN variants for CNN architectures.

  • 3.1 Background: Batch normalization accelerates training and preserves discrimination, whereas instance normalization removes contrast and appearance variation but can diminish useful information.IBN-Net unifies both normalizations to introduce appearance invariance without hurting feature discrimination.
  • 3.1 Background: IBN-Net places IN mainly in shallow layers, where appearance divergence is concentrated, while retaining BN features and avoiding IN in the deepest layers.This placement preserves deep content discrimination and part of the content information in shallow layers.
  • 3.2 Instance-Batch Normalization Networks: In the ResNet implementation, IN is added to the first three residual groups, with half the residual-path channels using BN and the other half using IN.The fourth group remains unchanged, and the identity path stays clean for optimization.
  • 3.2 Instance-Batch Normalization Networks: IBN-Net-a uses moderate IN placement to preserve content information, while IBN-Net-b adds IN after residual additions in early network stages.IBN-Net-b uses three IN layers after conv1, conv2 x, and conv3 x.
  • 3.2 Instance-Batch Normalization Networks: Variants concatenate IN and BN outputs or combine their placement schemes to retain generalizable and discriminative features.Concatenating both outputs introduces more parameters.

4 Experiments

Experiments show that IBN-Net improves within-domain modeling and cross-domain generalization across ImageNet and Cityscapes-GTA5, while requiring no target-domain data for adaptation comparisons. Results also indicate that IN is most useful in shallow layers, whereas BN preserves deeper content information.

  • ImageNet Classification: IBN-Net improves both modeling capacity and generalization under appearance transforms on ImageNet.IBN-Net-a improves original-image top1/top5 error by 1.73%/0.76%, while IBN-Net-b reduces performance drops on transformed domains.
  • ImageNet Classification: IBN-Net achieves consistent improvement over classic CNN architectures without adding parameters.IBN-ResNet101 reaches comparable or higher accuracy than ResNeXt101 and SE-ResNet101, with only marginal inference calculations.
  • ImageNet Classification: Adding IN to shallow residual groups improves performance, but adding it to the deepest group decreases performance.The result supports preserving BN in deep layers to retain important content information.
  • Cross Domain Experiments: IBN-Net improves within-domain modeling and cross-domain generalization on Cityscapes and GTA5.IBN-Net-a outperforms ResNet50 by 4.6% and 3.8% on the two datasets, while IBN-Net-b improves cross-evaluation performance by 8.5% from Cityscapes to GTA5 and 7.5% in the reverse direction.
  • Cross Domain Experiments: IBN-Net’s performance gain is comparable to domain adaptation methods despite using no target-domain data.The comparison uses a different setting because domain adaptation methods require target-domain data during training.
  • Cross Domain Experiments: With only 30% of Cityscapes training data, IBN-Net50-a outperforms ResNet50 finetuned on all Cityscapes data.This result indicates that the more generalizable model can reduce the data required for finetuning.
  • Feature Divergence Analysis: Feature divergence caused by appearance differences is reduced in IBN-Net, especially after IN layers in IBN-Net-b.The reduction persists into deeper layers, suggesting that appearance variance is reduced in deep features.

5 Conclusions

IBN-Net unifies instance and batch normalization to improve both modeling capacity and generalization across domains. The paper concludes that IN introduces appearance invariance and improves generalization, while BN preserves discriminative content information.

  • Conclusions: IBN-Net unifies instance normalization and batch normalization in one network to improve modeling and generalization capacity.The architecture improves classic CNNs including VGG, ResNet, ResNeXt, and SENet on ImageNet.
  • Conclusions: IN introduces appearance invariance and improves generalization, while BN preserves content information in discriminative features.The conclusion covers appearance changes across image domains and the retention of content-related information.
Loading 1807.09441v3…