Source-linked AI summary
Revisiting Batch Normalization For Practical Domain Adaptation
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, Xiaodi Hou
TL;DR
DNN adaptation is difficult because source and target distributions differ, while conventional fine-tuning requires substantial labeled target data and computation. The paper proposes AdaBN, which replaces source-domain BN statistics with target-domain statistics without additional tunable parameters. AdaBN achieves state-of-the-art results across single- and multi-source benchmarks and shows practical effectiveness in cloud detection, while its simple per-feature transformations may not capture intrinsically nonlinear transfer functions.
Problem
DNNs depend strongly on their training datasets, and adapting them to new domains requires costly labeled data and computation because source and target distributions differ.
Method
AdaBN adapts batch-normalized DNNs by replacing source-domain statistics in each BN layer with target-domain statistics, without additional tunable parameters.
Results
AdaBN establishes state-of-the-art results on standard single- and multi-source domain-adaptation benchmarks and demonstrates effectiveness in cloud detection for large-size remote-sensing images.
Takeaways & Limitations
AdaBN requires minimal effort to extend to multiple source domains and semi-supervised settings and is complementary to other deep-learning domain-adaptation methods.
Takeaways & Limitations
The paper questions whether AdaBN’s simple translation and scaling operations can approximate an intrinsically nonlinear domain-transfer function.
Abstract
from arXiv · showhide
Deep neural networks (DNN) have shown unprecedented success in various computer vision applications such as image classification and object detection. However, it is still a common annoyance during the training phase, that one has to prepare at least thousands of labeled images to fine-tune a network to a specific domain. Recent study (Tommasi et al. 2015) shows that a DNN has strong dependency towards the training dataset, and the learned features cannot be easily transferred to a different but relevant task without fine-tuning. In this paper, we propose a simple yet powerful remedy, called Adaptive Batch Normalization (AdaBN) to increase the generalization ability of a DNN. By modulating the statistics in all Batch Normalization layers across the network, our approach achieves deep adaptation effect for domain adaptation tasks. In contrary to other deep learning domain adaptation methods, our method does not require additional components, and is parameter-free. It archives state-of-the-art performance despite its surprising simplicity. Furthermore, we demonstrate that our method is complementary with other existing methods. Combining AdaBN with existing domain adaptation treatments may further improve model performance.
1 INTRODUCTION
The paper addresses domain shift in DNN training by introducing AdaBN, which adapts batch-normalization statistics between source and target domains without extra tunable parameters. It reports strong benchmark performance and practical cloud-detection experiments.
- Imperfectly matched source and target image distributions can cause DNNs to capture dataset bias and perform worse on the target domain.
- Existing deep domain-adaptation methods commonly add optimization steps and parameters, increasing the computational burden of DNN training.
- AdaBN adapts a batch-normalized DNN by modulating layer statistics, treating weights as label-related knowledge and normalization statistics as domain-related knowledge.
- AdaBN is straightforward to implement, parameter-free, and requires minimal computational resources.
- AdaBN validates effectively on single-source and multi-source benchmarks, where it outperforms state-of-the-art methods.
- Experiments on cloud detection for remote-sensing images further demonstrate the approach in practical use.
2 RELATED WORK
Related work frames domain adaptation as correcting distribution differences and notes that deep feature transfer remains difficult. Prior approaches include shallow transfer functions, fine-tuning, discrepancy losses, and feature-correlation alignment, while BN supplies the relevant normalization framework.
- Domain transfer addresses dataset bias and covariate shift by modeling differences between source and target distributions.
- Many earlier transfer functions use computationally manageable shallow projections built from kernel transformations and linear mappings.
- Fine-tuning is a common transfer procedure but requires considerable labeled target data and substantial resources to retrain the network.
- Deep adaptation methods include MMD-based representation regularization in CNNs and related approaches that directly address unlabeled target-domain classification.
- CORAL aligns second-order source and target statistics by whitening source features and re-correlating them to target features.
- Batch Normalization standardizes features within mini-batches and learns a slope and bias, while global training statistics normalize test batches.
- BN has been reported to accelerate convergence and improve final performance, becoming standard in architectures including ResNet and Inception V3.
3 THE MODEL
The model analyzes domain shift in BN statistics and proposes AdaBN, which adapts a pretrained DNN by recalculating target-domain statistics across BN layers. The approach is designed to align representations with minimal computational and parameter cost.
- 3.1 A PILOT EXPERIMENT: Both shallow and deep DNN layers are influenced by domain shift, so adapting only the output layer is insufficient.
- 3.1 A PILOT EXPERIMENT: BN-layer statistics contain traits of the data domain, motivating adaptation through BN layers.
- 3.2 ADAPTIVE BATCH NORMALIZATION: AdaBN adapts a pretrained model by modulating BN statistics for a target domain while retaining the learned network weights.
- 3.2 ADAPTIVE BATCH NORMALIZATION: The method standardizes samples using statistics from their own domain, including separate domain statistics for multi-domain adaptation.
- 3.2 ADAPTIVE BATCH NORMALIZATION: AdaBN requires zero parameters to tune and minimal computational resources, and can also be combined with fine-tuning in supervised settings.
- 3.3 FURTHER THOUGHTS ABOUT ADABN: AdaBN’s simple translation and scaling operations raise the question of whether they can approximate intrinsically nonlinear domain-transfer functions.
4 EXPERIMENTS
The experiments evaluate AdaBN on standard domain-adaptation datasets, analyze adapted features, and test a practical remote-sensing application.
- The evaluation covers standard domain-adaptation datasets.
- The experiments empirically analyze the features produced after adaptation.
- The method is also evaluated in a practical application involving remote-sensing images.
4.1 EXPERIMENTAL SETTINGS
The evaluation uses Office and Caltech-Bing for single- and multi-source domain adaptation, compares AdaBN with shallow and deep baselines, and follows established training protocols.
- Datasets: Office contains 4652 images in 31 classes across Amazon, DSRL, and Webcam domains, supporting six pairwise and three multi-source transfer tasks.
- Datasets: Caltech-Bing contains 30,607 Caltech-256 images and 121,730 Bing images across 256 categories from two domains.
- Protocols: Table 1 reports single-source domain-adaptation results on Office-31 under the standard unsupervised adaptation protocol.
- Baselines: AdaBN is compared with four shallow methods and four deep methods, including CORAL, DDC, DAN, RevGrad, and Deep CORAL.
- Protocols: Single-source experiments follow the full protocol, while multi-source experiments train on all source samples and test on all target samples.
4.2 RESULTS
AdaBN improves domain-adaptation performance across single-source and multi-source Office and Caltech-Bing evaluations, while also benefiting from combination with CORAL in some settings.
- AdaBN improves the baseline and outperforms competing methods on average in both single-source and multi-source Office-31 adaptation.
- 0.5% increase in performance results from combining CORAL with AdaBN in the Office single-source experiment.
- 1.7% improvement over the baseline advances state-of-the-art results on the evaluated Office single-source dataset.
- 2.3% gain over the baseline is achieved by AdaBN in the Office multi-source setting.
- 1.8% improvement over the baseline is achieved by AdaBN on the Caltech-Bing dataset, although adaptation from Caltech-256 to Bing is difficult.
4.3 EMPIRICAL ANALYSIS
The empirical analysis measures domain discrepancy across network depth and examines how many target examples are needed to estimate BN statistics. AdaBN reduces discrepancy in both analyzed layers and works with relatively few target examples.
- 4.3.1 ANALYSIS OF FEATURE DIVERGENCE: The feature-divergence analysis uses outputs from the second convolution layer and the last Inception module before ReLU.
- 4.3.1 ANALYSIS OF FEATURE DIVERGENCE: Symmetric KL divergence measures the distance between source- and target-domain feature distributions in the shallow and deep layers.
- 4.3.1 ANALYSIS OF FEATURE DIVERGENCE: AdaBN reduces the source-target domain discrepancy in both a shallow layer and a deep layer.The experiment compares output distributions before and after adaptation using symmetric KL divergence.
- 4.3.2 SENSITIVITY TO TARGET DOMAIN SIZE: AdaBN obtains good results using only a small part of the target examples to calculate BN statistics.Accuracy is evaluated while varying the number of mini-batches used for the statistics, with batch size 64.
4.4 PRACTICAL APPLICATION FOR CLOUD DETECTION IN REMOTE SENSING IMAGES
The paper applies AdaBN to cloud detection across satellite-image domains with different visual distributions. On GF1 and Tianhui targets, the results show significant improvement while requiring no extra parameters and few computations.
- Remote-sensing images from different satellites form distinct domains because their sensors and resolutions produce different textures, colors, and value ranges.
- Cloud detection is evaluated as semantic segmentation using GF2 for training and GF1 and Tianhui as target datasets.The self-collected datasets contain 635 GF2 images, 324 GF1 images, and 113 Tianhui images.
- AdaBN significantly improves over the baseline on the GF1 and Tianhui remote-sensing datasets.The relatively low baseline results indicate substantial distribution disparity among images from different satellites.
- AdaBN requires no extra parameters and very few computations for this large-size segmentation adaptation problem.
5 CONCLUSION AND FUTURE WORKS
AdaBN adapts batch-normalized neural networks by replacing source-domain BN statistics with target-domain statistics. The method is parameter-free, extends to multiple-source and semi-supervised settings, and performs effectively on benchmarks and remote-sensing cloud detection.
- AdaBN replaces source-domain BN statistics with target-domain statistics for domain adaptation in batch-normalized neural networks.The approach modifies BN statistics rather than the network weights.
- The method is easy to implement, parameter-free, and requires almost no effort to extend to multiple-source and semi-supervised settings.
- AdaBN establishes state-of-the-art results on standard benchmarks for both single-source and multiple-source domain adaptation.
- Experiments on cloud detection for large-size remote-sensing images demonstrate the method's effectiveness in practical use.Figure 6 presents visual cloud-detection results on the GF1 dataset, with white pixels representing detected cloud regions in two outputs.
- Because AdaBN modifies only BN statistics, it is fully complementary to existing deep-learning domain-adaptation methods.The paper suggests that different methods may be unified under one framework.