PaperlayerView exact revision on arXiv

Source-linked AI summary

The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization

M. Jehanzeb Mirza, Jakub Micorek, Horst Possegger, Horst Bischof

arXiv:2112.00463v2cs.CV

TL;DR

Domain shifts can substantially impair neural networks, while existing adaptation methods often require labeled data or large target datasets. DUA dynamically adapts batch-normalization statistics online from a small fraction of unlabeled shifted data. Across varied tasks and architectures, it achieves competitive or state-of-the-art performance with minimal computational overhead.

  • Problem

    Domain shifts degrade neural-network performance, while practical adaptation may lack labeled training data or large amounts of unlabeled target data.

  • Method

    DUA sequentially adapts batch-normalization running statistics using incoming unlabeled target samples, an adaptive momentum scheme, and no back propagation.

  • Results

    DUA achieves competitive results with less than 1% of unlabeled test data and shows strong performance across varied benchmarks, tasks, and architectures.

  • Takeaways & Limitations

    Online batch-normalization adaptation provides a simple, low-overhead approach for dynamically shifting distributions across batch-normalization-based architectures.

Abstract

from arXiv · show

Domain adaptation is crucial to adapt a learned model to new scenarios, such as domain shifts or changing data distributions. Current approaches usually require a large amount of labeled or unlabeled data from the shifted domain. This can be a hurdle in fields which require continuous dynamic adaptation or suffer from scarcity of data, e.g. autonomous driving in challenging weather conditions. To address this problem of continuous adaptation to distribution shifts, we propose Dynamic Unsupervised Adaptation (DUA). By continuously adapting the statistics of the batch normalization layers we modify the feature representations of the model. We show that by sequentially adapting a model with only a fraction of unlabeled data, a strong performance gain can be achieved. With even less than 1% of unlabeled data from the target domain, DUA already achieves competitive results to strong baselines. In addition, the computational overhead is minimal in contrast to previous approaches. Our approach is simple, yet effective and can be applied to any architecture which uses batch normalization as one of its components. We show the utility of DUA by evaluating it on a variety of domain adaptation datasets and tasks including object recognition, digit recognition and object detection.

1. Introduction

Domain shifts can sharply degrade neural-network performance, while existing adaptation settings often require labeled training data or large unlabeled target datasets. DUA instead adapts batch-normalization statistics online using a tiny fraction of unlabeled shifted data, with low overhead and broad empirical coverage.

  • Even slight distribution shifts between training and testing can significantly degrade neural-network performance.
  • DUA addresses scarce-data settings by adapting running means and variances online as shifted test samples arrive.It uses training statistics as a prior and updates them sequentially on incoming samples.
  • Less than 1% of unlabeled test data enables DUA to perform competitively with strong baselines using the entire test set.
  • DUA requires no back propagation and has negligible computational overhead, making it suitable for real-time applications.
  • DUA achieves state-of-the-art results on most evaluated benchmarks and remains competitive on the others.
  • The method is evaluated across different tasks and architectures, including dynamic adaptation for object detection.

2. Related Work

Unsupervised domain adaptation methods commonly reduce domain discrepancy, use adversarial or self-supervised objectives, or correct domain statistics. DUA differs by updating training-derived statistics online from individual incoming samples without back propagation.

  • UDA approaches include discrepancy reduction, adversarial alignment, self-supervised adaptation, and domain-statistics correction.
  • Discrepancy-reduction methods align moments or embeddings between source and target domains at selected network layers.
  • Adversarial approaches use domain-confusion objectives to bring source and target features closer.
  • Self-supervised approaches adapt at test time through auxiliary tasks such as rotation prediction.
  • Batch-normalization approaches improve generalization by recalculating or learning mixtures of domain statistics.
  • DUA updates training-derived statistics online, uses less than 1% of unlabeled test data, and avoids back propagation for single-frame dynamic adaptation.

3. Approach

The approach adapts batch-normalization statistics online to align shifted test activations with training distributions while keeping the rest of the network fixed. Incoming samples are augmented into small batches, and an adaptive update controls sequential adaptation.

  • 3.1. Batch Normalization: Batch normalization normalizes activations using training-data mean and variance, with learned scale and shift parameters.During training, running statistics are updated from batch statistics; during inference, they are normally fixed.
  • 3.2. Dynamic Unsupervised Adaptation: DUA adapts only the running mean and variance to target statistics while keeping all other network parameters fixed.The source training statistics serve as a prior, and target examples update them sequentially as they arrive.
  • 3.2. Dynamic Unsupervised Adaptation: DUA aligns shifted-test activation distributions with training distributions, addressing the misalignment associated with performance degradation.The figure compares clean training outputs with Contrast Level-5 outputs before and after adaptation.
  • 3.2. Dynamic Unsupervised Adaptation: The adaptive momentum update gives later samples smaller influence as momentum decays, producing more stable adaptation than default parameters.The decay parameter ω lies between 0 and 1, while ζ supplies a positive lower bound for momentum.
  • 3.2. Dynamic Unsupervised Adaptation: DUA uses random flipping, cropping, and rotation to form a small adaptation batch from each incoming sample.The augmentations are chosen not to correlate with the shifted test corruption used in the experiments.

4. Results

DUA is evaluated across corruption, domain, and detection benchmarks using sequential adaptation on less than 1% of unlabeled test data. It achieves strong or competitive performance across tasks, including substantial gains for fog and rain detection.

  • Evaluation Setup: DUA evaluates sequential online adaptation across image classification, object detection, digit recognition, Office-31, VIS-DA, and day-to-night or cross-dataset detection shifts.The evaluation covers multiple datasets, tasks, and domain-shift settings.
  • Evaluation Setup: Less than 1% of unlabeled test data is used for DUA adaptation, while the reported baselines adapt on the complete test set.DUA adapts on each incoming sample sequentially; exact sample counts are listed in the supplemental material.
  • Image Classification: DUA achieves a new state-of-the-art on the highest-severity CIFAR-10C benchmark and outperforms TTT and NORM while remaining competitive with TENT on CIFAR-100C.The comparison uses top-1 classification error at severity Level 5.
  • Image Classification: DUA performs on-par with all baselines on ImageNet-C at the highest severity level using a ResNet-18 backbone.The reported metric is top-1 error.
  • Object Detection: DUA is tested on object detection under covariate shifts from rain and fog and under domain shifts between datasets.The detection experiments use a KITTI-pretrained YOLOv3 model.
  • Object Detection: 17.7% mAP improvement is achieved for the most severe fog condition, compared with the KITTI source model.For the most severe rain condition, the mean improvement is 7.1% mAP.

5. Ablation Studies

Ablations show that DUA is stable across sample order, supports rapid dynamic switching between weather domains, and benefits from adapting all batch-normalization layers with augmented micro-batches.

  • 5.1. Sample Order Does Not Matter: 27.2 ± 0.09 error is reached after 100 samples, with performance stable across 300 shuffled runs on CIFAR-10C.The initial source-only mean error is 49.2%, and performance reaches 28.3 ± 0.19 after 25 samples.
  • 5.2. Continuous Dynamic Adaptation: DUA quickly recovers prior-domain performance after switching from day to fog and back, despite a 5.8% drop from the day baseline at iteration 100.The authors conjecture that unchanged learned weights help retain information from the previous domain.
  • 5.3. Batch Normalization Layers: Adapting all batch-normalization layers gives the best performance, while individual improvements are slightly larger in later layers.The ablation fixes unselected layers while adapting batch-normalization layers in specific ResNet-26 blocks.

6. Conclusion

DUA sequentially adapts batch-normalization statistics on unlabeled out-of-distribution samples, using an adaptive momentum scheme for fast and stable adaptation. It achieves competitive results with only a fraction of test data across tasks and architectures, with minimal computational overhead.

  • 6. Conclusion: DUA adapts a trained model sequentially on each unlabeled out-of-distribution test sample.The method modifies batch-normalization statistics without requiring access to training data.
  • 6. Conclusion: An adaptive momentum scheme is introduced to support fast and stable adaptation.
  • 6. Conclusion: DUA achieves competitive results to strong baselines using only a fraction of test data.
  • 6. Conclusion: Experiments across challenging benchmarks and tasks demonstrate utility on a broad range of batch-normalization-based architectures.
  • 6. Conclusion: Minimal computational overhead makes DUA suitable for real-time systems and embedded devices.

Appendices

The appendices provide reproducibility details and additional experimental results. These materials are organized into evaluation details and more extensive results.

  • Appendices: The appendices summarize evaluation details for reproducibility and provide additional detailed results.

A. Evaluation Details

The evaluation details specify adaptation sample usage, augmentation procedures, training configurations, model architectures, and corruption-benchmark experiments. They also report CIFAR-10C adaptation results across repeated runs.

  • A. Evaluation Details: DUA uses only a tiny fraction of each unlabeled test set, after which adaptation performance saturates.Experiments were conducted on a single NVIDIA GeForce RTX 3090.
  • A. Evaluation Details: Batch augmentations use random cropping, random horizontal flipping, and rotations by 0, 90, 180, or 270 degrees.The ablation study found that augmentations further improve adaptation performance.
  • A. Evaluation Details: CIFAR-10/100 corruption models use a ResNet-26 trained for 150 epochs with SGD, batch size 128, and scheduled learning-rate milestones at 75 and 125 epochs.
  • A. Evaluation Details: Classification experiments use ImageNet-pretrained ResNet-18 or ResNet-50 models finetuned on Office-31 or VIS-DA training splits.
  • A. Evaluation Details: Digit-recognition experiments use a two-convolution, two-linear-layer architecture with two batch-normalization and two dropout layers.The model is trained for 15 epochs with ADADELTA and batch size 64.
  • A. Evaluation Details: Object-detection experiments retrain a YOLOv3 model pretrained on MS-COCO for 100 epochs on each dataset's training split.The batch size is 20.
  • A. Evaluation Details: CIFAR-10C adaptation results are reported over 300 runs, using mean error across 15 corruptions and the corresponding standard deviation after each incoming sample.

B.1. Sample Order Does Not Matter

Additional experiments examine sample-order stability, momentum decay, and performance across digit and visual-recognition benchmarks. DUA remains stable across runs, improves digit-recognition results, and is presented as broadly applicable rather than task-specialized state-of-the-art replacement.

  • B.1. Sample Order Does Not Matter: DUA is consistently stable across all 300 runs, and the sample order does not matter.
  • B.1. Sample Order Does Not Matter: A momentum decay of ω = 0.94 empirically balances stable adaptation with fast convergence.Higher ω values make adaptation unstable, whereas lower values make it slower.
  • B.1. Sample Order Does Not Matter: DUA improves results across all reported popular cross-dataset digit-recognition benchmarks.
  • B.1. Sample Order Does Not Matter: DUA improves results on both Office-31 and VISDA using finetuned ImageNet-pretrained ResNet models.
  • B.1. Sample Order Does Not Matter: These visual-recognition evaluations demonstrate applicability across tasks and architectures, and DUA can precede established domain-adaptation methods.The evaluations are not intended to outperform task-specialized state-of-the-art methods.

C.3. Corruption Benchmarks

The appendix reports DUA results across corruption benchmarks, natural domain shifts, and degraded-weather object detection. DUA consistently improves performance across corruption types and severity levels and provides gains across multiple detection scenarios.

  • Corruption benchmarks: DUA achieves consistent improvements across all corruption types and severity levels evaluated for CIFAR-10C, CIFAR-100C, and ImageNet-C.Lower severity levels 1–4 are reported in the appendix, while severity level 5 appears in the main manuscript.
  • Natural domain shifts: DUA provides notable gains when adapting object detectors across day-to-night, KITTI-to-SODA10M, and SODA10M-to-KITTI shifts.The experiments use YOLOv3 and report domain adaptation results using mAP@50.
  • Degrading weather: DUA increases detection performance for lower-severity rain and fog conditions in KITTI-Fog and KITTI-Rain.The appendix evaluates lower severity levels, while the highest-severity or lowest-visibility setting is included in the main manuscript.
  • Corruption benchmarks: A figure illustrates constructing a batch from one ImageNet-C image through random augmentation under Level 5 Gaussian Noise Corruption.This example shows the batch-creation procedure used in the corruption setting.
  • Corruption benchmarks: The corruption experiments compare error rates across individual corruption types using specified ResNet backbones and baselines including TTT, NORM, and TENT.CIFAR-10C and CIFAR-100C use ResNet-26 for comparisons with TTT and NORM, while ImageNet-C uses ResNet-18.
Loading 2112.00463v2…