Source-linked AI summary
On Feature Normalization and Data Augmentation
Boyi Li, Felix Wu, Ser-Nam Lim, Serge Belongie, Kilian Q. Weinberger
TL;DR
Recognition models often discard feature moments as noise, although instance-level moments can contain style, shape, and label-relevant information. MoEx exchanges moments between training examples and interpolates their labels to make models use moments alongside normalized features. Across models, tasks, and datasets, the paper reports consistent improvements and complementarity with existing augmentation methods.
Problem
Recognition normalization typically discards feature moments as noise, despite evidence that instance-level moments contain style, structure, and label-relevant information.
Method
MoEx swaps the mean and variance of one training example with those of another and interpolates their labels, forcing learning from moments and normalized features.
Results
MoEx consistently improves classification accuracy and robustness across many datasets, architectures, and prediction tasks, while complementing existing augmentation methods.
Takeaways & Limitations
Because MoEx operates in feature space and is simple to implement, it can be combined with existing augmentation approaches for discriminative deep networks.
Takeaways & Limitations
MoEx assumes the network uses an invertible intra-instance normalization.
Abstract
from arXiv · showhide
The moments (a.k.a., mean and standard deviation) of latent features are often removed as noise when training image recognition models, to increase stability and reduce training time. However, in the field of image generation, the moments play a much more central role. Studies have shown that the moments extracted from instance normalization and positional normalization can roughly capture style and shape information of an image. Instead of being discarded, these moments are instrumental to the generation process. In this paper we propose Moment Exchange, an implicit data augmentation method that encourages the model to utilize the moment information also for recognition models. Specifically, we replace the moments of the learned features of one training image by those of another, and also interpolate the target labels -- forcing the model to extract training signal from the moments in addition to the normalized features. As our approach is fast, operates entirely in feature space, and mixes different signals than prior methods, one can effectively combine it with existing augmentation approaches. We demonstrate its efficacy across several recognition benchmark data sets where it improves the generalization capability of highly competitive baseline networks with remarkable consistency.
1. Introduction
Recognition models often discard instance-level feature moments as noise, despite evidence that these statistics contain label-relevant structure. MoEx exchanges moments between examples and interpolates their labels so models learn from both moments and normalized features.
- Motivation: Recognition normalization typically removes mean and standard deviation as noise, whereas generation methods treat them as informative style or structural signals.The contrast motivates retaining moments for recognition rather than discarding them.
- Motivation: Feature moments from a first-layer ResNet representation can preserve enough visual information for class labels to remain visually inferable.Figure 1 uses PONO mean and standard deviation to illustrate this structural information.
- Moment Exchange: MoEx swaps mean and variance between images, then trains on an interpolation of their labels so the network uses both normalized features and injected moments.For example, cat features receive plane moments, combining signals from both images.
- Moment Exchange: MoEx operates in feature space and can be combined with input-space or label-perturbing augmentations because it alters a different signal.The paper describes this effect as complementary and stackable with existing augmentation gains.
- Evidence: A ResNet using only moments can make nontrivial CIFAR-100 predictions, while removing PONO moments hurts performance and MoEx improves use of both signals.Figure 2 reports error rates averaged over three random runs.
- Results: Across models and tasks, MoEx yields improvements and is reported as applicable beyond computer vision to speech recognition and natural language processing.The introduction presents these results as evidence of broad effectiveness across data types.
2. Background and Related Work
Feature normalization historically removes moments computed across batches, but instance-wise normalization exposes moments that describe individual feature distributions. The paper positions these intra-instance moments as useful signals for feature-space augmentation alongside established input and label augmentation methods.
- Feature Normalization: Batch normalization removes moments computed across instances because they capture dataset biases viewed as non-predictive, while improving optimization and generalization.This rationale differs from instance-wise normalization, which treats each sample as its own feature distribution.
- Feature Normalization: Instance-wise normalization extracts per-sample mean and variance, referred to as intra-instance moments, rather than computing moments across a mini-batch.These moments describe the distribution of an individual instance’s features.
- Feature Normalization: Prior generative work indicates that intra-instance moments can encode attributes of data instances and can therefore serve as signals for data augmentation.The paper connects these attributes to useful information in generative models.
- Data Augmentation: Data augmentation includes label-preserving input transformations, text-specific perturbations, implicit feature or loss changes, and label-perturbing methods such as Mixup and Cutmix.This history frames MoEx within several existing augmentation families.
3. Moment Exchange
Moment Exchange combines intra-instance feature normalization with augmentation by exchanging moments between training examples while retaining normalized features. The exchanged representation is paired with interpolated labels so the network uses both normalized features and moment information.
- Moment Exchange: MoEx fuses normalized features from one instance with moments from another, creating an asymmetric composition in feature space.The method applies the inverse normalization using the recipient's normalized features and the donor's moments.
- Normalization: Intra-instance normalization decomposes each feature representation into normalized features, a first moment, and a second moment.The inverse function reconstructs the feature representation from these components; PONO computes moments across channels.
- Motivation: Because intra-instance moments are computed from individual samples, they can contain label-relevant signal rather than only batch-level bias.The paper treats normalized features and moments as distinct views of the same instance, with the first moment described as conveying primarily structural information.
- Label mixing: MoEx encourages use of injected moments by predicting both source labels with an interpolation weight λ.The exchanged features contain the donor image's moments inside the recipient image's normalized features.
- Implementation: MoEx can be applied at one layer, with the preferred layer depending on the normalization method, and mini-batch permutations provide efficient pairings.The procedure introduces negligible training overhead and typically uses p = 0.5, while ImageNet uses p = 1 and λ = 0.9 works across datasets.
- Properties: Operating entirely in feature space lets MoEx complement input-space and label-perturbing augmentations such as cropping, flipping, Mixup, and Cutmix.The paper describes the effects as complementary and recommends combining MoEx with established augmentation methods.
4. Experiments
MoEx is evaluated across image classification, object detection, and 3D model classification, consistently improving competitive baselines and combining effectively with existing augmentation methods.
- Image Classification on CIFAR: MoEx achieves the lowest error rates in nearly all CIFAR-100 settings and improves further when combined with CutMix or ShakeDrop.
- Image Classification on ImageNet: MoEx improves ImageNet classification across model architectures, reaching 20.9/5.7 Top-1 and Top-5 error with CutMix.
- Object Detection: MoEx pretraining provides better initialization than the baseline ResNet-50 and remains competitive with CutMix for Pascal VOC object detection.
- 3D Model Classification: MoEx reduces ModelNet10 and ModelNet40 classification errors from 6.0% to 5.3% and 9.2% to 8.8%, respectively.
5. Ablation Study
The ablation studies show that MoEx benefits depend on both exchanging feature moments and interpolating labels, with performance varying by normalization method, layer, and model capacity. Experiments beyond image classification further indicate improvements in speech recognition, while tiny models may benefit less from augmentation.
- MoEx Design Choices: Label interpolation is crucial: MoEx provides no improvement when λ = 1.
- MoEx Design Choices: Moment exchange is also necessary, because removing it reduces MoEx to label smoothing with significantly smaller benefits.
- Normalization Methods: MoEx generally works with LayerNorm, InstanceNorm, PONO, GroupNorm, and local response normalization, but PONO has a slight yet significant edge.
- Normalization Methods: The best exchange layer depends on normalization: PONO and GroupNorm favor the first layer, LayerNorm the third stage, and InstanceNorm the second stage.
- Hyper-parameters: Higher λ generally leads to lower ImageNet error, likely because normalized features contain more information than moments.
- Beyond Computer Vision: MoEx improves Speech Command baselines significantly in all but one case, with DenseNet-BC-100 as the exception because of its small parameter count.
- Beyond Computer Vision: MoEx extracts and injects feature statistics rather than interpolating images or features, while leaving other statistics such as principal components as possible alternatives.
6. Conclusion and Future Work
MoEx is presented as a simple feature-space augmentation method that makes recognition networks use normalization moments rather than discard them. Across datasets, architectures, and prediction tasks, it consistently improves classification accuracy and robustness and can be combined with existing augmentation methods.
- MoEx forces recognition networks to use moments extracted by intra-instance normalization instead of disregarding them.
- MoEx consistently improves classification accuracy and robustness across many datasets, model architectures, and prediction tasks.
- Because MoEx operates in feature space, it is complementary to existing state-of-the-art augmentation methods and readily combinable with them.
A. MoEx PyTorch Implementation
The PyTorch implementation normalizes a feature batch, permutes instance moments across samples, reinjects them, and returns both original and moment-associated labels for interpolated loss computation.
- The implementation defines MoEx over a batch of features, one-hot labels, and a selected normalization type.
- MoEx randomly permutes batch indices, replaces each sample’s moments with another sample’s moments, and returns the original and permuted labels.
- The interpolated loss combines the losses for the original label y and the moment-associated label y_b using λ and 1 − λ.
- The normalization dimensions depend on the selected normalization type, including instance normalization over channels, height, and width.
- Normalization computes each feature instance’s mean and variance, standard deviation, and normalized representation.
B.1. Machine Translation on IWSLT 2014
On four IWSLT 2014 machine-translation tasks, MoEx consistently improves the DynamicConv baseline by about 0.2 BLEU and 0.2% BERT-F1 across three-run evaluations.
- MoEx is evaluated with DynamicConv on German-to-English, English-to-German, Italian-to-English, and English-to-Italian translation.
- MoEx consistently improves the baseline on all four tasks by about 0.2 BLEU and 0.2% BERT-F1.
- The evaluation reports BLEU and scaled BERTScore F1 averages with standard errors over three runs, with higher scores better.
C. More Examples of MoEx
Figure 4 presents additional MoEx examples by displaying the top five features among 64 channels. The features of image A are normalized and infused with PONO moments from image B.
- Figure 4 shows additional examples of MoEx using the top five features selected from 64 channels.
- MoEx normalizes image A’s features before infusing them with image B’s PONO mean and standard deviation.
- The displayed examples illustrate moment exchange across images at the feature level.