Source-linked AI summary
MixStyle Neural Networks for Domain Generalization and Adaptation
Kaiyang Zhou, Yongxin Yang, Yu Qiao, Tao Xiang
TL;DR
Neural networks often fail on unseen domains, while collecting diverse labeled source data is costly. MixStyle mixes feature statistics between instances in shallow CNN layers to synthesize domains without changing training objectives, and the paper reports improved OOD generalization across varied tasks and learning settings.
Problem
Neural networks often generalize poorly from i.i.d. training data to out-of-distribution domains, while obtaining diverse annotated source domains is expensive and time-consuming.
Method
MixStyle is a parameter-free feature-level augmentation module that mixes instance feature statistics in shallow CNN layers and extends to labeled–unlabeled instance pairs.
Results
MixStyle yields significant improvements in OOD generalization and shows encouraging results across object recognition, person re-identification, and reinforcement learning.
Takeaways & Limitations
Mixing feature statistics efficiently simulates visual changes associated with colors, textures, illumination, and backgrounds, supporting representations invariant to those shifts.
Takeaways & Limitations
MixStyle may be less effective for geometrical shifts such as rotation or viewpoint changes, and the optimal layers to apply remain unclear for new tasks.
Abstract
from arXiv · showhide
Neural networks do not generalize well to unseen data with domain shifts -- a longstanding problem in machine learning and AI. To overcome the problem, we propose MixStyle, a simple plug-and-play, parameter-free module that can improve domain generalization performance without the need to collect more data or increase model capacity. The design of MixStyle is simple: it mixes the feature statistics of two random instances in a single forward pass during training. The idea is grounded by the finding from recent style transfer research that feature statistics capture image style information, which essentially defines visual domains. Therefore, mixing feature statistics can be seen as an efficient way to synthesize new domains in the feature space, thus achieving data augmentation. MixStyle is easy to implement with a few lines of code, does not require modification to training objectives, and can fit a variety of learning paradigms including supervised domain generalization, semi-supervised domain generalization, and unsupervised domain adaptation. Our experiments show that MixStyle can significantly boost out-of-distribution generalization performance across a wide range of tasks including image recognition, instance retrieval and reinforcement learning.
1 Introduction
The introduction frames out-of-distribution generalization as difficult because CNNs often fail under domain shifts, while collecting diverse labeled domains is costly. MixStyle addresses this by mixing shallow-layer feature statistics to synthesize novel domains without changing model objectives, and extends to partially labeled settings.
- CNNs often perform poorly on out-of-distribution target datasets despite strong results on i.i.d. data.
- Collecting diverse, annotated source domains is expensive and time-consuming, constraining domain-generalization models.
- MixStyle mixes means and standard deviations between random instances in shallow CNN layers to synthesize novel domains through implicit feature-level augmentation.
- The module is plug-and-play, parameter-free, requires no training-objective changes, and is easy to implement with a few lines of code.
- Experiments cover domain generalization, semi-supervised domain generalization, and unsupervised domain adaptation across recognition, retrieval, and reinforcement learning.
- The expanded paper adds unlabeled-data mixing, evaluates settings with as few as five labels per category, and reports extensive semi-supervised and adaptation experiments.
2 Related Work
Related work addresses domain generalization and related domain-shift problems through distribution alignment, domain-specific ensembles, meta-learning, image-level augmentation, and reinforcement-learning generalization methods. MixStyle is positioned as a feature-statistics-based alternative for synthesizing novel domains.
- Domain Generalization: Domain-generalization methods commonly align feature distributions across multiple source domains using discrepancy metrics such as MMD, contrastive losses, and adversarial objectives.
- Domain Generalization: Other domain-generalization approaches learn domain-specific classifiers or normalization layers for ensemble prediction, or construct pseudo-train and pseudo-test episodes through meta-learning.
- Data Augmentation: Prior augmentation methods synthesize novel-looking images by adversarial perturbation, learned perturbation networks, or optimal-transport-based mappings.
- Generalization in Deep RL: Reinforcement-learning generalization research uses regularization, image translation, domain randomization, randomized preprocessing, and label-preserving transformations.
- Unsupervised Domain Adaptation: Unsupervised domain adaptation assumes unlabeled target data, unlike domain generalization, and frequently applies domain-alignment methods.
- MixStyle: MixStyle’s schematic places feature-statistics mixing within CNN layers, using shallow-layer feature maps and statistics computed from paired instances.
3 MixStyle Neural Networks
MixStyle augments CNN training by mixing instance-level feature statistics to synthesize new styles, with efficient extensions for labeled and unlabeled data.
- MixStyle mixes CNN feature statistics between instances to implicitly synthesize new styles for feature-level data augmentation.The statistics are channel-wise means and standard deviations, which characterize visual domains through image style.
- The module is parameter-free, requires no buffer or objective modification, and can be inserted between CNN layers with minimal implementation effort.It can be implemented in a few lines using frameworks such as PyTorch or TensorFlow.
- 3.2 MixStyle Module: MixStyle computes statistics for two feature-map instances, forms a random convex mixture using λ sampled from Beta(α, α), and applies it to style-normalized features.The authors suggest α = 0.1 in practice.
- 3.2 MixStyle Module: During training, MixStyle may be activated probabilistically, is not used at test time, and blocks statistic gradients to preserve augmentation effects.The module uses activation probability 0.5 in the forward pass.
- 3.3 Extension to Semi-Supervised Learning: For unlabeled data, the semi-supervised framework mixes labeled and unlabeled instances, using unlabeled source data for semi-supervised DG or unlabeled target data for UDA.Pseudo-labeling assigns labels from model predictions, with a confidence threshold used to filter low-confidence estimates.
- MixStyle achieves comparable performance with L2A-OT while synthesizing novel-domain data in feature space rather than the input space.The caption characterizes the feature-space augmentation as more efficient.
4 Experiments
Experiments evaluate MixStyle across object recognition, instance retrieval, reinforcement learning, semi-supervised domain generalization, and unsupervised domain adaptation. Across these settings, MixStyle generally improves out-of-distribution performance with low computational and labeling requirements, though gains depend on the benchmark and domain shift.
- Object Recognition: MixStyle consistently improves vanilla models on PACS and Office-Home, outperforming pixel-space Mixup by around 4% and 1%, respectively.It also beats Manifold Mixup and shows clear advantages over DropBlock across domains.
- Object Recognition: MixStyle outperforms most state-of-the-art domain-generalization methods while adding negligible overhead to a supervised classifier.On PACS, cross-domain MixStyle surpasses L2A-OT by nearly 1% on average, while avoiding L2A-OT’s heavier generation network and engineering cost.
- Object Recognition: With ResNet50 on DomainBed, MixStyle beats ERM on PACS and VLCS but underperforms ERM by a large margin on OfficeHome.Its advantage over ERM shrinks as the model becomes larger.
- Instance Retrieval: MixStyle consistently outperforms strong vanilla person re-identification models across datasets, settings, and architectures.Random and cross-domain mixing perform generally comparably, indicating that domain labels are not required for the mixing strategy.
- Reinforcement Learning: On Coinrun, MixStyle improves unseen-environment performance for both the baseline and IBAC-SNI, while also reducing variance.Its compatibility with IBAC-SNI indicates complementary benefits in reinforcement-learning generalization.
- Semi-Supervised Domain Generalization: In low-data semi-supervised domain generalization, MixStyle raises PACS average accuracy by over 6% with 10 labels per class and nearly 7% with 5 labels per class.Its extension using unlabeled data produces results that can rival StyleMatch while remaining more efficient through feature-statistics mixing.
- Unsupervised Domain Adaptation: For unsupervised domain adaptation, MixStyle mixes labeled source and pseudo-labeled target styles and improves FixMatch by nearly 3%.Similar observations hold in the multi-source setting.
- Ablation Study and Analysis: Applying MixStyle to multiple layers performs better, while random shuffling is favored over a fixed shuffle across selected layers.Feature styles cluster by domain in earlier residual blocks, whereas later features correlate more strongly with class labels; mixing is also more diverse than replacing styles.
5 Discussion and Conclusion
MixStyle provides efficient feature-space augmentation for style-related domain shifts, extending across labeled and unlabeled settings and multiple applications. Its effectiveness is constrained for geometrical shifts, while layer placement remains a task-dependent practical choice.
- MixStyle bypasses pixel-space image synthesis by mixing feature statistics to achieve highly efficient data augmentation.The method exploits the relation between visual domains, image styles, and shallow-layer feature statistics.
- A modified mixing strategy allows MixStyle to handle unlabeled data in semi-supervised domain generalization and unsupervised domain adaptation.The paper reports potential for these settings rather than establishing a universal guarantee.
- Applying MixStyle to multiple shallow CNN layers is recommended, but the optimal layer set remains unclear for a new task.Layer selection is described as the main hyper-parameter requiring practical tuning.
- MixStyle improves out-of-distribution generalization across object recognition, person re-identification, and reinforcement learning.The reported applications involve shifts related to colors, textures, illuminations, backgrounds, and other image-style factors.
- MixStyle may be less effective for geometrical shifts such as rotation or changing viewpoints.The paper associates its less appealing Office-Home gains with viewpoint-driven rather than style-driven domain shifts.