Source-linked AI summary
Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization
Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia, Lei Zhang
TL;DR
Existing feature-matching methods often assume Gaussian features and match only mean and standard deviation, although real-world distributions are more complex and high-order matching is costly. This paper introduces EFDM, which exactly matches feature eCDFs using EHM with Sort-Matching, and applies it to AST and DG. The method achieves new state-of-the-art results across a variety of AST and DG tasks with high efficiency.
Problem
Gaussian-based matching of feature mean and standard deviation is less accurate for complex real-world feature distributions, while high-order statistics impose intensive computational overhead.
Method
EFDM exactly matches image-feature eCDFs using EHM, with the fast Sort-Matching algorithm providing a plug-and-play implementation.
Results
The method achieves new state-of-the-art results on a variety of AST and DG tasks with high efficiency.
Takeaways & Limitations
EFDM provides an efficient way to perform exact feature distribution matching for visual learning applications including AST and DG.
Takeaways & Limitations
EFDM has n log(n) complexity rather than AdaIN’s linear complexity and assumes different feature channels are independent.
Abstract
from arXiv · showhide
Arbitrary style transfer (AST) and domain generalization (DG) are important yet challenging visual learning tasks, which can be cast as a feature distribution matching problem. With the assumption of Gaussian feature distribution, conventional feature distribution matching methods usually match the mean and standard deviation of features. However, the feature distributions of real-world data are usually much more complicated than Gaussian, which cannot be accurately matched by using only the first-order and second-order statistics, while it is computationally prohibitive to use high-order statistics for distribution matching. In this work, we, for the first time to our best knowledge, propose to perform Exact Feature Distribution Matching (EFDM) by exactly matching the empirical Cumulative Distribution Functions (eCDFs) of image features, which could be implemented by applying the Exact Histogram Matching (EHM) in the image feature space. Particularly, a fast EHM algorithm, named Sort-Matching, is employed to perform EFDM in a plug-and-play manner with minimal cost. The effectiveness of our proposed EFDM method is verified on a variety of AST and DG tasks, demonstrating new state-of-the-art results. Codes are available at https://github.com/YBZh/EFDM.
1. Introduction
AST and DG can be framed as feature distribution matching, but Gaussian-based matching of mean and standard deviation is inadequate for complex real-world features. The paper proposes EFDM, which exactly matches feature eCDFs through EHM and Sort-Matching, and reports strong performance across AST and DG.
- AST transfers styles through cross-distribution feature matching, while DG uses style-based augmentation to generalize from source domains to unseen domains.
- Gaussian-based methods match feature means and standard deviations, but real-world feature distributions are usually too complex for accurate matching with these statistics alone.
- Matching high-order statistics is computationally intensive and would theoretically require infinitely many central moments for exact distribution matching.
- The paper proposes EFDM by exactly matching image-feature eCDFs, implemented with EHM that distinguishes equivalent values and applies element-wise transformations.
- Sort-Matching provides a fast, plug-and-play EHM implementation, while EFDM supports one-shot cross-distribution matching and style mixing for AST and DG.
2. Related Work
AST methods increasingly interpret style transfer as feature distribution matching, moving from Gaussian statistics toward higher-order or histogram-based representations. The paper positions EFDM as a more accurate and efficient alternative that addresses HM’s handling of equivalent feature values.
- AST includes iterative optimization-based and one-shot feed-forward methods, with the latter generally faster and suitable for real-time applications.
- AdaIN matches feature mean and standard deviation, whereas WCT additionally matches cross-channel covariance through whitening and coloring transforms.
- High-order moments and HM relax Gaussian assumptions for more exact matching, but explicit moments are computationally intensive and HM is impeded by equivalent feature values.
- EFDM exactly matches image-feature eCDFs using EHM, and Sort-Matching provides an accurate, efficient implementation for more faithful AST results.
3. Methodology
This section develops feature distribution matching beyond Gaussian mean and standard deviation by matching empirical CDFs exactly with EFDM, implemented through fast Sort-Matching. It applies EFDM to AST and DG, including differentiable style transfer and mixed-style feature augmentation.
- AdaIN, HM and EHM: AdaIN matches feature mean and standard deviation under a Gaussian assumption, which is less accurate for non-Gaussian real-world feature distributions.Histogram matching instead targets the empirical CDF of the input vector to the target vector.
- AdaIN, HM and EHM: Exact Histogram Matching distinguishes equivalent feature values and applies element-wise transformations, improving empirical-CDF matching over histogram matching that merges equivalent values.Equivalent feature values are common in deep features because of discrete image pixels and activation functions such as ReLU.
- EFDM for AST and DG: Sort-Matching pairs sorted input and target vectors element-wise, requiring equal vector sizes in the focused AST and DG applications and having O(n log n) sorting complexity.For unequal sizes, interpolation or element dropping can make the vectors equal in size.
- EFDM for AST and DG: EFDM is implemented channel-wise without parameters, with a stop-gradient modification enabling back-propagation and plug-and-play use at minimal cost.The practical transformation is applied to feature tensors whose dimensions represent batch size, channels, height, and width.
- EFDM for AST and DG: For AST, EFDM matches encoded content and style features before decoding, while its style loss measures divergence against EFDM-transferred targets rather than only mean and standard deviation.The AST objective combines content loss and style loss with a balancing hyper-parameter.
- EFDM for AST and DG: EFDMix interpolates sorted feature vectors to create mixed-style augmentations, producing more diverse outputs when style features share the same mean and standard deviation.This extension targets feature augmentation for domain generalization.
4. Experiments
Experiments evaluate EFDM and EFDMix across arbitrary style transfer and domain generalization tasks, including qualitative, quantitative, efficiency, and ablation analyses. The results report stable style transfer, improved generalization, diverse style manipulation, and comparable runtime to AdaIN, while identifying higher complexity and channel-independence as limitations.
- AST experiments: EFDM produces stable results on both standard and photo-realistic style-transfer tasks.The experiments compare EFDM with existing methods using MS-COCO content images and WikiArt style images.
- AST experiments: Varying λ from 1 to 0 smoothly transitions outputs from the content image toward the target style.λ = 1 yields the vanilla content image, while λ = 0 yields the most stylized image.
- AST experiments: Style interpolation with EFDM produces new styles by interpolating feature maps from multiple style images.The interpolation uses weighted EFDM outputs for style images Y_1 through Y_K.
- DG experiments: EFDMix consistently outperforms MixStyle and other competing methods on PACS classification and cross-domain person re-identification.The advantage is especially evident in the single-source generalization setting, where training data are less diverse.
- DG experiments: 87.9% accuracy on PACS exceeds the ERM benchmark by 1.2%.This result follows the DomainBed experimental strategy.
- Discussions: Matching more feature statistics improves results: AdaIN beats AdaMean and AdaStd by more than 1%, while EFDM performs best.HM’s approximate eCDF matching cannot guarantee exact mean and standard-deviation matching and shows degenerated performance.
- Discussions: EFDM runs at 256 FPS for 512 × 512 images, with speed nearly matching AdaIN despite O(n log(n)) complexity.The paper attributes the comparable runtime to finite feature size.
5. Conclusion
EFDM exactly matches feature distributions for AST and DG, outperforming existing state-of-the-art methods in visual quality and quantitative measures while remaining efficient.
- EFDM is applied to arbitrary style transfer and domain generalization as an exact feature-distribution matching method.
- Sort-Matching implements EFDM efficiently in deep feature space as a fast Exact Histogram Matching algorithm.
- EFDM demonstrates superior visual quality and quantitative performance compared with existing state-of-the-art AST and DG methods.
- The framework may extend classical normalization beyond mean and standard deviation statistics.The conclusion identifies this as a direction for further investigation.