Source-linked AI summary
HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation
Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, Xian-Sheng Hua
TL;DR
Unsupervised domain adaptation needs better ways to align complex, non-Gaussian feature distributions than first- and second-order statistics provide. The paper proposes arbitrary-order Higher-order Moment Matching, an RKHS extension, and pseudo-label-based target clustering, with third- and fourth-order variants outperforming existing moment-matching methods and target clustering further improving adaptation.
Problem
Existing discrepancy-based UDA methods mainly match first- or second-order statistics, which cannot completely characterize complex non-Gaussian deep-feature distributions.
Method
HoMM unifies MMD and CORAL, generalizes them to arbitrary-order moment tensor matching, extends matching into RKHS, and clusters reliable target samples using pseudo-labels.
Results
Third- and fourth-order moment matching significantly outperform existing moment-matching methods, while discriminative target clustering consistently improves transfer performance.
Takeaways & Limitations
Higher-order statistics provide a domain-alignment approach that the experiments support as more effective than prior low-order moment-matching methods.
Takeaways & Limitations
For p ≥4, accuracy decreases as order increases, likely because fifth- and higher-order moments are difficult to estimate accurately with small sample sizes.
Abstract
from arXiv · showhide
Minimizing the discrepancy of feature distributions between different domains is one of the most promising directions in unsupervised domain adaptation. From the perspective of distribution matching, most existing discrepancy-based methods are designed to match the second-order or lower statistics, which however, have limited expression of statistical characteristic for non-Gaussian distributions. In this work, we explore the benefits of using higher-order statistics (mainly refer to third-order and fourth-order statistics) for domain matching. We propose a Higher-order Moment Matching (HoMM) method, and further extend the HoMM into reproducing kernel Hilbert spaces (RKHS). In particular, our proposed HoMM can perform arbitrary-order moment tensor matching, we show that the first-order HoMM is equivalent to Maximum Mean Discrepancy (MMD) and the second-order HoMM is equivalent to Correlation Alignment (CORAL). Moreover, the third-order and the fourth-order moment tensor matching are expected to perform comprehensive domain alignment as higher-order statistics can approximate more complex, non-Gaussian distributions. Besides, we also exploit the pseudo-labeled target samples to learn discriminative representations in the target domain, which further improves the transfer performance. Extensive experiments are conducted, showing that our proposed HoMM consistently outperforms the existing moment matching methods by a large margin. Codes are available at \url{https://github.com/chenchao666/HoMM-Master}
1. Introduction
Unsupervised domain adaptation addresses performance degradation across domains without requiring target labels. HoMM extends distribution matching beyond coarse first- and second-order statistics by aligning higher-order moments and using pseudo-labels for target discrimination.
- 1. Introduction: Unsupervised domain adaptation uses labeled source samples to annotate an unlabeled target domain when repeated target annotation is impractical.The broader goal is learning domain-invariant features by minimizing source–target feature-distribution discrepancy.
- 1. Introduction: MMD and CORAL primarily match first-order mean and second-order covariance statistics, which incompletely characterize complex non-Gaussian deep-feature distributions.Consequently, second-order or lower matching provides only coarse-grained alignment.
- 1. Introduction: Higher-order moment tensors, especially third- and fourth-order tensors, can represent feature distributions more accurately and support finer-grained domain alignment.The paper’s illustration uses point clouds and moment-tensor level sets with different orders to show this representational difference.
- 1. Introduction: HoMM unifies MMD and CORAL while generalizing first- and second-order matching to higher-order tensor matching for domain discrepancy minimization.The contribution emphasizes third- and fourth-order matching as outperforming existing discrepancy-based methods by a large margin.
- 1. Introduction: Pseudo-labels for reliable target samples are used to learn discriminative target-domain clusters, further improving transfer performance.This complements distribution alignment by addressing the lack of target labels.
2. Related Work
Related work addresses domain adaptation through distribution discrepancy metrics, higher-order statistics, adversarial or image-level adaptation, and discriminative target clustering. HoMM is positioned as a higher-order alternative within this landscape, while two-stream CNNs provide a shared-parameter adaptation architecture.
- Learning Domain-Invariant Features: MMD, CORAL, and Wasserstein distance are representative discrepancy metrics for learning domain-invariant features.DAN and JAN use multi-kernel or joint MMD, while CORAL aligns second-order source and target statistics.
- Learning Domain-Invariant Features: Prior work extends distribution alignment through RKHS-based CORAL, geodesic alignment, adversarial training, and pixel-level image-to-image transformation.These approaches target domain invariance through different representations or alignment objectives.
- Higher-order Statistics: CMD matches higher-order central moments, whereas HoMM matches higher-order cumulant tensors for domain alignment.The passage identifies HoMM as closely related to, but distinct from, CMD.
- Higher-order Statistics: Second-order or lower statistics may be insufficient for non-Gaussian feature distributions, motivating higher-order statistical representations.Higher-order statistics have also been used in classical and deep-learning recognition methods.
- Discriminative Clustering: Discriminative target clustering methods include entropy minimization, pseudo-labeling, distance-based metrics, and decision-boundary approaches.They address the paucity of target labels by seeking more discriminative target representations.
- Unsupervised Deep Domain Adaptation: Two-stream CNN adaptation uses shared parameters for source and target streams and aligns representations at the last fully connected layer.This architecture operates source and target samples separately while sharing model weights.
3. Method
HoMM aligns source and target feature distributions by matching higher-order moments, extending first- and second-order methods to arbitrary orders and RKHS representations. The method combines source classification, domain alignment, and target-domain discriminative clustering.
- Overall framework: The two-stream CNN uses tied weights for source and target samples and performs domain alignment at the last fully connected layer.The source domain provides labeled samples, while the target domain provides unlabeled samples.
- Moment matching: HoMM matches p-order tensor powers of adapted-layer activations, with first-order matching equivalent to linear MMD and second-order matching equivalent to Gram-matrix matching.After mean normalization, the centralized Gram matrix becomes the covariance matrix, connecting second-order HoMM to CORAL.
- Moment matching: For p ≥3, HoMM matches higher-order moment tensors to capture non-Gaussian feature distributions and enable finer-grained domain alignment.The paper emphasizes third- and fourth-order moment matching as practical higher-order choices.
- Efficient matching: Direct higher-order tensor construction has O(L^p) space complexity, motivating group matching and random sampling matching as compact alternatives.Group matching reduces complexity to O(n_g ·⌊L/n_g⌋^p), while random sampling reduces it to O(N) and supports arbitrary-order matching.
- RKHS extension: Kernelized HoMM extends higher-order matching into RKHS and becomes equivalent to KMMD when p = 1 using an RBF kernel.The kernel is k(x, y) = exp(−γ∥x − y∥^2).
- Target discrimination: The full objective integrates source classification loss, higher-order domain discrepancy loss, and discriminative clustering of reliable pseudo-labeled target samples.The clustering loss is activated after the total loss stabilizes, with λ_dc initially set to zero.
4. Experiments
Experiments evaluate HoMM on digits, Office-31, and Office-Home adaptation tasks, alongside feature visualization, order selection, parameter sensitivity, and convergence analyses. Higher-order matching and discriminative clustering consistently improve transfer performance, while fourth-order matching is especially strong on office datasets.
- Experimental setup: Experiments cover digits, Office-31, and Office-Home unsupervised domain adaptation datasets using modified LeNet or ResNet-50 backbones.Digits experiments include SVHN→MNIST, USPS→MNIST, and SYN→MNIST transfer tasks.
- Experimental results: 97.2% accuracy on SVHN→MNIST was achieved by third-order KHoMM, improving CORAL by +8%.The third-order HoMM slightly underperformed KHoMM and outperformed fourth-order HoMM on the digits experiments.
- Experimental results: Fourth-order HoMM achieved the best results among moment-matching methods on Office-31, outperforming third-order HoMM.The reported comparison identifies fourth-order matching as stronger than third-order matching on Office-31.
- Experimental results: HoMM significantly outperformed discrepancy-based and adversarial-training methods across the reported transfer tasks.The comparisons include DDC, CORAL, CMD, DANN, ADDA, and CyCADA; JAN was sometimes better than third-order HoMM but not fourth-order HoMM.
- Experimental results: Kernelized HoMM consistently outperformed plain HoMM, while discriminative clustering consistently improved transfer performance across tasks.Entropy regularization helped when test accuracy was high but contributed little or reduced performance when confidence was lower.
- Order analysis: Third- and fourth-order matching outperformed other tested orders; accuracy increased through p=3, then decreased as order increased beyond p=4.The authors attribute weaker fifth-order and higher-order performance partly to inaccurate moment estimation from small sample sizes.
- Feature visualization: HoMM features aligned source and target samples while improving category discrimination more effectively than KMMD and CORAL in the SVHN→MNIST visualization.The source-only features showed substantial domain shift, whereas KMMD and CORAL aligned global distributions but retained many misclassified samples.
- Sensitivity and convergence: The model was sensitive to λdc, converged fastest among compared methods, and showed a test-error mutation when clustering loss Ldc was enabled.Parameter sensitivity was evaluated on SVHN→MNIST and A→W.
5. Conclusion
The paper presents higher-order moment matching for domain alignment, integrating MMD and CORAL while extending moment matching to arbitrary orders. Experiments show that third- and fourth-order matching outperform existing moment-matching methods, with RKHS extension and target-domain clustering further improving adaptation.
- HoMM integrates MMD and CORAL into a unified framework for domain discrepancy minimization.
- HoMM generalizes first- and second-order moment matching to arbitrary-order moment matching.
- Third- and fourth-order moment matching significantly outperform existing moment-matching methods.
- Extending HoMM into RKHS and learning discriminative target-domain clusters further improves adaptation performance.