Source-linked AI summary
Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning
Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, Susanne Saminger-Platz
TL;DR
The paper addresses unsupervised domain adaptation, where target labels are unavailable and existing moment-matching methods do not explicitly match higher-order moments. It proposes Central Moment Discrepancy for hidden activation distributions, proving metric and convergence properties and reporting strong benchmark performance with reduced computational requirements.
Problem
Unsupervised domain adaptation must learn from labeled source data and unlabeled target data, while existing approaches do not explicitly match higher-order moments order by order.
Method
CMD minimizes order-wise differences between higher-order central moments of domain-specific hidden activation distributions, avoiding computationally expensive distance and kernel matrices.
Results
CMD achieves state-of-the-art performance on most Office tasks and outperforms Gaussian-kernel MMD, VFAE, and DANN on most Amazon reviews tasks.
Takeaways & Limitations
CMD's distribution matching is theoretically metric-based and yields convergence in distribution on compact intervals, while its default K = 5 achieves the reported benchmark results.
Takeaways & Limitations
MMD-based approaches require kernel-parameter tuning and kernel-matrix computation, which becomes inefficient for large data sets.
Abstract
from arXiv · showhide
The learning of domain-invariant representations in the context of domain adaptation with neural networks is considered. We propose a new regularization method that minimizes the discrepancy between domain-specific latent feature representations directly in the hidden activation space. Although some standard distribution matching approaches exist that can be interpreted as the matching of weighted sums of moments, e.g. Maximum Mean Discrepancy (MMD), an explicit order-wise matching of higher order moments has not been considered before. We propose to match the higher order central moments of probability distributions by means of order-wise moment differences. Our model does not require computationally expensive distance and kernel matrix computations. We utilize the equivalent representation of probability distributions by moment sequences to define a new distance function, called Central Moment Discrepancy (CMD). We prove that CMD is a metric on the set of probability distributions on a compact interval. We further prove that convergence of probability distributions on compact intervals w.r.t. the new metric implies convergence in distribution of the respective random variables. We test our approach on two different benchmark data sets for object recognition (Office) and sentiment analysis of product reviews (Amazon reviews). CMD achieves a new state-of-the-art performance on most domain adaptation tasks of Office and outperforms networks trained with MMD, Variational Fair Autoencoders and Domain Adversarial Neural Networks on Amazon reviews. In addition, a post-hoc parameter sensitivity analysis shows that the new approach is stable w.r.t. parameter changes in a certain interval. The source code of the experiments is publicly available.
1 INTRODUCTION
Unsupervised domain adaptation uses labeled source data and unlabeled target data, motivating methods that align latent representations across domains. The paper proposes CMD to explicitly match higher-order central moments without expensive kernel or distance-matrix computations.
- Unsupervised domain adaptation transfers information from labeled source domains to unlabeled target domains when collecting target labels is expensive.
- Recent neural domain adaptation methods commonly encourage similarity between domain-specific latent representations.
- CMD explicitly matches higher-order moments for each order and hidden coordinate, extending beyond approaches focused on first moments or weighted sums of raw moments.
- CMD uses central moments to define a metric because distributions on compact intervals are equivalently represented by their moment sequences.
- The paper proves that CMD is a metric and that convergence under CMD implies convergence in distribution on compact intervals.
2 HIDDEN ACTIVATION MATCHING
The model trains a neural classifier on labeled source examples while regularizing hidden activations from source and target samples. Its domain regularizer compares activation distributions, producing more similar hidden activation distributions under the domain-regularized objective.
- The adaptation setting provides labeled source samples and unlabeled target samples, with the goal of minimizing target classification risk without target labels.
- The architecture is a three-layer neural network trained by backpropagation using the domain-regularized objective.
- The network uses bounded hidden activations, such as hyperbolic tangent or sigmoid outputs, mapped into [a, b]^N.
- The training objective combines empirical source loss with a weighted domain-discrepancy regularizer on source and target hidden activations.
- Domain regularization produces more similar hidden activation distributions than optimization with the unregularized objective.
3 RELATED WORK
Prior domain discrepancy methods include Proxy A-distance, MMD, and mean-activation matching, while CMD explicitly matches higher-order central moments in hidden activation space. The paper motivates CMD as a more explicit and computationally efficient alternative for domain matching.
- Existing domain discrepancy methods: Proxy A-distance trains a classifier to discriminate source from target samples, with the discrepancy determined by its generalization error.The approach requires simultaneously training a new classifier with a gradient reversal layer.
- Illustration: Domain-regularized training produces more similar source and target hidden activation distributions than standard-objective training in the illustrated network.Figure 2 compares standard training on the left with domain-discrepancy-regularized training on the right.
- Existing domain discrepancy methods: MMD uses kernel products and, with a Gaussian kernel, requires tuning β and computing a kernel matrix with complexity O(n^2 + nm + m^2).The paper focuses on quadratic-time MMD with Gaussian kernels and transfer-learning-specific cross-validation for parameter tuning.
- Existing domain discrepancy methods: Mean-activation matching minimizes a modified Kullback–Leibler divergence between empirical mean activations, thereby matching only the first moment.The paper identifies this approach as MKL and contrasts it with methods that match higher-order moments.
- CMD contribution: CMD explicitly minimizes differences between higher-order central moments for each moment order and hidden coordinate.Central moments are used because they are translation invariant and have a natural geometric interpretation.
- CMD contribution: CMD has lower computational complexity than quadratic-time MMD and achieves state-of-the-art performance on most Office domain adaptation tasks.It also outperforms networks trained with MMD, variational fair autoencoders, and domain adversarial neural networks on Amazon reviews.
4 CENTRAL MOMENT DISCREPANCY (CMD)
The paper defines CMD as a central-moment distance for probability distributions on compact intervals and uses an empirical version as a neural-network domain regularizer. Its theoretical properties depend on compactness and, for joint-distribution matching under marginal approximation, independence assumptions.
- CMD metric: CMD is defined from differences between the first expectation term and higher-order central-moment vectors of distributions on a compact interval.Central moments are translation invariant and have a natural geometric interpretation.
- CMD metric: CMD satisfies the metric properties on probability distributions over compact intervals, including nonnegativity, symmetry, triangle inequality, and identity of indiscernibles.The paper uses moment-sequence equivalence on compact intervals to establish the remaining metric property.
- Convergence: Minimizing CMD between hidden activation distributions is intended to make the corresponding domain-specific cumulative distribution functions approach each other.The paper connects this behavior to convergence in distribution through a stated theorem.
- Empirical regularizer: The empirical CMD regularizer computes marginal central moments, limits the moment order with K, and replaces expectations with empirical estimates.These are the three approximation steps identified in the definition and its discussion.
- Scope and approximations: Under marginal approximation, dependent coordinates do not provide the joint-distribution matching guarantees obtained for independent marginals.Convergence in distribution is then established only for the marginals, while computational complexity becomes linear in the number of samples.
- Scope and approximations: Higher-order marginal CMD terms have an upper bound that strictly decreases with moment order and converges to zero.The paper reports that higher-order terms can therefore contribute less to the overall distance value.
- Empirical regularizer: Replacing expected values with empirical expectations yields a consistent estimate through the weak law of large numbers and the continuous mapping theorem.The paper also states that CMD gradients can be implemented with simple aggregations of standard-function derivatives.
5 EXPERIMENTS
Experiments evaluate CMD on Amazon reviews and Office domain-adaptation benchmarks against regularized and state-of-the-art baselines. CMD performs strongly across tasks and shows limited sensitivity to several hyperparameter and hidden-dimension changes, unlike MMD in the reported analyses.
- Datasets: The experiments use Amazon reviews with 12 category-transfer tasks and Office with six transfer tasks across amazon, webcam, and dslr domains.Amazon reviews contain four product categories; Office contains 2817 images in its largest domain and 31 classes.
- Experimental setup: Amazon reviews models compare Source Only, MMD, CMD, VFAE, and DANN using 2000 labeled source and 2000 unlabeled target training examples per task.The evaluations are repeated 10 times with different data shuffles, reporting mean accuracies and standard deviations.
- Amazon reviews results: CMD attains the highest accuracy in 9 of 12 Amazon reviews tasks and the second-best accuracy in the remaining 3.On the remaining tasks, the difference from the best method is smaller than the standard deviation across data shuffles.
- Office results: CMD outperforms related state-of-the-art methods on all but two Office tasks, where it performs similarly, and improves AdaBN by more than 3.2% in average accuracy.The Office comparison uses a CMD model with K = 5 and λ = 1 on VGG16 latent representations.
- Parameter sensitivity: CMD accuracy ratios differ by less than 0.5% for K = 5 versus K ∈ {3, 4, 6, 7}, while K = 1 and K = 2 often produce worse accuracy.The experiments therefore report similar results for K ≥ 3 and prefer higher K values over K = 1 or K = 2.
- Parameter sensitivity: CMD accuracy improvement remains between 4% and 6% across tested hidden dimensions without an observed decrease, whereas MMD improvement decreases as hidden dimensionality increases.The reported MMD behavior requires additional tuning of λ and β for accurate performance; MMD is also more sensitive to parameter changes than CMD.
6 CONCLUSION AND OUTLOOK
The paper concludes that CMD supports domain-invariant representation learning with strong benchmark performance and stable parameter behavior, while identifying broader MMD comparisons as future work.
- CMD achieves state-of-the-art performance on most Office tasks and outperforms Gaussian-kernel MMD, VFAE, and DANN on most Amazon reviews tasks.
- The reported results use the default setting K = 5, and classification accuracy is not sensitive to K for K ≥3.The authors conclude that computationally expensive hyper-parameter selection is unnecessary.
- The experimental comparison uses Gaussian-kernel MMD with a quadratic time estimate.
- Future work includes evaluating other MMD kernels, parameter-selection procedures, and linear-time estimators, alongside applications to generative models.The authors also plan to investigate the approximation quality of the empirical CMD estimate.
A THEOREM PROOFS
The appendix establishes theoretical properties of CMD for probability distributions on compact intervals and derives bounds using central moments of bounded random variables.
- Theorem 1: Theorem 1 concerns probability distributions on a compact interval and the CMD defined in equation (5).
- Theorem 1: If CMD is zero, equality of all centered moments leads to equality of the distributions under compact support.The proof uses equality of joint moment generating functions for mean-centered random variables.
- Theorem 2: Theorem 2 states that CMD convergence for distributions on a compact interval implies convergence in distribution.The proof proceeds through convergence of centered moments, moment generating functions, and means.
- Moment bounds: The central moments c_k(X) are vectors of kth-order sample central moments of the marginal distributions.
- Moment bounds: The appendix bounds central-moment differences using bounded random variables on [a, b] and known bounds for absolute central moments.