Moment Matching for Multi-Source Domain Adaptation

1. Introduction

• Multi-source adaptation is challenging because single-source UDA methods may be ineffective when there are multiple sources, source domains shift from one another, and large-scale multidomain datasets are scarce.Source: §1 · p. 1
• DomainNet contains 569,010 images across 345 categories and six domains, addressing the shortage of large-scale multidomain benchmarks.The authors evaluate state-of-the-art single-domain adaptation methods on DomainNet.Source: §1 · p. 2
• The paper reports that M3SDA outperforms existing single- and multi-source approaches on DomainNet and existing datasets.The introduction also reports that many existing datasets show saturation, with methods achieving about 90% accuracy on Office, 99% on Digit-Five, and 92% on Syn2Real.Source: §1 · p. 2
• M3SDA aligns source domains with the target and with one another by directly matching moments of deep feature distributions, rather than using complex adversarial training procedures [45].The authors report this as the first empirical demonstration that aligning source domains benefits multi-source adaptation.Source: §1 · p. 2
• The paper extends theoretical analysis to moment-based divergence between source and target domains, providing insights for moment-matching approaches in domain adaptation [14].Source: §1 · p. 2

Domain Adaptation Datasets

• DomainNet contains about 600k images across 345 categories and six domains, addressing the small scale and environment-specific limits of earlier benchmarks.Earlier datasets include Office and Office-Caltech10, while more recent benchmarks focus on settings such as offices, homes, and museums.Source: §2 · p. 2

Single-source UDA

• Single-source UDA methods are grouped into discrepancy-based approaches such as MMD, adversarial approaches [24] [40], and reconstruction-based approaches [3] [10].The passage notes that reconstruction uses an encoder-decoder or a GAN discriminator, including dualGAN [46], cycle-GAN [50], disco-GAN [16], and CyCADA.Source: §2 · pp. 2–3

Multi-Source Domain Adaptation

• Prior MSDA work includes theoretical bounds and weighted source hypotheses and applied methods such as DCTN [45], normalized solutions [14], and Domain Selection Machine.The proposed model instead matches all distributions through their moments and provides a proof for why matching moments works in multi-source adaptation.Source: §2 · p. 3

Moment Matching

• Prior moment-matching approaches align second moments [39], RKHS covariance, high moments [47], or GAN-based feature and distribution moments [29] [22] [20]; this work focuses on multiple domains.The passage also describes MMD as matching the first moments of two distributions.Source: §2 · p. 3

• DomainNet contains six domains, each with 345 common-object categories: Clipart, Infograph, Painting, QuickDraw, Real, and Sketch.Source: §3 · pp. 3–4
• 423.5k images remain from 1.2 million web-crawled images after 20 annotators spent around 2,500 hours filtering, requiring agreement by two annotators per image.Average images per category are around 150 for Clipart and Infograph, 220 for Painting and Sketch, and 510 for Real.Source: §3 · p. 4
• QuickDraw’s raw stroke points are connected into drawings with the B-spline algorithm [5], and 500 images per category yield 172.5k images.Source: §3 · p. 4

4. Moment Matching for Multi-Source DA

• M3SDA maps source and target data into a shared feature space, minimizes moment-related distances, and trains one source classifier per domain with cross-entropy loss.The overall objective also includes a trade-off parameter λ.Source: §4 · p. 4
• M3SDA-β uses two classifiers per domain to align p(y|x) as well as p(x), periodically maximizing target-output discrepancy between classifier pairs and then minimizing it through feature-generator training, following [38].The discrepancy is the L1 distance between classifier outputs.Source: §4 · pp. 4–5
• At test time, M3SDA can average classifier outputs or weight them using source-only accuracy as a proxy for each source’s closeness to the target.The weighted outputs use a weight vector whose entries sum to one.Source: §4 · p. 5

4.1. Theoretical Insight

• Theorem 1 states that the learned hypothesis’s target-error upper bound depends on the k-th-order cross-moment divergence between the target domain and each source domain, motivating alignment of each target–source pair’s moments.The paper extends the analysis in to derive this moment-based multi-source domain-adaptation bound.Source: §4.1 · p. 5
• 86.13% accuracy for M3SDA and 87.65% accuracy for M3SDA-β on digit classification, outperforming other baselines by a large margin.The table abbreviates MNIST, USPS, SVHN, Synthetic Digits, and MNIST-M as mt, up, sv, sy, and mm.Source: Table 2 · p. 6
• For two sources, the sum of their moment divergences to the target is at least their pairwise divergence, motivating alignment among source pairs as well as target–source pairs.The passage notes that perfect target alignment to every source is not possible when the sources themselves are unaligned.Source: §4.1 · p. 6

5. Experiments

• The evaluation spans digit and image-recognition tasks and totals 714 experiments; λ was fixed at 0.5, with little observed performance variation from 0.1 to 1.Runs used a 24-GPU cluster and exceeded 21,440 GPU-hours.Source: §5 · p. 6

5.1. Experiments on Digit Recognition

• 87.65% average accuracy for M3SDA-β versus 86.13% for M3SDA on digit recognition; results on MNIST-M were lower, possibly due to negative transfer.All experiments used the same network architecture and reported means and standard deviations across five runs.Source: §5.1 · p. 6

5.2. Experiments on Office-Caltech10

• 96.4% average accuracy for M3SDA-β versus 96.1% for M3SDA on Office-Caltech10; the authors report their models achieve the best performance reported on this dataset.Experiments used ResNet-101 pre-trained on ImageNet.Source: §5.2 · pp. 6–7, Table 4 · p. 7

5.3. Experiments on DomainNet

• In painting→real experiments, methods performed well with few categories but declined at different rates as categories increased from 20 to 345; SE [7] was best with few categories and worst above 150.The figure illustrates this trend for painting→real; similar plots are reported in the appendix.Source: Figure 4 · p. 7, §5.3 · p. 8
• 42.6% accuracy for M3SDA-β versus 41.5% for M3SDA on DomainNet, outperforming single-best UDA, source-combine, and multi-source baselines.When quickdraw is the target, multi-source methods perform worse than source-only baselines, indicating negative transfer.Source: §5.3 · pp. 7–8, Table 5 · p. 8

6. Conclusion

• DomainNet is presented as the largest domain adaptation dataset, with notable domain gaps and many categories, for evaluating future single- and multi-source UDA methods.The paper reports that it collected, annotated, and evaluated the dataset.Source: §6 · p. 8
• M3SDA outperforms all multi-source baselines and the best single-source domain adaptation method on multi-source adaptation benchmarks.The method aligns multiple source domains with the target, incorporates moment matching into an end-to-end deep neural network, and derives an error bound under cross-moment divergence.Source: §6 · p. 8
• M3SDA aligns multiple source domains with the target using moment matching, with a derived error bound under cross-moment divergence.The moment-matching component is incorporated into a deep neural network trained end to end.Source: §6 · p. 8

8. Appendix

• The appendix covers source-source alignment ablations, the cross-moment divergence definition, and the proof and further discussion of Theorem 1.It also details experiments on the Digit-Five dataset and feature visualization with t-SNE.Source: §8 · p. 11
• The appendix examines how category counts affect state-of-the-art model performance and describes the ResNet baselines and DomainNet train/test split.It also provides DomainNet image samples and detailed dataset statistics.Source: §8 · p. 11
• The appendix includes a toy experiment on the importance of source-domain alignment and compares M3SDA's time consumption with a baseline.Source: §8 · p. 11

A. Ablation Study

• Source-to-target moment matching is the key factor in the performance boost, while matching source domains with each other further improves performance.The ablation compares S-S only, S-T only, and their gains over baseline.Source: A. Ablation Study · p. 11, Table 6 · p. 11
• The ablation results indicate that aligning source domains is essential for multi-source domain adaptation.Source: A. Ablation Study · p. 11

B. Cross-moment Divergence

• Cross-moment divergence is defined for two probability measures over a compact domain, with a k-th order divergence between them.The supplied passage introduces the definition but does not include its formula.Source: B. Cross-moment Divergence · p. 11
• For readability, the paper writes the divergence between domains D and D′ as dCM k(D, D′), meaning the divergence between their probability measures.Source: B. Cross-moment Divergence · p. 11

C. Proof of Theorem 1

• Lemma 3 approximates classification disagreement using cross-moment divergence with an arbitrary ε tolerance, and the proof notes that its approximation constants can be bounded.The constants can be bounded through the Weierstrass approximation result.Source: C. Proof of Theorem 1 · pp. 11–12
• Lemma 4 adapts Hoeffding’s inequality to empirical α-weighted source error and supports the uniform-convergence bound for a finite-VC hypothesis space.The lemma applies to fixed source weights and labeled samples drawn from the source domains.Source: C. Proof of Theorem 1 · p. 12
• For a finite-VC hypothesis space and fixed source weights, Theorem 1 bounds target error using VC-based generalization and cross-moment divergence, the latter making this the first such target-error bound in domain adaptation.The bound’s final summation controls the difference between the weighted-source error and target error.Source: C. Proof of Theorem 1 · pp. 12–13
• The proof combines the empirical-minimizer property, Lemma 4, and the classification-error triangle inequality to derive target-error control.The triangle inequality relates classification errors across labeling functions.Source: C. Proof of Theorem 1 · pp. 12–13

D. Details of Digit Experiments

• In the digital experiments, the feature extractor comprises three convolutional layers and two fully connected layers; the feature generator arranges them with batch normalization, ReLU activations, pooling, and dropout.Source: D. Details of Digit Experiments · p. 13
• The feature generator comprises conv-batch-normalization-ReLU-pooling blocks for conv1 and conv2, then a conv3 block, followed by fc1 with batch normalization, ReLU, and dropout, and fc2 with batch normalization and ReLU.Source: D. Details of Digit Experiments · p. 13
• The digit classifier is a single fully connected layer, fc3 (2048, 10).Source: D. Details of Digit Experiments · p. 13
• In the mm,mt,sv,sy→up setting, training errors for multiple classifiers decrease despite frequent deviations.This learning procedure is visualized in Figure 7.Source: D. Details of Digit Experiments · p. 13
• The Moment Discrepancy loss decreases steadily, while the paper reports gradual convergence of the MD and cross-entropy losses.Source: D. Details of Digit Experiments · p. 13

E. Feature visualization

• M3SDA-β learns more discriminative features than DAN in the mm,mt,up,sy→sv task, as shown by the t-SNE comparison.The paper identifies this as an observation from Figure 6(a) and 6(b).Source: E. Feature visualization · p. 13
• In A,D,W→C, M3SDA-β feature clusters are more compact than DAN clusters in the t-SNE visualization.The paper says this indicates a more desirable discriminative property.Source: E. Feature visualization · p. 13

F. Effect of Category Number

• Performance drops significantly as category count increases across four UDA settings, from 20 to 345 categories.The settings are painting→real, infograph→real, sketch→clipart, and quickdraw→clipart.Source: F. Effect of Category Number · p. 13, Figure 5 · p. 14
• Self-Ensembling [7] performs well with few categories but is unsuitable for large-scale domain adaptation.Source: F. Effect of Category Number · p. 13

G. ResNet baselines

• MCD [38] and SE [7] perform worse than their ResNet source-only baselines, indicating negative transfer in both scenarios.The paper offers the large category count as a preliminary possible explanation, while noting that investigating the cause is beyond its scope.Source: G. ResNet baselines · p. 14, Table 7 · p. 14

H. Train/Test Split

• DomainNet is split into training and testing sets using a 70%/30% ratio.Table 8 also reports the average number of images per category.Source: H. Train/Test Split · p. 14, Table 8 · p. 14

I. Image Samples

• The paper samples and presents images from all six DomainNet domains: clipart, infograph, painting, quickdraw, real, and sketch.The samples appear in Figures 9–14, respectively.Source: §I · p. 15

J. Dataset Statistics

• DomainNet contains 6 distinct domains, 345 categories, and ∼0.6 million images.Tables 10–12 are identified as showing detailed dataset statistics.Source: J. Dataset Statistics · p. 15
• The 345 categories are drawn from 24 divisions.These include Furniture, Mammal, Tool, Cloth, Electricity, Building, Office, Human Body, Road Transportation, Food, Nature, Cold Blooded, Music, Fruit, Sport, Tree, Bird, Vegetable, Shape, Kitchen, Water Transportation, Sky Transportation, Insect, and Others.Source: J. Dataset Statistics · p. 15
• Tables 10, 11, and 12 present detailed statistics of the DomainNet dataset.Source: J. Dataset Statistics · p. 15

K. Toy Experiment

• The source domains are not i.i.d., so they are not automatically aligned with each other.The passage argues that perfect alignment of the target with every source is not possible when the sources themselves are misaligned.Source: K. Toy Experiment · p. 15
• The target-error bound’s last term is lower bounded by pairwise source-domain divergences, motivating moment alignment between each source pair.Source: K. Toy Experiment · p. 15
• The toy experiment uses two source domains and one target domain, with a 2-layer fully connected neural network as its backbone.The sources are denoted by yellow and green, and the target by blue.Source: K. Toy Experiment · p. 15
• The source and target domains are better aligned when the model matches source-domain distributions.Source: K. Toy Experiment · p. 15

L. Time Consumption

• The time-consumption section reports training and testing time benchmarks run with PyTorch on a CentOS 7 server using a NVIDIA TITAN X GPU and 8 Intel CORE i7 processors.The setup used CUDA 8.0 and CuDNN 5.15.Source: L. Time Consumption · p. 15
• The time benchmarks use minibatches of 16 and images sized 224 x 224.Source: L. Time Consumption · p. 15