PaperlayerView exact revision on arXiv

Source-linked AI summary

Moment Matching for Multi-Source Domain Adaptation

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, Bo Wang

arXiv:1812.01754v4cs.CV

TL;DR

The paper addresses the limited single-domain assumption and scarce large-scale benchmarks in unsupervised domain adaptation. It introduces DomainNet, M3SDA for moment-based multi-source alignment, and supporting theory; experiments report performance exceeding single- and multi-source baselines, with a stated alignment limitation.

  • Problem

    UDA commonly assumes one source domain, while practical multi-source adaptation faces source heterogeneity, source-to-source shifts, and limited large-scale multidomain datasets.

  • Method

    M3SDA transfers knowledge from multiple labeled sources to an unlabeled target by aligning moments between source-target and source-source feature distributions.

  • Results

    M3SDA outperforms all multi-source baselines and the best single-source domain adaptation method on multi-source benchmarks.

  • Takeaways & Limitations

    DomainNet provides a challenging benchmark with six domains, 345 categories, and approximately 0.6 million images for evaluating future single- and multi-source UDA methods.

  • Takeaways & Limitations

    M3SDA assumes that aligning p(x) automatically aligns p(y|x), an assumption that may not hold in practice.

Abstract

from arXiv · show

Conventional unsupervised domain adaptation (UDA) assumes that training data are sampled from a single domain. This neglects the more practical scenario where training data are collected from multiple sources, requiring multi-source domain adaptation. We make three major contributions towards addressing this problem. First, we collect and annotate by far the largest UDA dataset, called DomainNet, which contains six domains and about 0.6 million images distributed among 345 categories, addressing the gap in data availability for multi-source UDA research. Second, we propose a new deep learning approach, Moment Matching for Multi-Source Domain Adaptation M3SDA, which aims to transfer knowledge learned from multiple labeled source domains to an unlabeled target domain by dynamically aligning moments of their feature distributions. Third, we provide new theoretical insights specifically for moment matching approaches in both single and multiple source domain adaptation. Extensive experiments are conducted to demonstrate the power of our new dataset in benchmarking state-of-the-art multi-source domain adaptation methods, as well as the advantage of our proposed model. Dataset and Code are available at \url{http://ai.bu.edu/M3SDA/}.

1. Introduction

The paper targets multi-source domain adaptation, where multiple labeled source domains and their mutual shifts challenge transfer to an unlabeled target. It introduces DomainNet, M3SDA, and moment-matching theory to address these challenges.

  • Domain adaptation performance degrades on novel visual domains because domain shift separates training and test distributions.
  • Multi-source domain adaptation is more practical than single-source UDA but involves multiple source domains, source-to-source shifts, and limited large-scale datasets.
  • DomainNet contains six domains, 345 categories, and approximately 0.6 million images, addressing benchmark saturation from small or limited datasets.
  • M3SDA aligns source domains with the target and with one another by directly matching moments of deep feature distributions.
  • The paper extends theoretical analysis to moment-based divergence for source-target adaptation and related moment-matching approaches.

2. Related Work

Prior domain adaptation work includes discrepancy-, adversarial-, and reconstruction-based single-source methods, while multi-source methods address combinations of source hypotheses or domain discriminators. Moment-matching methods align distribution moments, but existing MSDA analyses may overlook source-to-source shift.

  • Single-source UDA: Single-source UDA methods are grouped into discrepancy-based, adversarial-based, and reconstruction-based approaches.
  • Single-source UDA: Discrepancy-based methods reduce domain shift using measures including MMD, correlation alignment, KL divergence, and H divergence.
  • Multi-Source Domain Adaptation: Multi-source domain adaptation uses multiple training sources and includes theoretical bounds, weighted source hypotheses, and applied domain-discriminator models.
  • Multi-Source Domain Adaptation: Existing applied MSDA works align source domains with the target but neglect domain shift among source domains.
  • Moment Matching: Moment-matching methods range from first- and second-moment matching to high-moment regularization and infinite-dimensional covariance alignment.

3. The DomainNet dataset

DomainNet is a large, diverse domain-adaptation dataset spanning six visual domains and 345 object categories. Its construction combines web collection, manual filtering, and processed Quickdraw data.

  • DomainNet contains six domains, each covering 345 common-object categories, making it the largest domain-adaptation dataset described in the paper.
  • The six domains are Clipart, Infograph, Painting, Quickdraw, Real, and Sketch, representing clipart, infographics, paintings, drawings, photographs, and sketches.
  • Images for five domains were collected through image-search queries combining category and domain names, then manually filtered for outliers.
  • Twenty annotators spent approximately 2,500 hours filtering outliers, retaining images accepted by both assigned annotators.
  • Quickdraw data were converted from temporal point sequences into complete drawings using B-splines, with 500 images selected per category and 172.5k images total.

4. Moment Matching for Multi-Source DA

M3SDA transfers knowledge across multiple labeled source domains by matching feature-distribution moments while training domain-specific classifiers. Its theoretical analysis motivates aligning source domains with one another, and experiments report strong digit-classification performance.

  • M3SDA: M3SDA maps source and target data into a common latent space, minimizes a moment-related distance, and trains N classifiers on labeled source domains.The overall objective combines moment matching with softmax cross-entropy losses weighted by trade-off parameter λ.
  • M3SDA-β: M3SDA-β uses two classifiers per source domain and alternates classification training, classifier-pair discrepancy maximization, and feature-generator discrepancy minimization.The classifier discrepancy is defined as the L1-distance between paired classifier outputs on the target domain.
  • Ensemble Schema: The testing ensemble averages classifier outputs or weights them by source-only accuracy measuring closeness between each source and the target.The weighted prediction assumes the N-th domain is the target and normalizes source-specific accuracies into a weight vector.
  • Theoretical Insight: Moment-based theory bounds target error using pairwise moment divergences between the target and each source, while source-source divergences lower-bound the combined target-source divergences.For two sources, dCM_k(D1, DT) + dCM_k(D2, DT) ≥ dCM_k(D1, D2), motivating pairwise source alignment.
  • Experiments: M3SDA achieves 86.13% accuracy and M3SDA-β achieves 87.65% accuracy on the reported Digits Classification Results, outperforming the listed baselines.These are the reported overall accuracies in Table 2.

5. Experiments

Experiments evaluate M3SDA across digit, Office-Caltech10, and DomainNet benchmarks, with results showing strong performance but difficulty under many categories and some negative transfer.

  • Experimental setup: 714 experiments cover digit classification, Office-Caltech10, and DomainNet image recognition tasks.The evaluation used 24 GPUs and exceeded 21,440 GPU-hours.
  • Digit recognition: M3SDA-β reaches 87.65% average accuracy on digit recognition, exceeding M3SDA’s 86.13% average and other baselines.All experiments use the same network architecture and report means and standard deviations over five runs.
  • Office-Caltech10: M3SDA-β achieves 96.4% average accuracy on Office-Caltech10, compared with 96.1% for M3SDA.These experiments use ResNet-101 pretrained on ImageNet.
  • DomainNet: M3SDA and M3SDA-β achieve 41.5% and 42.6% accuracy on DomainNet, outperforming single-best, source-combine, and multi-source baselines.When quickdraw is the target, multi-source methods can perform worse than single-source and source-only baselines, indicating negative transfer.
  • Effect of category number: Model performance drops rapidly as the number of categories increases, with performance becoming worst above 150 categories in one reported comparison.The category-count analysis considers settings including painting→real, infograph→real, sketch→clipart, and quickdraw→clipart.

6. Conclusion

The paper concludes that DomainNet provides a challenging large-scale benchmark and that M3SDA aligns multiple sources with the target effectively. Its theory and experiments support moment matching for multi-source adaptation.

  • Conclusion: DomainNet is presented as the largest collected and annotated domain adaptation dataset, with notable domain gaps and many categories.The authors intend it to support evaluation of future single- and multi-source UDA methods.
  • Conclusion: M3SDA aligns multiple source domains with the target using a moment-matching approach and derives an error bound under cross-moment divergence.The model incorporates moment matching into a deep neural network trained end-to-end.
  • Conclusion: Experiments show M3SDA outperforms all reported multi-source baselines and the best single-source domain adaptation method.The conclusion frames these experiments as evidence for the proposed model’s performance across multi-source benchmarks.
  • Conclusion: The appendix provides ablations, cross-moment-divergence definitions, theorem proofs, dataset splits, baselines, and category-number analyses.These materials extend the theoretical, experimental, and dataset documentation.

A. Ablation Study

The ablation study separates source-source and source-target alignment to identify their contributions. Matching source distributions to the target is the main performance factor, while source-source matching adds further gains.

  • Ablation Study: Matching source distributions to the target is the key factor behind the observed performance boost.This compares the source-target alignment component against the baseline in the ablation study.
  • Ablation Study: Matching source domains with each other further improves performance beyond source-target moment matching alone.The results empirically support source-domain alignment as important for multi-source domain adaptation.
  • Cross-moment divergence: The cross-moment divergence formalizes the distance between probability measures through their k-th order cross-moments.For domains D and D′, the paper uses dCM_k(D, D′) as shorthand for the corresponding measure divergence.

C. Proof of Theorem 1

The proof establishes a multi-source target-error bound using empirical weighted source error, VC-dimension generalization, and cross-moment divergence terms. It also explains why the moment-based terms provide theoretical motivation for moment matching.

  • Theorem 1 bounds target error for an empirical minimizer over labeled samples from multiple source domains with a fixed source-weight vector.The theorem uses a hypothesis space with finite VC dimension and source sample sizes proportional to β_jm.
  • The proof combines uniform-convergence inequalities, the empirical-minimizer property, Lemma 4, and the triangle inequality for classification error.Lemma 4 supplies the concentration step for empirical α-weighted source error.
  • The bound’s second term controls the gap between empirical and true α-weighted error through VC-dimension-based generalization error.This term is described as an upper bound on the difference between empirical error and true expected error.
  • The bound’s final summation controls the difference between α-weighted source error and target error, with constants that can be meaningfully bounded.The proof notes that the constants are bounded using Lemma 3 and the Weierstrass Approximation Theorem.
  • The target-error bound explicitly incorporates cross-moment divergence dCM_k(D_j, D_T), providing theoretical motivation for moment matching in single- and multi-source adaptation.The paper presents this as the first domain-adaptation target-error bound explicitly incorporating a moment-based divergence between source(s) and target domains.

D. Details of Digit Experiments

The digit experiments use a convolutional and fully connected feature extractor, then visualize DAN and M3SDA-β features with t-SNE. The visualizations are reported to show more discriminative and more compact M3SDA-β representations.

  • The digit-experiment feature extractor contains three convolutional layers followed by two fully connected layers.The convolutional layers are conv1, conv2, and conv3; the fully connected layers are fc1 and fc2.
  • t-SNE visualizations compare DAN and M3SDA-β features in the mm,mt,up,sy→sv and A,D,W→C tasks.Different markers and colors denote different categories in the visualizations.
  • The visualizations report that M3SDA-β learns more discriminative features than DAN in the compared digit and object-domain tasks.
  • M3SDA-β features form more compact clusters than DAN features in the A,D,W→C visualization.The paper presents cluster compactness as evidence from the t-SNE comparison.

F. Effect of Category Number

The category-number analysis evaluates four DomainNet transfers as categories increase from 20 to 345. All models perform well with few categories but decline rapidly as the category count grows.

  • The experiments increase categories from 20 to 345 in painting→real, infograph→real, sketch→clipart, and quickdraw→clipart settings.
  • All models perform well with few categories, but their performance drops rapidly as the number of categories increases.Figure 5 describes the decline as significant across the four selected UDA settings.
  • Self-Ensembling performs well with a small number of categories but is not suitable for large-scale domain adaptation.
  • The reported ResNet baselines use column-wise source domains and row-wise target domains, with averages over domains and all 30 source-target combinations.Table 7’s caption identifies green values as row or column averages and red values as the average across all combinations.

J. Dataset Statistics

DomainNet contains six domains and 345 categories, with categories organized across 24 divisions. The dataset statistics are presented in detailed tables, alongside a time-consumption table for the model and baseline.

  • DomainNet contains 6 distinct domains, 345 categories, and approximately 0.6 million images.
  • Table 10, Table 11, and Table 12 provide detailed DomainNet dataset statistics.
  • Table 9 reports time consumption for the proposed model and the baseline.

K. Toy Experiment

The toy experiment examines whether aligning source-domain distributions improves alignment among two source domains and one target domain. With a two-layer fully connected backbone, the model produces better source–target alignment when source distributions are matched.

  • Toy Experiment: The source domains are not automatically aligned because multi-source data are not i.i.d., motivating simultaneous alignment of source domains.The target error bound includes pairwise divergences between source domains.
  • Toy Experiment: Matching source-domain distributions improves alignment among the two source domains and the target domain in the toy experiment.The domains are represented as yellow and green sources and a blue target.
  • Toy Experiment: The experiment uses two source domains, one target domain, and a 2-layer fully connected neural network backbone.
  • Toy Experiment: Training and testing time are benchmarked in PyTorch using minibatches of 16 and images sized 224 x 224 on an NVIDIA TITAN X GPU.The benchmark environment uses CentOS 7, eight Intel CORE i7 processors, CUDA 8.0, and CuDNN 5.15.
Loading 1812.01754v4…