Source-linked AI summary
Deep Transfer Network: Unsupervised Domain Adaptation
Xu Zhang, Felix Xinnan Yu, Shih-Fu Chang, Shengjin Wang
TL;DR
Labeled data can be costly or unavailable, and domain adaptation methods using neural networks have rarely explicitly matched dataset distributions. DTN uses shared feature-extraction and discrimination layers to match marginal and conditional distributions; the authors report effectiveness across benchmarks, especially large-scale problems, with linear computational complexity.
Problem
Labeled training data can be expensive or unavailable, and few general domain-adaptation works explicitly match dataset distributions through neural networks.
Source: §1 · pp. 1–2Method
DTN uses shared feature-extraction layers to match marginal distributions and discrimination layers to match conditional distributions by classifier transduction.
Source: §1 · p. 2Results
DTN has O(n) computational complexity and, across benchmark datasets, is reported effective and significantly better than competitive methods, especially on large-scale problems.
Source: §5 · p. 11Takeaways & Limitations
The authors conclude that neural-network structure supports efficient modeling and optimization of distribution matching for large-scale domain adaptation.
Source: §5 · p. 11The paper considers the setting where both source-target marginal and conditional distributions differ.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Domain adaptation aims at training a classifier in one dataset and applying it to a related but not identical dataset. One successfully used framework of domain adaptation is to learn a transformation to match both the distribution of the features (marginal distribution), and the distribution of the labels given features (conditional distribution). In this paper, we propose a new domain adaptation framework named Deep Transfer Network (DTN), where the highly flexible deep neural networks are used to implement such a distribution matching process. This is achieved by two types of layers in DTN: the shared feature extraction layers which learn a shared feature subspace in which the marginal distributions of the source and the target samples are drawn close, and the discrimination layers which match conditional distributions by classifier transduction. We also show that DTN has a computation complexity linear to the number of training samples, making it suitable to large-scale problems. By combining the best paradigms in both worlds (deep neural networks in recognition, and matching marginal and conditional distributions in domain adaptation), we demonstrate by extensive experiments that DTN improves significantly over former methods in both execution time and classification accuracy.
1. Introduction
Domain adaptation addresses performance drops when a model trained on one dataset is applied to a related dataset with different feature or label distributions. DTN uses neural-network layers to match marginal and conditional distributions, and the paper reports improved accuracy and computation time.
1. Introduction
- Domain adaptation matters because labeled training data can be costly or unavailable, and performance may drop when source and target distributions differ.The paper describes domain adaptation as transferring knowledge from a labeled source dataset to a related but non-identical target dataset.Source: §1 · p. 1
- Prior approaches include instance reweighting and subspace learning, while classifier transduction adapts classifiers using conditional-distribution regularization.Methods that match both marginal and conditional distributions have shown strong benchmark performance, but some have O(n^2) or O(n^3) complexity.Source: §1 · pp. 1–2
- Across extensive experiments, DTN outperforms state-of-the-art methods in computation time and accuracy, with a reported 28.95% improvement on USPS/MNIST.The introduction reports this accuracy improvement for the USPS/MNIST dataset.Source: §1 · p. 2
- DTN has O(n) optimization complexity, compared with O(n^2) and O(n^3) for cited alternatives, making it more suitable for large-scale domain adaptation.Here, n is the number of training data points.Source: §1 · p. 2
- DTN’s shared feature-extraction layers match marginal distributions, while its discrimination layers match conditional distributions through classifier transduction.The network is designed to model and match both distributions.Source: §1 · p. 2
2. Deep Transfer Network
DTN uses neural-network layers to align source and target feature marginals in a shared representation and classifier transduction to match conditional distributions. Its formulation addresses unsupervised adaptation when both distributions differ, using target pseudo-labels because target labels are unknown.
2.1. Problem Formulation
- DTN considers unsupervised adaptation with labeled source data, unknown target labels, and differences in both marginal and conditional distributions.The goal is to use the available data to minimize prediction error on target samples.Source: §2.1 · p. 2
2.2. Preliminary
- The framework uses empirical MMD to measure distribution discrepancy and distinguishes feature-extraction layers from a discrimination layer, instantiated in MLPs as the first l−1 fully connected layers and a final softmax layer.The paper notes that its approach can extend to other neural-network types.Source: §2.2 · pp. 2–3
2.3. Matching Marginal Distributions
- DTN minimizes MMD between source and target features at the output of the first l−1 layers, bringing their feature distributions close in a shared subspace.The source and target feature matrices are combined to model this marginal distance.Source: §2.3 · p. 3
2.4. Matching Conditional Distributions
- The discrimination layer measures conditional-distribution distance with classifier-output posteriors and uses iteratively updated pseudo-labels because target labels are unknown.A non-transfer neural network serves as the base classifier for target pseudo-labels.Source: §2.4 · pp. 3–4
2.5. Final Objective Function
- DTN’s final objective combines distribution-matching terms with negative log-likelihood, and minimizes the network weights using marginal and conditional regularization parameters.The formulation is described as extendable beyond MLPs, including to CNNs and DBNs.Source: §2.5 · pp. 4–5
3. Optimization of Deep Transfer Network
DTN optimization uses mini-batches to match source and target distributions and iteratively updates target labels. Its backpropagation cost grows linearly with the number of training samples.
3.1. The Optimization Procedure
- Backpropagation computes parameter gradients from the objective, and stochastic gradient descent updates the network weights.The derivatives use gradients of both the feature representation and classifier output with respect to each weight.Source: §3.1 · p. 5
- Mini-batches replace inefficient full-dataset MMD computation, and matching each batch is intended to match the datasets overall.The batches contain source and target samples; the procedure computes MMD for each batch rather than over all samples at once.Source: §3.1 · p. 5
- The smaller dataset is randomly sampled and copied to equalize dataset sizes, then batches draw half their samples from each domain.Batch size should be large enough for each batch to reflect whole-dataset variance.Source: §3.1 · p. 6, Figure 2 · p. 6
- Iteratively updating target pseudo-labels during training significantly improves DTN performance, especially on large-scale datasets.The optimization begins with target labels predicted by a baseline neural network.Source: §3.1 · p. 6
3.2. Computational Complexity
- O(n) total backpropagation cost lets DTN scale linearly with training samples, unlike Transfer Joint Matching (O(n2)) and Adaptation Regularization (O(n3)).The paper defines n as the larger dataset size.Source: §3.2 · p. 6
4. Experiments
Evaluation covers Office-Caltech’s 12 cross-domain pairs and two large-scale settings. Across these tests, DTN improves classification accuracy and shows nearly linear execution-time growth as training data increase.
4.1. Domain Adaptation on Small-Scale Datasets
- DTN averages 51.68% accuracy across Office-Caltech’s 12 transfers, 3.73% above ARRLS and 6.60% above its non-transfer MLP baseline.It is best on 6 of 12 transfers and is less than 1% below the best baseline on two others.Source: Table 1 · p. 7, §4.1 · p. 8
4.2. Domain Adaptation on Large-Scale Datasets
- DTN reaches 81.04% on USPS/MNIST and 73.60% on CIFAR/VOC, gaining 28.95% and 1.87% over ARRLS, respectively.DTN performs significantly better than other baselines on USPS/MNIST; the paper attributes this to its CNN base classifier.Source: Table 2 · p. 8, §4.2 · p. 10
- DTN’s execution time grows almost linearly with training-sample count, while ARRLS grows faster; ARRLS needs over 100GB memory versus DTN’s less than 3GB.DTN also has competitive speed against ARRLS, the best-performing baseline.Source: Figure 4 · p. 9, Table 3 · p. 10, §4.2 · p. 10
4.3. Parameter Sensitivity
- Classification accuracy remains strong for λ and µ chosen from [1, 100]; the experiments use λ = µ = 10 by default.λ controls marginal-distribution matching and µ controls conditional-distribution matching.Source: §4.3 · p. 10
- Bigger batches lead to better performance, but limited GPU memory constrains batch size; CIFAR/VOC batches of 4,000 and 8,000 failed.The chosen sizes are 4,000 for USPS/MNIST and 2,000 for CIFAR/VOC.Source: §4.3 · p. 10
- Classification accuracy increases steadily and eventually converges as target labels are updated; large-scale experiments use T = 20.T controls how many times target labels are updated during training.Source: §4.3 · p. 10
5. Conclusion
Deep Transfer Network combines neural networks with adaptation of both marginal and conditional distributions. It has O(n) computational complexity and performs effectively across benchmark datasets, significantly outperforming competitive methods, especially on large-scale problems.
5. Conclusion
- DTN combines neural networks with domain adaptation by matching both marginal and conditional distributions.Source: §5 · p. 11
- Neural-network structure supports efficient modeling and optimization of the distribution-matching process.Source: §5 · p. 11
- DTN significantly outperforms competitive methods, especially on large-scale problems, and is effective across a variety of benchmark datasets.The paper reports computational complexity O(n), making DTN suitable for large-scale domain adaptation.Source: §5 · p. 11