Source-linked AI summary
Domain Adaptive Neural Networks for Object Recognition
Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang
TL;DR
Object recognition suffers when source and target domains have mismatched distributions, while representation-learning approaches for this problem had not been well studied. The paper introduces DaNN, embedding MMD regularization in supervised training to align hidden representations. Experiments report good domain-adaptation performance on SURF features and raw pixels, and DAE-pretrained DaNN outperforms SVM-based baselines, GFK, and TSC on almost all Office domain pairs.
Problem
Domain adaptation seeks to recognize objects when training and test distributions differ, a setting where traditional supervised learning cannot guarantee good performance.
Method
DaNN embeds Maximum Mean Discrepancy as a regularizer in supervised back-propagation to encourage similar hidden-layer representation distributions across domains.
Results
DaNN performs well on the Office image set using SURF features and raw pixels, while DAE-pretrained DaNN outperforms SVM-based baselines, GFK, and TSC on almost all domain pairs.
Takeaways & Limitations
MMD regularization is reported as effective for domain adaptation in neural networks on the evaluated Office image set.
Takeaways & Limitations
Raw-pixel performance remains below SURF-feature performance, and the assumed universal Gaussian kernel may not detect every underlying distribution mismatch.
Abstract
from arXiv · showhide
We propose a simple neural network model to deal with the domain adaptation problem in object recognition. Our model incorporates the Maximum Mean Discrepancy (MMD) measure as a regularization in the supervised learning to reduce the distribution mismatch between the source and target domains in the latent space. From experiments, we demonstrate that the MMD regularization is an effective tool to provide good domain adaptation models on both SURF features and raw image pixels of a particular image data set. We also show that our proposed model, preceded by the denoising auto-encoder pretraining, achieves better performance than recent benchmark models on the same data sets. This work represents the first study of MMD measure in the context of neural networks.
1 Introduction
The paper addresses domain adaptation for object recognition when training and test distributions differ, including changes in viewpoints, backgrounds, and transformations. It proposes using MMD regularization within neural-network training to align latent representations across domains.
- Domain adaptation is needed because probability-distribution mismatch between training and test samples can undermine object recognition in real-world settings.
- Differences in viewpoints, backgrounds, and transformations can arise when labeled data for the target environment are limited or unavailable.
- The Office dataset is a standard benchmark for domain adaptation, commonly using SURF descriptors as model inputs.
- SURF features require careful engineering and can increase complexity during real-time feature extraction.
- Representation learning reduces dependence on manual feature engineering, but its use for distribution-mismatch problems had not been well studied.
- The proposed model embeds MMD in supervised back-propagation to reduce mismatch between hidden representations from different domains.
2 Preliminaries
The preliminaries introduce MMD, feed-forward neural networks, and denoising auto-encoders as the components underlying the proposed approach. MMD compares distributions from samples, while DAE pretraining learns robust representations from noisy inputs.
- Maximum Mean Discrepancy: MMD measures the difference between two probability distributions using samples without initially estimating their density functions.
- Maximum Mean Discrepancy: With a universal RKHS function class, MMD equal to zero detects discrepancies between the compared distributions.
- Maximum Mean Discrepancy: In domain adaptation, MMD has been used to bring source and target distributions closer in PCA-based and sparse-coding methods.
- Maximum Mean Discrepancy: The paper differs from TCA and TSC by applying MMD regularization within supervised training rather than unsupervised learning.
- Feed Forward Neural Networks: A feed-forward neural network contains input, hidden, and output layers connected by learned weights and biases.
- Feed Forward Neural Networks: The network uses softplus-like rectifier activations and softmax outputs, giving predictions a probabilistic interpretation.
- Denoising Auto-encoder: A denoising auto-encoder learns robust representations by reconstructing clean inputs from noisy counterparts.
- Denoising Auto-encoder: DAE pretraining in this work uses unlabeled images from both source and target domains, with performance evaluated with and without pretraining.
3 Domain Adaptive Neural Networks
DaNN extends a feed-forward neural network by adding MMD regularization to supervised back-propagation. The regularizer encourages source and target hidden representations to become invariant while the supervised objective remains optimized.
- Model: DaNN incorporates MMD as a regularizer in supervised back-propagation to encourage domain-invariant hidden-layer representations.
- Objective: The loss combines the source-domain supervised criterion with an MMD contribution controlled by regularization constant γ.
- MMD Regularization: The method uses a Gaussian kernel for MMD computation, selected because it is treated as universal and useful in practice.
- MMD Regularization: MMD is computed on linear-combination outputs before nonlinear activation, producing a biased estimate of discrepancy between activated hidden representations.
- Optimization: Training alternates standard mini-batched stochastic updates for the supervised loss with full-batched updates of the first-layer weights using the MMD gradient.
- Optimization: The algorithm initializes weight-bias matrices, updates both layers through supervised forward-backward passes, then re-updates the first layer using MMD.
4 Experiments and Analysis
Experiments evaluate DaNN across Office-domain shifts using SURF features and raw pixels, comparing it with baselines and domain-adaptation methods. They also examine DAE pretraining, semi-supervised adaptation, and in-domain performance.
- Experimental setup: DaNN is a shallow 256-node hidden-layer network accepting either raw pixels or SURF features, and is compared with SVM baselines, GFK, TSC, and NN.NN uses the same structure and parameters as DaNN without MMD regularization.
- Experimental setup: Experiments cover six Office domain shifts across amazon, webcam, and dslr, using unsupervised and semi-supervised adaptation settings.The dataset contains 10 object classes and 1,410 instances; domain-shift directions include A →W, W →A, A →D, D →A, W →D, and D →W.
- SURF features: On SURF features, DaNN and TSC outperform the other approaches, while TSC is strongest for webcam-dslr shifts and GFK fails to surpass baselines in several cases.TSC is less efficient than neural-network approaches because of longer feature-extraction time.
- Raw pixels: On raw pixels, DaNN improves accuracy over SVM baselines and NN for every domain pair and performs better than TSC in most cases.TSC matches DaNN on webcam-dslr pairs, which have lower-level mismatch than the other pairs.
- DAE pretraining: DAE pretraining improves NN and DaNN across all domain pairs, generally lowers standard deviations over 30 runs, and makes DAE + DaNN best in almost all cases.The qualitative analysis reports more distinctive filters after pretraining, especially when amazon images are included.
- Additional analysis: With some labeled target images, discrepancies between NN and DaNN become smaller, suggesting that MMD regularization and DAE pretraining are less impactful in semi-supervised adaptation.In-domain NN is best on both training and test images, while domain-mismatch accuracies are often lower for amazon or webcam targets; adaptation nevertheless helps on W →D.
5 Conclusions and Future Work
The DaNN uses MMD regularization to align hidden-layer distributions and performs well on the Office image set, especially with raw pixels. DAE-pretrained DaNN outperforms several baselines across almost all domain pairs, while raw-pixel performance and kernel selection remain open limitations.
- DAE-pretrained DaNN outperforms SVM-based baselines, GFK, and TSC on the Office image set in almost all domain pairs.
- MMD regularization encourages similar hidden-layer representation distributions between domains during supervised back-propagation training.
- DaNN performs well on the Office image set, particularly when raw image pixels are used as inputs.
- Raw-pixel performance remains worse than performance on SURF features, despite the importance of raw-pixel representation learning.
- The Gaussian kernel assumption for MMD may not detect every underlying distribution mismatch in the Office data set.