Source-linked AI summary
Learning unbiased features
Yujia Li, Kevin Swersky, Richard Zemel
TL;DR
The paper addresses learning models across domains and learning identity features invariant to lighting conditions. It uses MMD to align representations and match generated samples to data, with reported gains across applications and realistic generated samples.
Problem
Domain adaptation uses labeled source data and unlabeled target data, with the goal of learning a model that works well on the target domain.
Source: §3.1 · p. 2Method
The paper applies MMD penalties to align domain representations, learn features invariant to lighting or noise, and match generative-model samples to data.
Source: §3.2 · p. 3, §3.3 · p. 4, §3.4 · p. 4Results
The neural network with MMD dominates on most review-domain tasks; test accuracy rose from 72% without MMD to around 82% with a Gaussian-kernel penalty on face recognition.
Source: §3.1 · p. 2, §3.2 · p. 3Takeaways & Limitations
With Gaussian-kernel MMD, examples of each person under different lighting conditions grouped together, although the penalty used only lighting condition.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
A key element in transfer learning is representation learning; if representations can be developed that expose the relevant factors underlying the data, then new tasks and domains can be learned readily based on mappings of these salient factors. We propose that an important aim for these representations are to be unbiased. Different forms of representation learning can be derived from alternative definitions of unwanted bias, e.g., bias to particular tasks, domains, or irrelevant underlying data dimensions. One very useful approach to estimating the amount of bias in a representation comes from maximum mean discrepancy (MMD) [5], a measure of distance between probability distributions. We are not the first to suggest that MMD can be a useful criterion in developing representations that apply across multiple domains or tasks [1]. However, in this paper we describe a number of novel applications of this criterion that we have devised, all based on the idea of developing unbiased representations. These formulations include: a standard domain adaptation framework; a method of learning invariant representations; an approach based on noise-insensitive autoencoders; and a novel form of generative model.
2 Maximum Mean Discrepancy
The paper uses maximum mean discrepancy (MMD) as a sample-based measure of bias in representations. MMD measures distance between distributions and can support two-sample tests; with a universal reproducing kernel Hilbert space, it asymptotically equals zero if and only if the distributions are equal.
- MMD measures distance between two distributions and can be used to perform two-sample tests.Source: §2 · p. 1
- The approaches to learning unbiased features use a sample-based measure of bias in the representation.Source: §2 · p. 1
- With a universal reproducing kernel Hilbert space, MMD is asymptotically zero if and only if P = Q.The kernel trick applies to inner products to use an implicit feature space.Source: §2 · p. 1
3 Applications
Across four applications, the paper uses MMD to shape representations across domains and perturbations, and to match generative samples to data. The reported outcomes include stronger domain-adaptation results, more noise-invariant features, and realistic generated samples.
3.1 Domain Adaptation
- The neural network with an MMD penalty dominates most Amazon review domain-adaptation tasks against SVM, unpenalized neural-network, and TCA baselines.The experiments evaluate adaptation between pairs of four product-category domains; the paper also reports that NN MMD performs better than most baselines with basic word-count features.Source: §3.1 · p. 2
3.2 Learning Invariant Features
- On Yale B face recognition, Gaussian-kernel MMD raises test accuracy from 72% without the penalty to around 82%, while a linear kernel reaches 78%.The penalty aligns hidden representations across five lighting-condition domains; examples of each person under different lighting conditions group together.Source: §3.2 · p. 3, Figure 1 · p. 3
3.3 Noise-Insensitive Autoencoders
- The MMD autoencoder outperforms the other tested autoencoders on noise-invariance measurement, while the denoising autoencoder performs worst.The measure is an SVM's accuracy at distinguishing clean from noisy test-data representations; lower accuracy indicates greater invariance.Source: §3.3 · pp. 3–4, Table 2 · p. 4
3.4 Learning Generative Deep Models
- For generative modeling, the paper directly minimizes MMD between data and model samples, avoiding the minimax formulation's opposing objectives; generated samples are described as realistic.The method uses backpropagation through the MMD penalty and network to bring model samples close to the data.Source: §3.4 · p. 4
4 Appendix
The appendix reports domain-adaptation experiments and details, relates contractive auto-encoder penalties to MMD, and describes auto-encoder and generative-model training. It also reports cleaner, more localized MMD auto-encoder filters.
4.1 More Details of the Domain Adaptation Experiments
- Experiments use 10 random splits of four-domain product reviews, with source labels, unlabeled target training data, and target test accuracy for evaluation.The neural network applies MMD to its second hidden layer; TCA uses linear kernels and RBF SVMs on normalized 64-dimensional features.Source: §4.1 · p. 6
- NN MMD learns useful domain-adaptation features with word-count inputs and outperforms baselines on most tasks; TF-IDF boosts all methods.The appendix reports results across source-target pairs.Source: §4.1 · p. 7
4.2 Relationship Between Contractive Auto-Encoders and MMD
- The contractive auto-encoder penalty can be expressed as an MMD penalty with a linear kernel.A Taylor expansion around elementary input vectors connects the summed hidden-unit changes to the contractive penalty.Source: §4.2 · p. 7
4.3 Auto-Encoder Training Details
- The auto-encoder uses a stochastic contraction penalty by sampling Bernoulli-corrupted inputs, then applies Gaussian-kernel MMD between original and perturbed hidden representations.Each network has one layer of 100 sigmoid units and is trained with stochastic gradient descent and momentum.Source: §4.3 · p. 7
4.4 Auto-Encoder Weight Filters
- MMD auto-encoder weight filters are cleaner and more localized than those of the other variants.Figure 3 visualizes the weight matrices for each auto-encoder variety.Source: §4.4 · p. 7, Figure 3 · p. 8
4.5 Training Details for the Generative Experiments
- The generative model uses 32 stochastic hidden units with independent uniform priors on [−1, 1], a two-layer ReLU mapping, and a 784-unit sigmoid output.New latent samples are generated from the prior after every 200 updates to the mapping.Source: §4.5 · p. 7