PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Principled Unsupervised Learning

Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, Danilo Rezende, Tim Lillicrap, Oriol Vinyals

arXiv:1511.06440v2cs.LG

TL;DR

The paper addresses the mismatch between unsupervised objectives and supervised tasks by introducing the Output Distribution Matching cost, which aligns prediction and label distributions. ODM is consistent with perfect supervised mappings and supports learning with very few labels, including restricted one-shot domain adaptation. Its applicability depends on a sufficiently large output space and sufficiently similar shared structure between input and output domains.

  • Problem

    Unsupervised objectives can be unrelated to the supervised task of interest, making it difficult to improve supervised performance without labelled examples.

    Source: §1 · p. 1

  • Method

    The Output Distribution Matching cost constrains a model by matching the distribution of its predictions to the distribution of unlabelled outputs.

    Source: §1 · p. 2

  • Results

    The method achieved 4.7% test error on MNIST classification using 4 labelled examples, and demonstrated one-shot domain adaptation in certain settings.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    ODM provides a supervised-consistent approach to unsupervised learning and can train useful mappings without input-output examples in supported settings.

    Source: §1 · p. 2, §7 · p. 8

    ODM requires a large output space and sufficiently similar shared structure between the input and output spaces, while one-shot adaptation works only in restricted settings.

    Source: §6.2 · p. 8, §6.1 · p. 8

Abstract

from arXiv · show

General unsupervised learning is a long-standing conceptual problem in machine learning. Supervised learning is successful because it can be solved by the minimization of the training error cost function. Unsupervised learning is not as successful, because the unsupervised objective may be unrelated to the supervised task of interest. For an example, density modelling and reconstruction have often been used for unsupervised learning, but they did not produced the sought-after performance gains, because they have no knowledge of the supervised tasks. In this paper, we present an unsupervised cost function which we name the Output Distribution Matching (ODM) cost, which measures a divergence between the distribution of predictions and distributions of labels. The ODM cost is appealing because it is consistent with the supervised cost in the following sense: a perfect supervised classifier is also perfect according to the ODM cost. Therefore, by aggressively optimizing the ODM cost, we are almost guaranteed to improve our supervised performance whenever the space of possible predictions is exponentially large. We demonstrate that the ODM cost works well on number of small and semi-artificial datasets using no (or almost no) labelled training cases. Finally, we show that the ODM cost can be used for one-shot domain adaptation, which allows the model to classify inputs that differ from the input distribution in significant ways without the need for prior exposure to the new domain.

1 INTRODUCTION

Unsupervised learning struggles because its objectives may not reflect the supervised task of interest. The paper introduces ODM, which matches prediction and label distributions and can improve supervised performance when outputs are sufficiently large, while also supporting restricted one-shot domain adaptation.

Motivation

  • Unsupervised objectives often fail to improve supervised tasks because they lack information about which task the available data should support.Density modeling and reconstruction are cited as examples whose objectives can be unrelated to the supervised cost.Source: §1 · p. 1
  • The paper motivates the approach through speech recognition, where linguistic plausibility of outputs can be evaluated without knowing whether each output is correct for its input.This generalizes the idea of using known output statistics to constrain an unknown input-output mapping.Source: §1 · p. 1

ODM cost

  • The output distribution matching (ODM) cost matches the distributions of model outputs and labels using unlabelled samples from both domains.The constraint is valid because any function that perfectly maps every possible input to its label also satisfies the matching condition.Source: §1 · p. 2
  • ODM is most informative when the output space is large, because matching distributions becomes trivial when the output space is small.Under the large-output condition, the constraint can act like an additional labelled training case conveying substantial information.Source: §1 · p. 2

Applications

  • ODM also supports one-shot domain adaptation by aligning a substantially different test distribution with the training distribution through a generative model.The capability is not universal and applies only in certain restricted settings.Source: §1 · p. 2

2 RELATED WORK

Related work uses output statistics as an unsupervised signal and studies mappings between distributions, while neural-network pre-training develops objectives intended to transfer across tasks.

Matching statistics

  • Matching output statistics has been used for unsupervised OCR, machine translation, and lexicon induction, including language-decipherment formulations related to this work.The cited applications infer mappings from observed output statistics without direct paired supervision.Source: §2 · p. 3

Domain adaptation

  • Domain-adaptation methods that map a new distribution back to the training distribution are closely related to the ODM objective.Both approaches transform distributions toward a target training-domain distribution.Source: §2 · p. 3

Unsupervised pre-training

  • Neural-network pre-training typically first optimizes an unsupervised objective and then finishes training with the supervised cost.Examples include autoencoder-style objectives, k-means, Skip-gram, and Skip-thought vectors.Source: §2 · p. 3

3 METHODS

The methods develop generative and autoencoder-based ways to optimize ODM by matching an output distribution to a target distribution without supervision. The dual autoencoder shares inner weights to align representations across domains, while GAN training offers another way to enforce distribution matching.

3.1 ODM COSTS AS GENERATIVE MODELS

  • ODM training seeks a conditional Pθ(y|x) whose induced output distribution matches the marginal distribution of y.When P(x) models the input marginal well, minimizing Ey∼D[−log Pθ(y)] is equivalent to the ODM cost up to an additive constant.Source: §3.1 · p. 3
  • The xyh-model introduces a continuous shared variable h while allowing x and y to be continuous or discrete, and it can also use labelled pairs when available.With limited model capacity, it is intended to represent structure common to both domains in P(h) and domain-specific structure in conditional components.Source: §3.1 · pp. 3–4

3.2 THE DUAL AUTOENCODER

  • The paper proposes the dual autoencoder after early xyh-model experiments produced poor results; it uses two autoencoders with shared innermost weights.Sharing W1...WN encourages compatible representations for the two modalities, aligning x with y without a direct supervised signal.Source: §3.2 · p. 4
  • The dual autoencoder minimizes reconstruction losses for x and y using two autoencoders whose shared weights encourage cross-domain compatibility.The cost combines expected losses L1(x, x′) and L2(y, y′), with L1 and L2 chosen as appropriate loss functions.Source: §3.2 · p. 4

3.1 ODM COSTS AS GENERATIVE MODELS

  • GAN training enforces ODM by requiring F(x) to produce samples statistically indistinguishable from the target distribution.The method uses the GAN procedure to train the unsupervised objective Distr[F(x)] = Distr[y].Source: §3.3 · p. 4

4 EXPERIMENTS

The experiments test unsupervised correspondence learning with dual autoencoders on permuted images and ciphered text, then use ODM-based GAN training for MNIST classification with few labels. Dual autoencoders recover several mappings without input-output examples, while GAN training reaches low MNIST error with four labelled examples but is sensitive to training choices.

4.1 DUAL AUTOENCODERS ON MNIST PERMUTATION TASK

  • The dual autoencoder recovered the unknown pixel permutation without input-output examples on the MNIST permutation task.The task uses MNIST digits whose pixels share an unknown permutation, and the model learns that permutation unsupervised.Source: §4.1 · p. 5
  • The sigmoid dual autoencoder performed far better than the ReLU and Tanh variants on the MNIST permutation task across a wide range of hyperparameters.The authors suggest slower sigmoid learning may have helped by allowing the shared network capacity to grow gradually.Source: §4.1 · p. 5, Figure 3 · p. 5

4.1.1 DUAL AUTOENCODER FOR CIPHERS

  • Character- and word-level dual autoencoders achieved perfect symbol-mapping identification in fewer than 10,000 updates with sigmoid nonlinearities.The character architecture was 100-25-25-100, while the word architecture was 1000-100-100-1000 with a 1000-symbol vocabulary.Source: §4.1.1 · p. 5
  • Dual-autoencoder success depended on similar underlying statistics: training on English and Spanish corpora failed to recover correspondences between similarly meaningful words.The result indicates sensitivity to systematic differences between the two data distributions.Source: §4.1.1 · pp. 5–6

4.2 GENERATIVE ADVERSARIAL NETWORKS FOR MNIST CLASSIFICATION

  • GAN training was fragile and highly sensitive to learning-rate choices despite being extremely rapid.The experiments specify separate learning rates for the generator, discriminator, and supervised squared-loss objective.Source: §4.2 · pp. 6–7
  • 4.7% test error was achieved on MNIST classification using 4 labelled examples, whereas 2 labelled digits yielded 72% test error.GAN training of the ODM cost alone was insufficient for an accurate classifier, so the reported result combines ODM training with four labelled MNIST examples.Source: §4.2 · p. 7

5 ONE-SHOT LEARNING AND DOMAIN ADAPTATION

The paper proposes one-shot domain adaptation by inferring a mapping from each out-of-distribution test sample back to the training distribution, then applying the existing classifier. On inverted MNIST, this generative approach reconstructs most test digits correctly, while the method remains restricted to settings where the mapping is identifiable from one sample.

One-shot domain adaptation

  • A single test sample from an unknown distribution can identify a simple mapping back to the training distribution, enabling classification without additional training.The mapping is inferred separately for each test case, allowing the classifier to handle inputs whose statistics differ substantially from training data.Source: §5 · p. 7

Scope

  • The approach succeeds only when the likelihood has few parameters that one sample can uniquely determine, so one-shot adaptation is not universal.The paper characterizes the demonstrated results as preliminary and restricts the broader claim to settings where the cross-domain mapping is identifiable.Source: §5 · pp. 7–8

Experimental setting

  • The experiment uses MNIST for training and 1-MNIST for testing, where each pixel intensity is replaced by 1 −u, making the test distribution substantially different.Networks trained directly on MNIST cannot correctly classify 1-MNIST instances, establishing the domain shift addressed by the experiment.Source: §5 · p. 7

Results

  • About 5% of test digits were reconstructed incorrectly in the one-shot domain-adaptation demonstration.Figure 5 compares original data, the inverted test observation, the inferred training-domain image, and the reconstructed observation; the CNN is inferred separately for each test case.Source: Figure 5 · p. 7, §5 · p. 7

Method

  • The method infers x for each 1-MNIST test image by optimizing log Pθ(y|x)P(x), using a generative model of MNIST and a CNN likelihood.P(x) is modeled with an LSTM, while Pθ(y|x) is a small convolutional network; optimization uses L2 regularization, 10^4 Adagrad steps, and five random restarts.Source: §5 · p. 7

6 DISCUSSION

The discussion explains when ODM can determine or improve the supervised function: the output space must be informative, and the function class must not freely reshape distributions. Its broader use is limited by the need for sufficiently similar shared structure between input and output spaces.

6.1 WHEN IS SUPERVISION NEEDED?

  • If the function class can map any input distribution to any output distribution, ODM cannot recover the function, although consistency may still eliminate unsuitable functions.ODM also becomes weak when the output space is small because the matching constraint conveys too little information.Source: §6.1 · p. 8
  • ODM can completely determine the best supervised function when input and output distributions contain long-range dependencies that a local function cannot modify.In that setting, matching output distributions constrains the function strongly enough to recover the desired mapping without a supervised objective.Source: §6.1 · p. 8

6.2 LIMITATIONS

  • ODM-based training requires a large output space and sufficiently similar shared hidden structure between the input and output spaces.The paper notes that the simple techniques used may fail when those shared structures are insufficiently similar, while better generative-model training could broaden applicability.Source: §6.2 · p. 8

7 CONCLUSIONS

The paper concludes that ODM is a generic unsupervised objective consistent with the supervised cost and provides evidence that it can train functions without input-output examples. Its broader applicability remains dependent on improved optimization methods.

7 CONCLUSIONS

  • ODM provides a generic unsupervised-learning approach consistent with the supervised cost function.The authors present evidence that ODM can train functions without input-output examples, although they could not reliably train an attention model with it.Source: §7 · p. 8

Related papers

Loading 1511.06440v2…