PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Learning via Meta-Learning

Kyle Hsu, Sergey Levine, Chelsea Finn

arXiv:1810.02334v6cs.LGcs.AIcs.CVstat.ML

TL;DR

Unsupervised learning typically uses proxy objectives to learn representations for downstream tasks, while meta-learning usually requires labeled data and predefined task distributions. This paper combines unsupervised embeddings with automatically constructed tasks to learn a transferable procedure without labels, improving downstream learning across multiple image datasets and task settings.

  • Problem

    Unsupervised objectives are rough proxies for downstream performance, while meta-learning depends on large labeled datasets and hand-specified task distributions.

  • Method

    The method embeds unlabeled data, constructs classification tasks from clustered partitions, and applies meta-learning to acquire a transferable learning procedure.

  • Results

    Across four image datasets and varied downstream classification tasks, the method consistently improves learning over the unsupervised embeddings used to construct its meta-training tasks.

  • Takeaways & Limitations

    Simple embedding-based task construction can make unsupervised meta-learning effective for a wide range of human-specified few-shot classification tasks.

  • Takeaways & Limitations

    Performance depends substantially on the quality of task-generating embeddings, and the gap from supervised meta-learning may persist because supervised tasks are human-designed for evaluation.

Abstract

from arXiv · show

A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data. Many prior unsupervised learning works aim to do so by developing proxy objectives based on reconstruction, disentanglement, prediction, and other metrics. Instead, we develop an unsupervised meta-learning method that explicitly optimizes for the ability to learn a variety of tasks from small amounts of data. To do so, we construct tasks from unlabeled data in an automatic way and run meta-learning over the constructed tasks. Surprisingly, we find that, when integrated with meta-learning, relatively simple task construction mechanisms, such as clustering embeddings, lead to good performance on a variety of downstream, human-specified tasks. Our experiments across four image datasets indicate that our unsupervised meta-learning approach acquires a learning algorithm without any labeled data that is applicable to a wide range of downstream classification tasks, improving upon the embedding learned by four prior unsupervised learning methods.

1 INTRODUCTION

The paper proposes unsupervised meta-learning to optimize directly for learning downstream tasks from limited labeled data. It automatically constructs tasks from unlabeled embeddings and finds that simple clustering-based task generation supports effective learning across human-specified tasks.

  • Unsupervised objectives are only rough proxies for downstream performance because downstream objectives require supervision.
  • Meta-learning seeks efficient learning procedures but typically requires large labeled datasets and hand-specified task distributions.
  • The proposed method learns a transferable learning procedure from raw unlabeled observations for new human-specified tasks with modest labeled datasets.
  • Tasks are constructed by embedding unlabeled data, partitioning it into categorizations, and deriving classification tasks for meta-learning.
  • Across four image datasets, the method consistently supports downstream character, object, and facial-attribute classification without labels or hand-designed meta-training tasks.
  • The procedure is illustrated with clustered embeddings producing two-cluster classification tasks using one training example per cluster.

2 UNSUPERVISED META-LEARNING

The method frames unsupervised learning as acquiring a transferable procedure from unlabeled data, then automatically constructs structured classification tasks for meta-training. It uses embeddings, repeated clustered partitions, and image-based meta-learning to support few-shot downstream tasks.

  • Problem setting: The problem setting uses unlabeled data during meta-training and targets M-way downstream classification from K labeled examples per class, with K between 1 and 50.
  • Problem setting: Unsupervised meta-training does not access downstream task information beyond an upper bound N on the number of classes, so it must learn a general prior.
  • Task construction: The approach constructs classification tasks from unlabeled data and learns how to solve them efficiently, replacing human labels with structured partitions.
  • Task construction: Random partitions create diverse but unstructured tasks, producing no consistency between training and query data and failed meta-learning.
  • CACTUs: CACTUs clusters unsupervised embeddings with k-means, repeatedly applying random dimension scaling to generate diverse partitions and sampling clusters uniformly for tasks.
  • CACTUs: The method meta-learns on images rather than embeddings so the learned procedure can adapt from raw inputs to each evaluation task.

3 RELATED WORK

CACTUs combines prior unsupervised embedding methods with meta-learning rather than replacing them. It applies established meta-learning algorithms to automatically constructed tasks and learns representations intended for downstream use.

  • The method aims to acquire a transferable learning procedure without labels, addressing the central unsupervised learning problem.
  • Its task-construction procedure runs embedding learning, repeated randomly scaled k-means clustering, partition sampling, and cluster sampling for classification tasks.
  • CACTUs uses prior unsupervised methods to learn embeddings, then constructs meta-learning tasks from those embeddings.
  • Unlike semi-supervised learning, the problem setting does not require unlabeled data to share underlying labels with downstream labeled data.
  • The approach applies MAML and prototypical networks to tasks constructed without supervision.

4 EXPERIMENTS

The experiments test whether CACTUs-based unsupervised meta-learning benefits downstream learning across embeddings, task types, oracle comparisons, and task-construction choices. CACTUs-MAML generally outperforms direct embedding-based methods, transfers across task settings, and benefits from diverse structured tasks, while performance depends on the quality of the task distribution and the downstream setting.

  • Experimental design: The experiments evaluate CACTUs against embedding-based methods, supervised meta-learning oracles, multiple embeddings, task types, and task-construction alternatives.They use Omniglot, miniImageNet, and CelebA, with additional analyses of overlapping distributions and miniImageNet pre-training on ILSVRC 2012.
  • Benefit of meta-learning: CACTUs-MAML consistently achieves more successful downstream performance than unsupervised alternatives, including methods trained directly on the task-generating embedding.CACTUs-ProtoNets shows the same result for 1-shot tasks, but ProtoNets are less effective when meta-training and meta-testing shots differ.
  • Different embedding spaces: CACTUs works across embedding methods: the strongest prior embedding generally corresponds to the strongest unsupervised meta-learner.ACAI leads on Omniglot, while DeepCluster leads on miniImageNet.
  • Applicability to different tasks: CACTUs-MAML learns an effective prior across character, object, and facial-attribute tasks and supports M-way K-shot settings with M ≤ N and K ≤ 50.Although models are trained on N-way 1-shot unsupervised tasks, MAML transfers to varied downstream task sizes.
  • Oracle: The penalty for lacking ground-truth labels ranges from substantial to severe, depending on downstream-task difficulty.Easier tasks with fewer classes or more supervision incur less penalty than harder tasks.
  • Task construction ablation: Random tasks and pixel-space clusters produce priors less useful than other methods, whereas CACTUs-MAML outperforms Hyperplanes-MAML across tested margins.Multiple partitions help but are not strictly necessary; nonzero hyperplane margins are crucial for miniImageNet but not Omniglot.
  • Note on overfitting: Multiple partitions usually regularize the meta-learner by reducing overfitting and increasing the applicability of the learned prior.The authors observe little substantial overfitting to unsupervised meta-training tasks, partly because many tasks can be generated.

5 DISCUSSION

The method improves downstream learning by applying meta-learning to tasks generated from unsupervised embeddings, while its effectiveness depends on embedding quality and task alignment. Its scope extends beyond visual classification, but important limitations remain regarding arbitrary class definitions and dataset structure.

  • Empirical findings: Meta-learning on embedding-based tasks improves downstream human-specified tasks across benchmark datasets, task difficulties, and embedding-learning methods.The experiments fix key hyperparameters across settings.
  • Empirical findings: CACTUs performance depends significantly on the quality and nature of task-generating embeddings.Improved embeddings may provide more diverse and distinctive clusterings.
  • Limitations: K-means clusters are unlikely to approximate arbitrary human-designed class definitions universally.The authors identify finding ineffective downstream cases as future work for clarifying practical scope.
  • Limitations: Results on CelebA suggest effectiveness beyond perfectly balanced benchmark classes, although highly unstructured unlabeled datasets remain insufficiently characterized.MNIST, Omniglot, and miniImageNet have balanced classes, while CelebA lacks neatly balanced classes or attributes.
  • Broader scope: The approach derives a downstream-task-oriented representation from an existing unsupervised representation and may extend to regression, speech, language, and reinforcement learning.These domains are identified as future applications rather than evaluated results here.
  • Related methods: The compared embedding methods include ACAI, BiGAN, DeepCluster, and InfoGAN, spanning reconstruction, generative modeling, discriminative clustering, and information-theoretic objectives.The methods collectively cover several major unsupervised-learning frameworks.

APPENDIX B DATASET INFORMATION

The appendix describes the Omniglot and miniImageNet datasets and the construction of supervised meta-learning tasks, alongside CelebA’s attribute-based binary tasks. CelebA evaluation uses 5-shot tasks because 1-shot tasks may be ill-defined when other attributes vary.

  • Datasets: Omniglot contains 1623 characters with 20 examples each, split into 1100, 100, and 423 characters for meta-training, validation, and testing.Alphabet identity is ignored when making the splits.
  • Datasets: miniImageNet contains 100 classes with 600 examples each, using 64/16/20 meta-training, validation, and testing class splits.The images are predominantly natural and realistic.
  • Task construction: Omniglot and miniImageNet N-way K-shot tasks sample N classes and K + Q examples per class, then assign randomly permuted one-hot labels.The same construction is used for supervised meta-learning and evaluation tasks.
  • Task construction: CelebA tasks are binary classifications defined by three attributes and an ordering of three Boolean values.Each class shares the specified attributes internally and differs from the other class on them.
  • Evaluation: CelebA assessment uses only 5-shot tasks because varying non-defining attributes can make 1-shot tasks ill-defined.

APPENDIX C TASK CONSTRUCTION VIA RANDOM HYPERPLANES

The random-hyperplane procedure partitions embedding space into candidate classes for meta-training tasks, while margins and minimum-size pruning reduce ill-defined or unusable subsets. Implementation samples combinations from precomputed hyperplanes to obtain the desired number of tasks.

  • Partition generation: A non-zero margin discards embeddings near hyperplane boundaries, reducing ambiguities caused by separating nearby points or grouping distant points.The margin is applied through signed point-plane distances.
  • Partition generation: Random hyperplanes partition embedding space into subspaces that define candidate classes for N-way meta-training tasks.The procedure uses H = ⌈log2 N⌉ hyperplanes to create sufficiently many subsets.
  • Partition generation: The procedure prunes subsets with fewer than R = K_m-tr + Q members and rejects partitions with fewer than N remaining subsets.
  • Implementation: The implementation pre-computes 1000 hyperplanes and generates partitions by sampling combinations and intersecting their associated subsets.It fixes 100 tasks per partition and chooses the number of partitions according to the desired meta-training tasks.

APPENDIX D MNIST EXPERIMENTS

The MNIST appendix evaluates 10-way tasks under a setup resembling semi-supervised learning, reports embedding-cluster matching results for ACAI and InfoGAN, and examines limitations with fewer clusters. CACTUs-MAML offers little benefit over cluster matching with stronger embeddings and remains below state-of-the-art semi-supervised methods in the best cases.

  • Setup: MNIST uses 70,000 handwritten digit examples, preserves the 60,000/10,000 split, and evaluates 10-way classification tasks.All digits appear in both meta-training and meta-testing, making the setting essentially semi-supervised without a fixed label permutation.
  • Embedding baselines: 96.00% accuracy is achieved by ACAI embedding cluster matching, compared with 96.83% for the InfoGAN embedding using categorical argmax.These are upper bounds for embedding cluster matching.
  • Additional evaluation: CACTUs-MAML is also evaluated on embeddings using a two-hidden-layer model with 64 units per layer and ReLU activation.Other MAML hyperparameters remain unchanged from the referenced configuration.
  • Limitations: With k = 10 and better ACAI or InfoGAN embeddings, CACTUs-MAML provides little benefit over embedding cluster matching.The reduced k setting is used deliberately to examine limitations.
  • Limitations: Even in its best cases, CACTUs-MAML falls short of state-of-the-art semi-supervised learning methods.

APPENDIX E HYPERPARAMETERS AND ARCHITECTURES

The appendix summarizes the architectures, optimization choices, and task-construction hyperparameters used for MAML and ProtoNets experiments.

  • E.1 MAML: MAML uses a four-block convolutional architecture for MNIST and Omniglot, with 32 filters per convolutional layer for consistency across datasets.The outer optimizer is Adam and the inner optimizer is SGD.
  • Architectures and implementations: ProtoNets follow the authors’ architecture and publicly available implementation, while MAML builds on the corresponding MAML codebase.These choices preserve the prior-work architectures and implementations used for the experiments.
  • E.1 MAML: MAML evaluation fixes five query examples per class to prevent batch normalization from exploiting batch statistics.Using one query per class produced significant but spurious accuracy improvements through a process-of-elimination strategy.
  • E.2 ProtoNets: ProtoNets use a four-block convolutional architecture with convolution, BatchNorm, ReLU, and max-pooling in each block.The embedding is the flattened output of the final block, and optimization uses Adam without a learning-rate scheduler.

E.4 USE OF UNSUPERVISED LEARNING METHODS

The appendix describes the unsupervised learning baselines used to generate embeddings and reports a failure mode for ACAI on larger image datasets.

  • ACAI: ACAI is run on MNIST and Omniglot with the authors’ architecture, image padding, and a 256-dimensional embedding.The method is not successfully run on 64 × 64 miniImageNet and CelebA.
  • ACAI: ACAI’s pixel-wise reconstruction loss on miniImageNet and CelebA emphasizes dominant features such as background color and pose.The resulting clusters cover only a limited range of factors and do not produce a useful downstream prior.
  • BiGAN: BiGAN uses a 50-dimensional latent prior for MNIST and a 200-dimensional prior for Omniglot, miniImageNet, and CelebA.The implementation follows the BiGAN authors’ dataset-specific architectures and latent specifications.
  • DeepCluster: DeepCluster is run on miniImageNet and CelebA using a 4096-dimensional output embedding reduced to 256 dimensions with PCA and whitening.The embedding is taken from the input to the linear output layer.
  • InfoGAN: InfoGAN is evaluated only on MNIST using a latent code combining a 10-way categorical distribution with a 2-dimensional uniform distribution.Image embeddings are obtained from the recognition network.

APPENDIX F EXPERIMENTAL RESULTS

The appendix reports classification accuracy over 1000 human-labeled test tasks for MNIST, Omniglot, miniImageNet, and CelebA.

  • Experimental protocol: Classification accuracy is averaged over 1000 tasks generated from human-specified labels on each testing split.Reported values include 95% confidence intervals.
  • MNIST: MNIST results are organized in a table comparing unsupervised and meta-learning approaches under digit-classification tasks.The table defines k as clusters per partition and P as the number of partitions used during meta-learning.
  • Omniglot: Omniglot results evaluate character classification and report embedding and task-construction settings alongside accuracy and confidence intervals.The table includes embedding dimension, hidden units, clusters, partitions, and hyperplane margin.
  • miniImageNet: miniImageNet results evaluate object classification using embedding and task-construction configurations, including clustering and hyperplane margins.Results are averaged over 1000 tasks with 95% confidence intervals.
  • CelebA: CelebA results report facial-attribute classification accuracy averaged over 1000 tasks with 95% confidence intervals.The table records embedding dimension, hidden units, cluster count, and partition count where applicable.

APPENDIX G IMAGENET EXPERIMENTS

The ImageNet-scale experiments increase the unlabeled meta-training set and evaluate transfer to miniImageNet tasks, comparing CACTUs with embedding-based methods.

  • ImageNet-scale meta-training: ImageNet-scale meta-training uses over 1,200,000 images from 1000 ILSVRC 2012 classes while retaining miniImageNet validation and test splits.The larger dataset includes the miniImageNet meta-validation and meta-testing data as a subset.
  • CACTUs: CACTUs obtains embeddings with DeepCluster using VGG-16, a 256-dimensional feature space, and 10,000 clusters.Training continues until normalized mutual information between consecutive data-cluster mappings converges.
  • Reported configurations and results: The appendix reports MAML hyperparameters for ImageNet-scale experiments and miniImageNet object-classification results averaged over 1000 tasks.The miniImageNet table reports 95% confidence intervals and includes embedding, cluster, and partition settings.
  • Results: The increased unlabeled meta-training data significantly improves all methods relative to their miniImageNet counterparts, except training from scratch.Training from scratch does not use the additional unlabeled data.
  • Results: CACTUs-MAML slightly outperforms the embedding linear classifier on 1-shot test tasks, while the linear classifier becomes better with more test-time supervision.Adding a small number of supervised tasks during meta-training slightly improves 1-shot performance.
Loading 1810.02334v6…