PaperlayerView exact revision on arXiv

Source-linked AI summary

Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods

Guo-Jun Qi, Jiebo Luo

arXiv:1903.11260v2cs.CV

TL;DR

This survey addresses training representation-learning models when labeled data are scarce but unlabeled data are abundant. It organizes unsupervised and semi-supervised methods, reviews their models and representation-learning principles, and identifies connections and future directions. The reviewed evidence includes comparable cross-dataset performance to supervised models and near-supervised ImageNet accuracy for contrastive learning.

  • Problem

    Training deep models often requires many labeled examples, but collecting sufficiently large labeled datasets is challenging when abundant unlabeled data are available.

  • Method

    The paper surveys unsupervised and semi-supervised methods, categorizing generative models and comparing principles including transformation equivariance, invariance, disentanglement, and self-supervision.

  • Results

    The reviewed unsupervised methods show comparable cross-dataset performance to fully supervised models, while contrastive learning achieves almost the same top-1 ImageNet accuracy as its fully supervised ResNet-50 counterpart.

  • Takeaways & Limitations

    The survey highlights unsupervised pretraining as a promising alternative to supervised pretraining and identifies unifying instance and transformation equivariances as a future direction.

  • Takeaways & Limitations

    Group convolutions guarantee transformation equivariance mathematically but restrict feature-map forms to the considered transformation group.

Abstract

from arXiv · show

Representation learning with small labeled data have emerged in many problems, since the success of deep neural networks often relies on the availability of a huge amount of labeled data that is expensive to collect. To address it, many efforts have been made on training sophisticated models with few labeled data in an unsupervised and semi-supervised fashion. In this paper, we will review the recent progresses on these two major categories of methods. A wide spectrum of models will be categorized in a big picture, where we will show how they interplay with each other to motivate explorations of new ideas. We will review the principles of learning the transformation equivariant, disentangled, self-supervised and semi-supervised representations, all of which underpin the foundation of recent progresses. Many implementations of unsupervised and semi-supervised generative models have been developed on the basis of these criteria, greatly expanding the territory of existing autoencoders, generative adversarial nets (GANs) and other deep networks by exploring the distribution of unlabeled data for more powerful representations. We will discuss emerging topics by revealing the intrinsic connections between unsupervised and semi-supervised learning, and propose in future directions to bridge the algorithmic and theoretical gap between transformation equivariance for unsupervised learning and supervised invariance for supervised learning, and unify unsupervised pretraining and supervised finetuning. We will also provide a broader outlook of future directions to unify transformation and instance equivariances for representation learning, connect unsupervised and semi-supervised augmentations, and explore the role of the self-supervised regularization for many learning problems.

1 INTRODUCTION

The survey examines unsupervised and semi-supervised methods for learning with few or no labels by exploiting abundant unlabeled data. It organizes their methods, principles, connections, and future directions into a unified landscape.

  • Motivation and Scope: The survey targets small-data settings where labeled examples are scarce but large volumes of unlabeled data are available.It focuses on exploiting unlabeled examples rather than auxiliary-task information from zero-shot or few-shot learning.
  • Motivation and Scope: Unlabeled-data distributions provide clues for learning robust, generalizable representations and can support both unsupervised and semi-supervised training.They can also help close domain gaps between tasks, motivating unsupervised and semi-supervised domain adaptation.
  • Method Landscape: The survey maps a spectrum from unlabeled-only methods to approaches combining unlabeled and labeled examples, with zero-shot and few-shot learning positioned alongside them through auxiliary tasks.Few-shot learning uses target-task labels together with information from auxiliary concepts, whereas zero-shot learning lacks labeled examples for target concepts.
  • Unsupervised Methods: Unsupervised representation learning is reviewed through transformation equivariance, generative models, disentanglement, and self-supervised methods.The generative-model coverage includes auto-encoders, GANs, flow-based models, and transformers, alongside autoregressive models for self-supervision.
  • Semi-Supervised Methods: Semi-supervised coverage includes generative extensions and teach-student models that encourage consistency between teacher and student predictions on labeled and unlabeled data.The survey categorizes teach-student approaches according to how teacher models are obtained.
  • Connections and Future Directions: The survey identifies shared principles across unsupervised and semi-supervised learning, including unlabeled-data exploration, transformations and noise, and adaptations of unsupervised models conditioned on labels.It proposes future work connecting transformation equivariance with supervised invariance, unsupervised pretraining with supervised finetuning, and transformation with instance equivariances.

2 UNSUPERVISED METHODS

Unsupervised representation learning seeks features that generalize to future tasks without labeled data, using transformation equivariance, generative models, and self-supervised signals. The survey organizes these approaches and examines how equivariant representations encode visual structure while balancing mathematical guarantees against representational flexibility.

  • Unsupervised representations are trained from unlabeled examples so they can generalize to new tasks, typically evaluated through downstream classification performance.
  • The survey groups unsupervised methods into transformation-equivariant representations, generative models, and self-supervised methods.It also evaluates these unsupervised methods in the appendix.
  • Transformation-Equivariant Representations: Transformation-equivariant representations encode intrinsic visual structures that transform predictably, supporting recognition of unseen concepts without requiring their labels.The discussion contrasts this with transformation invariance, which is associated with representations specialized to the labels of given tasks.
  • Transformation-Equivariant Representations: Group-equivariant convolutions produce feature maps that equivary to designated transformation groups, including compositions of rotations, translations, and reflections.Their defining property is that convolving a transformed input equals transforming the convolution of the original input.
  • Transformation-Equivariant Representations: Group convolutions guarantee mathematical equivariance but restrict feature-map form, motivating more flexible unsupervised approaches that exploit unlabeled-data distributions.Extensions include group-equivariant capsule networks, Spherical CNNs, and Steerable CNNs.

BiGAN and ALI: Adversarial Representation Learning

BiGAN and ALI augment GANs with an encoder for representation learning and use joint sample-noise discrimination to align data and latent representations. The survey also connects adversarial representation learning with disentanglement and reports theoretical results for encoder-generator inversion.

  • BiGAN and ALI: BiGAN and ALI learn a generator, an encoder, and a discriminator that distinguishes real sample-encoding pairs from generated sample-noise pairs.The encoder maps samples to latent noise, while the generator maps noise to samples.
  • BiGAN and ALI: Unlike classic GANs, these models add an encoder and give the discriminator a joint sample-noise pair rather than a single sample.
  • BiGAN and ALI: In the ideal case, the learned encoder inverts the generator almost everywhere, and joint training minimizes an autoencoder ℓ0 loss.
  • Hybrid generative models: Hybrid autoencoder-GAN methods extend adversarial representation learning, including approaches that match aggregated posterior and prior distributions to regularize generative modeling.Adversarial autoencoders train a discriminator to make q(z) indistinguishable from p(z).
  • Disentangled representations: Disentanglement methods organize latent variables into salient factors: InfoGAN maximizes mutual information between latent codes and generated samples, while β-VAE controls a latent information bottleneck.Increasing β strengthens the bottleneck and trades reconstruction fidelity against disentanglement.
  • Unsupervised representation learning: Contrastive learning achieved almost the same ImageNet top-1 accuracy as a fully supervised ResNet-50, illustrating the reported potential of unlabeled representation learning.

3 SEMI-SUPERVISED METHODS

The survey organizes semi-supervised learning into generative models and teacher-student models, showing how unlabeled and labeled data jointly support representation and classifier training. It also connects semi-supervised GANs to manifold-based label smoothness and reports adversarial teaching as a high-performing approach.

  • Semi-supervised generative models: Semi-supervised generative models extend unsupervised autoencoders, GANs, and disentangled representations by incorporating labeled examples.The survey emphasizes their intrinsic connection to corresponding unsupervised models.
  • Teacher-Student models: Teacher-student models use teacher predictions on unlabeled examples to supervise a student, including noisy, ensemble, and adversarial teacher variants.The survey describes teacher ensembles as responses to the limits of purely random-noise teachers.
  • Semi-supervised generative models: M1 trains a classifier on VAE representations learned from both labeled and unlabeled data, whereas M2 adds a latent class variable for generation and inference.M2 combines labeled and unlabeled marginal bounds and adds a classification cost using qφ(y|x).
  • Semi-supervised GANs: Semi-supervised GANs either classify samples into K real classes plus a fake class or enforce label smoothness along a learned data manifold.Feature matching supports the first paradigm, while localized generators connect the second to the Laplace-Beltrami operator.
  • Semi-supervised GANs: Localized GANs enable direct computation of manifold label smoothness and can use Jacobian orthogonality to prevent local manifold collapse.The approach avoids relying solely on graph-Laplacian approximations for the manifold operator.
  • Teacher-Student models: Virtual Adversarial Training constructs perturbations that maximally change student predictions and trains the student to remain consistent with the adversarial teacher.The survey reports VAT as achieving state-of-the-art semi-supervised performance.

4 DOMAIN ADAPTATION

The survey reviews domain adaptation as transferring representations or images between source and target distributions, using adversarial objectives and related regularization choices. It contrasts untied and tied representations and notes both semi-supervised extensions and optimization limitations.

  • Unsupervised domain adaptation: Unsupervised domain adaptation uses labeled source examples and unlabeled target examples to learn a classifier transferable across domains.Adversarial methods exploit generators or discriminators to align source and target distributions.
  • Design choices: Domain adaptation design choices concern weight sharing, the adapted base model, and the adversarial objective.Different combinations produce distinct adaptation models.
  • Adversarial Discriminative Domain Adaptation: ADDA uses separate source and target representation models, trains the classifier on labeled source data, and adversarially aligns target representations with source representations.The target encoder is optimized to confuse a domain discriminator after the source model is fixed.
  • Gradient Reversal Layer: GRL ties source and target representation weights and jointly trains the shared representation, classifier, and domain discriminator to make domains indistinguishable.Unlike ADDA, GRL optimizes the classifier and representation together.
  • Gradient Reversal Layer: GRL’s true minimax optimization is vulnerable to vanishing gradients, unlike the separate optimization structure used by ADDA.This is identified as a limitation of the GRL formulation.
  • Semi-supervised domain adaptation: PixelDA adapts source images toward the target distribution with a GAN, then trains a classifier using labeled adapted images and labeled target images.The method can also include content-similarity regularization.
  • Semi-supervised domain adaptation: Two-stream approaches model both similarity and difference between source and target domains instead of enforcing complete domain invariance.They separately train domain streams and regularize their weights toward related representations.

5 EMERGING TOPICS AND FUTURE DIRECTIONS

The survey identifies future directions that connect equivariant and invariant representations, unsupervised pretraining and supervised fine-tuning, and data or network augmentations. It reports promising transfer results for unsupervised pretraining while calling for stronger integration and evaluation.

  • 5.1 Transformation equivariance and invariance: A central open problem is integrating transformation equivariance for unsupervised learning with transformation invariance for supervised learning.The survey frames this as balancing generalization to new tasks with discriminative power for given tasks.
  • 5.2 Unsupervised pretraining for future tasks: Supervised pretraining can create gaps between labeled pretraining datasets and downstream problems, whereas unsupervised training can exploit larger unlabeled datasets.The survey presents unsupervised training as avoiding reliance on pre-labeled datasets.
  • 5.2 Unsupervised pretraining for future tasks: Unsupervised ImageNet pretraining achieves performance comparable to fully supervised Places pretraining and better object-detection performance than supervised pretraining.These cross-dataset results are presented as evidence that unsupervised pretraining is a promising alternative.
  • 5.2 Unsupervised pretraining for future tasks: Future evaluations should use more challenging transfer-learning protocols to assess increasingly powerful unsupervised representations.The survey explicitly calls for stronger evaluation scenarios.
  • 5.2 Unsupervised pretraining for future tasks: Unsupervised pretraining can be followed by lightweight classifier training with few labeled examples, decoupling representation learning from supervised classification.The survey contrasts this with jointly training a deep network on labeled and unlabeled data.
  • 5.3.1 Unifying instance and transformation equivariances: Future work could jointly maximize information about instances and transformations to unify instance-equivariant and transformation-equivariant representations.The survey links contrastive instance-context objectives with transformation-mutual-information objectives.
  • 5.3.2 Semi-supervised and unsupervised augmentations: The survey proposes combining supervised and unsupervised augmentations, including data and network perturbations, to improve robustness and generalizability.Network augmentation includes random or adversarial changes to weights and architectures.
  • 5.3.3 Self-supervision as a regularizer: Self-supervised learning is presented as a task-agnostic regularizer whose application and unified theory remain open across learning problems.The survey identifies broader applicability beyond the tasks already reviewed.

6 CONCLUSIONS

The paper surveys unsupervised and semi-supervised small-data methods, comparing their principles, generative models, domain-adaptation approaches, and connections.

  • The survey covers unsupervised and semi-supervised methods for learning with limited labeled data.
  • It reviews auto-encoders, GANs, flow-based models, and autoregressive models across supervised and semi-supervised categories.
  • The paper compares transformation equivariance, invariance, and disentanglement as principles for learning factorized and interpretable representations.
  • It reviews domain adaptation methods for bridging distribution gaps with unlabeled and labeled data.
  • The survey discusses connections between unsupervised and semi-supervised learning as future directions.

APPENDIX A EVALUATIONS ON UNSUPERVISED LEARNING

Unsupervised representations are evaluated through downstream classification tasks, with standard protocols intended to support fair comparisons across methods.

  • Unsupervised representations are assessed on downstream classification benchmarks including CIFAR-10, ImageNet, Places, and Pascal VOC.
  • Standard evaluation protocols have emerged to enable fair and direct comparisons among many recent unsupervised methods.
  • Some approaches are not compared under the common protocol because of legacy constraints.

A.1 Evaluation Protocol

The evaluation protocol separates unsupervised representation learning from supervised downstream assessment of generalization to new classification tasks.

  • Unsupervised representations are first trained using only unlabeled examples.
  • A supervised classifier is then trained on the learned representations to evaluate generalizability to a new classification task.
  • The ImageNet example uses AlexNet as the backbone, with five convolutional and three fully connected layers.

A.2 Results

The appendix reports unsupervised-learning evaluations using linear and nonlinear classifiers on ImageNet and Places, plus fine-tuned classification, detection, and segmentation on PASCAL VOC.

  • ImageNet: ImageNet evaluations compare unsupervised features using nonlinear classifiers trained on Conv4 and Conv5, with supervised and random models as bounds.
  • ImageNet: ImageNet linear evaluations train a 1,000-way classifier on resized feature maps from various convolutional layers.
  • Places: Places evaluation uses a frozen ImageNet-pretrained representation and a 205-way logistic regression classifier, alongside supervised and random baselines.
  • PASCAL VOC: PASCAL VOC evaluation covers classification and detection on VOC 2007 and segmentation on VOC 2012 after unsupervised ImageNet pretraining.
  • PASCAL VOC: PASCAL VOC reports mean Average Precision for classification and detection, and mean Intersection over Union for segmentation.

APPENDIX B EVALUATIONS ON SEMI-SUPERVISED LEARNING

This appendix section summarizes evaluation results for semi-supervised methods and points readers to additional comparison protocols.

  • The section summarizes selected results from semi-supervised methods.More evaluation protocols for comparing semi-supervised methods are available in reference.

B.1 Datasets

The evaluation uses CIFAR-10 and SVHN, training models with small labeled subsets while leaving the remaining training examples unlabeled.

  • CIFAR-10 Dataset: CIFAR-10 contains 50,000 training images and 10,000 test images across ten categories.
  • SVHN Dataset: SVHN contains 73,257 training and 26,032 test house-number images.
  • SVHN Dataset: The SVHN experiments use 50 or 100 labeled examples per digit, leaving the remaining examples unlabeled.

B.2 Results

Semi-supervised evaluations commonly use unlabeled training images alongside varying amounts of labeled data, standardized architectures and augmentations, and test-set error rates.

  • Evaluation Protocol: CIFAR-10 and SVHN evaluate semi-supervised models with all unlabeled training images and varying labeled examples, reporting error rates on separate test sets.
  • Evaluation Protocol: A 13-layer convolutional neural network is often adopted for fair comparison, with random translations and horizontal flips commonly used as augmentations.
  • Results: Teach-student models outperform other methods on both CIFAR-10 and SVHN, with VAT achieving the strongest performance among the compared methods.

APPENDIX C CHART OF UNSUPERVISED AND SEMI-SUPERVISED LEARNING

The appendix provides a lookup chart categorizing the survey’s unsupervised and semi-supervised methods, alongside tables reporting semi-supervised error rates on SVHN and CIFAR-10.

  • Method Categorization: The chart is provided for readers’ convenience when looking up relevant methods reviewed in the survey.
  • Evaluation Tables: Tables A.6 and A.7 report error rates on SVHN and CIFAR-10 using varying numbers of labeled examples.
  • Survey Scope: The survey’s scope includes representation learning with small labeled data and transformation equivariant representations.
  • Reviewed Methods: The chart lists Group Equivariant Convolutions, Steerable CNN, Spherical CNN, Group Equivariant Capsule Networks, AET, and AVT among the reviewed methods.
  • Method Categorization: Figure C.3 categorizes unsupervised and semi-supervised methods according to where they are reviewed in the survey.
Loading 1903.11260v2…