Source-linked AI summary
Semi-Supervised Learning with Deep Generative Models
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling
TL;DR
Semi-supervised learning needs methods that exploit abundant unlabelled data when labels are scarce, while existing generative approaches lack generality and scalability. The paper introduces deep generative models with scalable variational inference, obtaining highly competitive results and enabling class–style disentanglement. The approach remains costly when the number of classes is large.
Problem
Semi-supervised learning requires accurate classification from few labelled observations and many unlabelled ones, but existing generative approaches lack a generalised, scalable probabilistic solution.
Method
The paper combines deep neural-network density estimators with variational inference and models missing class labels as latent variables during inference.
Results
The stacked M1+M2 model outperforms previously best methods across tested conditions, while the models also separate class content from intra-class style.
Takeaways & Limitations
Deep generative models provide competitive semi-supervised classification and support probabilistic inference and image-generation queries.
Takeaways & Limitations
The models scale linearly with the number of classes because training requires re-evaluating the generative likelihood for each class.
Abstract
from arXiv · showhide
The ever-increasing size of modern data sets combined with the difficulty of obtaining label information has made semi-supervised learning one of the problems of significant practical importance in modern data analysis. We revisit the approach to semi-supervised learning with generative models and develop new models that allow for effective generalisation from small labelled data sets to large unlabelled ones. Generative approaches have thus far been either inflexible, inefficient or non-scalable. We show that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.
1 Introduction
Semi-supervised learning seeks accurate classification when labels are scarce but unlabelled data is abundant. The paper addresses the lack of a generalised, scalable probabilistic approach by combining generative models with deep neural networks and variational inference.
- Motivation: Semi-supervised learning uses unlabelled observations to improve classification beyond what labelled data alone can provide.The setting is practically important because labels may be expensive or impossible to obtain for the full dataset.
- Prior approaches: Existing methods include heuristic self-training, margin-based TSVMs, manifold methods, and non-parametric density models.The cited approaches differ in how they exploit unlabelled data, but the passage describes limitations including error reinforcement, inflexibility, and scalability concerns.
- Research gap: A generalised and scalable probabilistic approach for semi-supervised learning remains lacking.The paper identifies this as the gap motivating its contributions.
- Contributions: The proposed framework fuses probabilistic modelling with deep neural networks to form rich parametric density estimators.It is designed as a new generative framework for semi-supervised learning.
- Contributions: Stochastic variational inference enables joint optimisation of model and variational parameters at large-dataset scale.The paper presents this as the first application of variational inference to semi-supervised classification.
- Contributions: Experiments report state-of-the-art benchmark results and qualitative separation of class content from intra-class style.The models also support straightforward image analogies across datasets.
2 Deep Generative Models for Semi-supervised Learning
The paper develops deep generative models that use latent variables to represent data and class structure when labels are missing. Its M1, M2, and stacked M1+M2 models support classification through latent features or inference over latent classes.
- Latent-feature discriminative model (M1): M1 learns robust latent features with a deep generative model, then uses approximate posterior samples to train a separate classifier.The latent representation can reduce dimensionality and make observations more separable for classification.
- Latent-feature discriminative model (M1): The likelihood pθ(x|z) uses a nonlinear transformation of latent variables, parameterised by deep neural networks.The nonlinear transformation is intended to capture higher moments of the data density.
- Generative semi-supervised model (M2): M2 models each observation with a latent class variable y and continuous latent variable z, treating missing labels as latent during inference.Predictions for missing labels come from the inferred posterior pθ(y|x).
- Generative semi-supervised model (M2): M2 uses independent class and continuous latent variables to separate digit class specification from writing style.The model integrates over possible classes for unlabelled data, making classification an inference problem.
- Stacked generative semi-supervised model (M1+M2): M1+M2 stacks the models by learning a latent representation z1 first and then applying M2 to that representation instead of raw data.The resulting model has two layers of stochastic variables parameterised with deep neural networks.
3 Scalable Variational Inference
The paper uses amortized variational inference to make semi-supervised generative models tractable and scalable, jointly optimizing generative and inference parameters with stochastic gradients. Its models handle labelled and unlabelled observations, while their computational costs remain competitive with neural alternatives.
- Variational inference: Approximate posteriors replace intractable exact inference, using inference networks with global parameters to amortize posterior estimation across data points.The fixed-form qφ(z|x) approximates p(z|x), while recognition networks avoid separate variational parameters for each observation.
- Generative semi-supervised objective: For observed labels, the variational bound extends the latent-variable objective; for missing labels, y is treated as a latent variable and inferred jointly with z.The missing-label objective uses qφ(y,z|x), and qφ(y|x) serves as a discriminative classifier at test time.
- Generative semi-supervised objective: A classification loss is added so qφ(y|x) learns from labelled data, with α controlling the relative weight of generative and discriminative learning.The experiments use α = 0.1·N.
- Optimization: The unified bounds support joint optimization of θ and φ through deterministic reparameterization and Monte Carlo gradient estimates, followed by stochastic gradient-based updates.The implementation uses standard optimizers such as SGD, RMSprop, or AdaGrad; the experiments use AdaGrad.
- Computational complexity: The M1 update has complexity CM1 = MSCMLP, while M2 has complexity CM2 = LCM1 and evaluates its cost in a low-dimensional latent space.CMLP has form O(KD^2), where K is the number of MLP layers and D is their average layer dimension.
- Computational complexity: The approach is no more expensive than competitive auto-encoder or neural alternatives and remains fully probabilistic, enabling inferential queries unavailable to many alternatives.The computational comparison is based on the stated complexity of the models and competing approaches.
4 Experimental Results
The models are evaluated on semi-supervised MNIST, SVHN, and NORB classification, with generative latent features, conditional generation, and stacked models showing strong performance. The conditional model also supports style–content disentanglement and efficient analogical generation.
- Benchmark setup: MNIST experiments vary the labelled set from 100 to 3000 balanced labels, with repeated randomised sampling used to estimate performance.The benchmark splits 50,000 training points into labelled and unlabelled subsets.
- Benchmark classification: M1 outperforms models based on simple embeddings, while M2 achieves similar results without a separate TSVM classifier.M1 uses latent features for classification; M2 combines feature learning and classification within one conditional generative model.
- Benchmark classification: The stacked M1–M2 model obtains the best results across conditions and outperforms previously best methods.With all labels available, the model reaches 0.96% test-set performance on permutation-invariant MNIST.
- Conditional generation: Conditional generation separates handwriting style from class by varying z while fixing y, or varying y while fixing an inferred z.The model generates analogous images by inferring latent variables from test images and changing the class label.
- Conditional generation: On SVHN, the model preserves house-number style while changing the digit, extending the same disentanglement behavior to a more complex dataset.The paper describes these generations as the best current performance in simulation from generative models on MNIST and SVHN.
- Comparative evaluation: The approach improves on SFNN by replacing importance-sampling Monte Carlo EM with efficient joint inference that is easier to scale.SVHN and NORB experiments further evaluate latent-feature classification with nearest-neighbour and TSVM classifiers.
5 Discussion and Conclusion
The paper concludes that deep generative models with variational inference provide competitive semi-supervised learning, while identifying scalability and future architectural extensions as open issues.
- Discussion: Approximate inference can be extended to model parameters, supporting variational learning and principled model selection when labelled data is limited.The paper highlights efficient model selection as particularly important in small-data semi-supervised settings.
- Future work: Neural-network parameterisation makes convolutional and locally connected architectures a promising extension for image classification.The paper identifies convolutional neural networks as an area for future exploration rather than an evaluated component of the presented models.
- Limitations: The models scale linearly with the number of classes because training re-evaluates the generative likelihood for each class.The authors suggest posterior truncation or error-correcting output codes to reduce these evaluations.
- Conclusion: The paper reports new generative models and efficient variational optimisation that improve prediction by exploiting information in data density.It concludes that the models are among the most competitive currently available for semi-supervised learning.