Source-linked AI summary
Semi-Supervised Learning with Ladder Networks
Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, Tapani Raiko
TL;DR
Semi-supervised learning needs a way to use unlabeled structure without relying only on pre-training or preserving task-irrelevant input detail. The paper combines supervised training with layer-wise denoising in Ladder networks, achieving state-of-the-art results across semi-supervised MNIST and CIFAR-10 and improving fully labeled permutation-invariant MNIST.
Problem
Unsupervised auxiliary tasks are often limited to pre-training and may preserve input details that are irrelevant to the supervised task.
Method
The paper combines supervised learning with layer-wise denoising targets in Ladder networks compatible with MLP and CNN architectures.
Results
The model reaches state-of-the-art performance in semi-supervised MNIST, permutation-invariant MNIST, and CIFAR-10, while improving fully labeled permutation-invariant MNIST.
Takeaways & Limitations
Strong gains with very small numbers of labels and improved full-label performance indicate that the unsupervised task can complement rather than disturb supervised learning.
Takeaways & Limitations
The decoder can only optimally denoise conditionally independent distributions.
Abstract
from arXiv · showhide
We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on the Ladder network proposed by Valpola (2015), which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification, in addition to permutation-invariant MNIST classification with all labels.
1 Introduction
The paper combines simultaneous supervised and unsupervised learning in Ladder networks, extending denoising across layers while preserving compatibility with standard neural architectures. It reports state-of-the-art results across semi-supervised tasks and improved fully labeled permutation-invariant MNIST performance.
- Approach: The method combines supervised learning with layer-wise unsupervised denoising in a Ladder network.The approach extends Valpola’s unsupervised Ladder network by adding supervision.
- Compatibility with supervised methods: Unsupervised learning focuses on details relevant to the supervised task and can augment MLPs or CNNs.The authors report improvement over a state-of-the-art supervised starting point after adding simultaneous unsupervised learning.
- Scalability resulting from local learning: Local unsupervised targets at every layer make the approach suitable for very deep neural networks.The paper demonstrates this scalability with two deep supervised architectures.
- Computational efficiency: Adding the decoder approximately triples training computation, while computation per update retains similar scaling with a small multiplicative factor.The paper notes that better information utilization may reduce the corresponding increase in training time.
- Results: The method reaches state-of-the-art results in semi-supervised MNIST, permutation-invariant MNIST, and CIFAR-10, plus a new record for fully labeled permutation-invariant MNIST.The reported gains are attributed to combining skip connections and layer-wise unsupervised targets with supervision.
2 Derivation and justification
The paper motivates Ladder networks through their connection to hierarchical latent-variable models and denoising. Layer-wise denoising costs and skip connections provide a tractable, nested autoencoder formulation that learns representations and denoising functions together.
- Latent-variable motivation: Hierarchical latent-variable models are attractive for semi-supervised learning because they combine supervised and unsupervised learning in a principled way.Their training can be viewed through inference over unobserved variables and model learning.
- Latent-variable motivation: Efficient inference is a central challenge because latent-variable equations generally lack closed-form solutions and often require iteration.The Ladder network addresses this modeling context through learned denoising rather than explicit iterative inference.
- Denoising and probabilistic modeling: Denoising functions are connected to probabilistic modeling because optimal reconstruction under noise uses posterior information.The paper states that posterior-centered reconstruction minimizes expected squared denoising cost.
- Ladder network construction: A Ladder network applies denoising to latent representations at every layer, extending denoising autoencoders beyond reconstructing only observations.The formulation builds on denoising autoencoders and denoising source separation.
- Ladder network construction: Each layer compares clean and corrupted representations, while skip connections let higher layers leave discarded details to lower decoder levels.The layer-wise cost trains both denoising functions above and cleaner encoder features below.
3 Implementation of the Model
The Ladder network augments a feedforward encoder with a decoder and trains supervised classification together with layer-wise denoising. The implementation supports fully connected and convolutional encoders, noisy forward passes, normalization, and simplified decoder variants.
- Model construction: The model is built by taking a feedforward network as an encoder, adding a decoder, and minimizing the sum of supervised and unsupervised costs.The procedure applies in fully labeled or semi-supervised settings using standard optimization such as stochastic gradient descent.
- Model construction: Each encoder layer is analyzed through its conditional representation distribution to define a denoising function using corrupted and higher-level reconstructed representations.The decoder reconstructs clean z(l) from corrupted ˜z(l) and ˆz(l+1).
- Encoder and corruption: The encoder uses batch normalization, while Gaussian noise is added to inputs and after each batch-normalization operation, producing clean and corrupted passes.Batch normalization also prevents denoising objectives from favoring constant encoder outputs.
- Training costs: The supervised cost is the average negative log probability that the noisy classifier output matches the target labels.Noise is also used to regularize supervised learning.
- Training interaction: Supervised learning influences unsupervised representations indirectly because decoder modulation encourages lower-layer representations to retain information about higher-level abstractions.This connects the supervised abstractions selected at the top with representations learned by the unsupervised decoder.
- Variations: The decoder can extend to convolutional networks by mirroring encoder convolutions, and setting lower-layer denoising weights to zero yields the decoder-light Γ-model.The Γ-model places denoising cost only on the top layer and can be plugged into feedforward networks without implementing a decoder.
4 Experiments
The experiments evaluate Ladder networks and simpler variants across semi-supervised MNIST and CIFAR-10 settings, using carefully optimized supervised baselines. Results show strong performance, more reliable convergence for full Ladder networks, and gains from denoising targets, while some baselines remain unstable or overfit.
- Experimental setup: The experiments compare Ladder networks, the simpler Γ-model, input-layer denoising, and supervised baselines on MNIST and CIFAR-10.The study evaluates both fully connected and convolutional encoders, with CIFAR-10 tested only using the Γ-model.
- Experimental setup: All samples contribute to the decoder, while only N = 100, 1000, or all labels provide the supervised cost.The evaluation uses held-out validation and test sets, and avoids tuning on held-out test samples.
- Experimental variability: With N = 100 labels, all models sometimes failed to converge, and roughly 5 % of bottom-level or full-cost Ladder runs exceeded 2 % test error.The authors used 40 random-initialization runs to estimate average error reliably in the presence of outliers.
- MNIST: Convolutional results show that a single bottom-level convolution improves over the fully connected network, while full Ladder denoising converges more reliably than the Γ-model.More convolutions significantly improve the Γ-model, but its high variance suggests confirmation bias.
- CIFAR-10: The supervised CIFAR-10 baseline was carefully tuned, yet the Γ-model still improved it through a simple denoising target addition.The improvement was smaller than in the MNIST experiments.
5 Related Work
The related work situates Ladder networks among semi-supervised methods based on denoising, label propagation, multi-view learning, generative models, and alternative representation-learning objectives. The paper emphasizes simultaneous supervised and unsupervised learning, layer-wise denoising, and lateral connections as distinguishing features.
- Semi-supervised objectives: Earlier methods often used unsupervised auxiliary tasks for pre-training before ordinary supervised learning, whereas this paper combines both objectives during training.The related work includes methods that apply unsupervised costs simultaneously, but contrasts them with the common pre-training pattern.
- Label and view-based methods: The Γ-model is related to label propagation and co-training because corrupted inputs can encourage locally consistent labels or act like multiple views.These connections are presented as interpretations of the model’s behavior, not as identity with those methods.
- Label-based methods: The paper compares Ladder networks with pseudo-labeling, noting that similar scheduling could address the Γ-model’s apparent confirmation bias.The authors suggest that the optimal denoising weight may change during learning.
- Representation learning: Other related approaches include transformation-based convolutional pre-training, contractive costs, and deep autoencoders with lateral or pooling-related connections.These methods differ in their training objectives, connectivity, or use of transformations and sparsity.
- Generative models: Deep Boltzmann machines and variational autoencoders provide generative semi-supervised alternatives that model inputs and labels through inference or latent-variable structure.The stacked variational-autoencoder variant performed best among the cited configurations and concentrated classification in higher layers.
- Alternative learning algorithms: Target propagation is related through expectation-like propagation, but Ladder networks additionally rely on lateral encoder-decoder connections.The paper notes that the role of these lateral connections remained an open question.
6 Discussion
The simultaneous unsupervised task improves supervised CNN and MLP networks across semi-supervised settings while also improving fully labeled permutation-invariant MNIST. The approach is practical but leaves encoder-decoder design and temporal extensions as future research directions.
- The method improves CNN and MLP networks to state-of-the-art performance across several semi-supervised learning tasks.
- It also achieves state-of-the-art performance and significantly improves the baseline with full labels on permutation-invariant MNIST.
- The proposed model is easy to implement with existing feedforward architectures and uses backpropagation from a simple cost function.
- The largest performance improvements occur when models have many parameters relative to the available labeled samples.
- Future work includes identifying better-suited encoder-decoder structures and extending Ladder networks to temporal data.
A Specification of the convolutional models
The convolutional models are based on ConvPool-CNN-C and are adapted for the MNIST and CIFAR-10 experiments. Their architecture replaces fully connected layers with global mean pooling and uses Gaussian noise with convolutional batch normalization.
- Conv-Small and Conv-Large are used for MNIST and CIFAR-10, respectively, and both are inspired by ConvPool-CNN-C.
- The models replace fully connected layers with a global mean pooling layer immediately before the softmax function.
- Compared with ConvPool-CNN-C, the models use Gaussian noise instead of dropout and convolutional per-channel batch normalization.
B Formulation of the Denoising Function
The denoising function reconstructs each clean hidden representation from its corrupted value and the reconstruction above, with a nonlinear conditional parametrization. Comparisons indicate that the proposed form and its modulation term are important for performance, although evaluations used limited hyperparameter tuning.
- The denoising function maps a corrupted hidden representation and the reconstruction from the layer above to a reconstructed clean representation.
- The selected parametrization is linear in the corrupted representation while its slope and bias depend nonlinearly on the upper-layer input.
- The comparison systematically removes components or replaces the denoising function, using separately tuned hyperparameters but fixed corruption-noise standard deviation 0.3.
- The experiments evaluated 1000 labeled MNIST samples and reran the best comparison at 100 labels, with less exhaustive tuning than the final results.
- The proposed denoising parametrization outperforms alternative parametrizations in the tested model structure.
- The proposed function includes the augmented term ˜zu, whose removal hurts performance more than removing the nonlinearity in the tested comparisons.
- A standard autoencoder decoder did not significantly outperform the supervised model across decoder attachment points and label counts tested.