From neural PCA to deep unsupervised learning

Harri Valpola

Source-linked AI summary

From neural PCA to deep unsupervised learning

Harri Valpola

arXiv:1411.7783v2Nov 28, 2014stat.MLcs.LGcs.NE

TL;DR

The paper addresses how to exploit abundant unlabeled data for useful representations while avoiding the training difficulties of deep hierarchical latent-variable models. It proposes a ladder autoencoder with lateral shortcuts and layerwise cost terms, and preliminary experiments show efficient learning of abstract invariant features and faster training.

Problem
Unsupervised learning should use abundant unlabeled data to discover features that complement supervised features, but deep hierarchical models are difficult and slow to train.
Method
The paper proposes a ladder autoencoder with lateral shortcut connections and cost-function terms at every layer, combining local reconstruction training with network-wide gradient propagation.
Results
The experiments show that higher layers learn abstract invariant features, while higher-level cost terms speed up learning in the ladder network.
Takeaways & Limitations
Lateral shortcuts let lower layers represent details while higher layers focus on selected abstract features, and layerwise training signals support efficient learning in deeper networks.
Takeaways & Limitations
The higher-layer cost terms bias representations toward PCA-type solutions, which may be undesirable in some cases.

Abstract

from arXiv · show

A network supporting deep unsupervised learning is presented. The network is an autoencoder with lateral shortcut connections from the encoder to decoder at each level of the hierarchy. The lateral shortcut connections allow the higher levels of the hierarchy to focus on abstract invariant features. While standard autoencoders are analogous to latent variable models with a single layer of stochastic variables, the proposed network is analogous to hierarchical latent variables models. Learning combines denoising autoencoder and denoising sources separation frameworks. Each layer of the network contributes to the cost function a term which measures the distance of the representations produced by the encoder and the decoder. Since training signals originate from all levels of the network, all layers can learn efficiently even in deep networks. The speedup offered by cost terms from higher levels of the hierarchy and the ability to learn invariant features are demonstrated in experiments.

1 Introduction

The paper addresses why unsupervised learning has struggled to complement supervised learning and introduces a ladder network designed to learn abstract invariant features efficiently.

Motivation: Unsupervised learning is motivated by abundant unlabeled data whose statistical structure can contain far more information than sample labels.The paper notes that labels are difficult to obtain, whereas unlabeled images are plentiful.
Motivation: The paper argues that unsupervised methods often preserve information that supervised learning must filter out as irrelevant to the task.This mismatch is presented as a reason unsupervised learning has not consistently improved supervised results.
Contribution: The proposed network combines unsupervised learning with supervised training by minimizing a cost function and allowing higher layers to discard details.The network can continue learning alongside supervised learning rather than being limited to pre-training.
Contribution: Lateral shortcut connections give the autoencoder representational capacity analogous to hierarchical latent variable models, letting higher layers focus on abstract invariant representations.The structure is called a ladder network because vertical encoder and decoder paths are connected by horizontal lateral links.
Learning: Each hierarchy level receives a training target through a combination of denoising source separation and denoising functions, addressing slow learning in deep autoencoders.Without these level-specific targets, shortcuts can learn first and divert training signals from longer paths.
Results: Experiments report that higher levels focus on invariant representations and that higher-layer training targets speed up learning, although the results are preliminary.The paper presents these findings as demonstrations of the proposed network's intended benefits.

2 Ladder network: an autoencoder which can discard information

The ladder network addresses the mismatch between information-preserving unsupervised learning and task-focused supervised learning by combining hierarchical representations with autoencoder-based learning. Its lateral shortcuts let higher layers focus on abstract invariant features while lower layers retain details.

Ladder network: The ladder network adds lateral shortcut connections between encoder and decoder paths, giving it the representational capacity associated with hierarchical latent-variable models.The shortcuts allow decoder representations to recover information missing from higher layers.
Learning efficiency: The network’s mappings learn efficiently when each mapping is close to a cost-function term, even when gradients would propagate inefficiently through the mappings.This motivates distributing learning signals across the hierarchy rather than relying only on the final decoder output.
Motivation: Unsupervised learning should discard task-irrelevant information rather than preserve every input detail, so it can cooperate with supervised feature selection.The paper frames this as the central requirement for combining unlabeled and labeled data.
Complementary learning: Unsupervised learning can continue alongside supervised learning by tuning representations toward features that correlate with supervised selections.Such correlated features may improve generalization when selected features are absent or occluded.
Hierarchical latent variables: Hierarchical latent-variable models let higher layers represent selected abstract features while lower layers encode details, but their inference and learning can be complex and iterative.Exact inference is typically intractable, and approximate methods may require iterative updates across layers.

3 Parallel learning on every layer

The ladder network distributes learning across layers by combining lateral connections with layer-specific denoising objectives. This structure supports abstract higher-level representations while improving learning efficiency, though the decorrelation design can bias solutions toward PCA-type projections.

3 Parallel learning on every layer: Standard autoencoder training becomes inefficient in deep ladder networks because shortcuts leave higher layers with a shrinking share of reconstruction error.Each shortcut can contribute to reconstruction, reducing the error signal available to higher layers.
3 Parallel learning on every layer: The learning rule combines denoising source separation with denoising functions that remove injected noise, placing training signals at every hierarchical level.Layer-specific costs compare latent variables with their denoised versions, bringing supervision closer to each layer’s parameters.
3 Parallel learning on every layer: Lateral connections let intermediate representations retain information independently of higher layers while higher levels focus on abstract invariant features.Detailed information can remain in lower layers, allowing higher layers to avoid representing every input detail.
3.1 From neural PCA to denoising source separation: The underlying denoising source-separation approach evolved from neural PCA and uses whitening, decorrelation, and denoising to learn latent representations.Nonlinear PCA with whitened inputs and orthogonalized projections can be interpreted as an efficient approximation to EM for an ICA-tuned latent-variable model.
3.4 Decorrelation term for the cost function: The covariance decorrelation cost penalizes deviations from the identity matrix, but it treats overly small and overly large eigenvalues symmetrically even though only small values threaten collapse.The determinant of the covariance matrix is presented as an alternative information-content measure based on the product of eigenvalues.
3 Parallel learning on every layer: The experiments show that higher ladder levels can discard information to focus on invariant representations and that higher-layer training targets speed learning, but the results remain preliminary.The authors also note that mappings close to individual cost terms can learn efficiently even when gradient propagation through the mappings is inefficient.

4 Experiments

The experiments test denoising functions, lateral connections, and higher-layer cost terms as mechanisms for improving ladder-network learning and representations.

The experiments examine how denoising functions represent probability distributions and how lateral connections support abstract invariant features.
Higher-layer cost-function terms are tested for their ability to speed up learning.
Ladder-network cost computations compare clean activations with reconstructions from a corrupted path at each layer.
The experiments progressively develop a two-layered ladder network, moving from distribution modeling to linear ICA and then higher-order variance modeling.

4.1 Representing distributions with denoising functions

The first experiment relates activation distributions to denoising functions by comparing super-Gaussian, sub-Gaussian, and Gaussian inputs under unit-variance corruption.

Three zero-mean, unit-variance distributions—super-Gaussian, sub-Gaussian, and Gaussian—are evaluated with denoising functions trained under unit-variance corruption.
The model uses a single hidden tanh neuron with a bypass connection when no hidden forward layers are present.
Large corruption noise smooths the denoising function because the noise variance is comparable to the input variance.
For Gaussian observations, the denoising function is linear and the theoretical optimum is ˆx = ˜x/2.
The estimated Gaussian denoising function closely follows the theoretical optimum.

4.2 ICA model

The ICA experiment studies whether lateral connections let a ladder network recover non-Gaussian sources instead of favoring principal-component directions.

The ICA data comprise 10,000 samples from 15 linearly mixed sources: five super-Gaussian, five sub-Gaussian, and five Gaussian.
The model incorporates unit-wise hidden denoising for source independence and a lateral mapping capable of representing observation covariance.
The model recovers the original sources up to permutation and scaling, as measured through the normalized loading matrix.
Without the lateral mapping B, the network primarily extracts the principal subspace, so PCA overrides ICA when independent components do not align with it.
With B and 11 hidden units, the ten non-Gaussian sources achieve loadings of 0.958–0.994, averaging 0.981.
Removing B slows convergence by about 100 times, although later training improves average loading to 0.870.
Pre-whitening restores the autoencoder’s ability to recover independent components, but may be harder to apply in more complex cases.

4.3 Hierarchical variance model

The hierarchical variance experiment adds a second layer to model dependencies among first-layer variances, while lateral shortcuts preserve lower-level information for reconstruction.

4.3 Hierarchical variance model: A second layer is added to capture nonlinear dependencies, including correlations among variances of first-layer activations.
4.3 Hierarchical variance model: Variance information modulates denoising connection strength, allowing higher-layer representations to influence first-layer reconstruction.
4.3 Hierarchical variance model: The dataset contains four groups of four Gaussian sources whose changing variances are controlled by four higher-order variance sources.
4.3 Hierarchical variance model: The data are pre-whitened, while the changing source variances make their marginal distributions super-Gaussian.
4.3.2 Model structure: The network separates source subspaces and models correlations between source variances; with 1000 iterations, it had practically converged.
4.3.3 Results: The learned denoising modulation follows the norm of the activation vector within a source subspace, consistent with quadratic variance features.
4.3.3 Results: Lateral shortcuts are necessary because variance features alone do not specify reconstruction direction, while higher layers lack space to represent all lower-layer activations.
4.3.3 Results: Adding higher-layer cost terms lowers C(0) faster than optimizing C(0) alone; approximately one third of the improvement is attributed to C(1).

5 Discussion

The ladder network learned abstract invariant features efficiently in preliminary, shallow experiments, while several extensions and validations remain open. The discussion highlights simple function-approximation costs, connections to related models, and unresolved depth and supervised-learning questions.

Empirical findings: The experiments learned abstract invariant representations by the second layer despite networks having no more than six layers between ˜x and ˆx.The two-layer case roughly corresponds to the simple- and complex-cell architecture described by Hubel and Wiesel.
Limitations: Much larger experiments are needed to establish whether the ladder network supports substantially deeper hierarchies.The existing experiments used networks with no more than six layers between ˜x and ˆx.
Open validation: Ladder networks may support useful supervised or semi-supervised learning by discarding higher-level detail while retaining task-relevant features, but this remains unverified.All reported experiments were unsupervised.
Modeling perspective: The approach replaces probabilistic modeling with function approximation and a simple cost function, while an MLP learned higher-level sources capturing dependencies among lower-level sources.The reported source-separation behavior arose from the input structure rather than a model tailored specifically for independent subspace analysis.
Future directions: Future work includes extending denoising mappings, corruption procedures, sampling, density calculation, and multiple rounds of corruption and denoising.Sampling from the model is identified as particularly useful, but the corruption procedure would need extension.

6 Conclusions

The conclusions propose ladder networks as autoencoders whose lateral shortcuts enable abstract invariant representations and whose layerwise costs improve deep unsupervised learning. Preliminary experiments support these properties, but deeper scaling and supervised compatibility still require larger validation.

6 Conclusions: Lateral shortcut connections give each layer the representational capacity of stochastic latent variables in hierarchical latent variable models.This lets higher layers discard information and focus on abstract invariant features.
6 Conclusions: Layerwise cost terms provide every mapping with direct training signals from local reconstruction errors while also propagating gradients throughout the network.The design targets efficient unsupervised learning in deep ladder networks.
6 Conclusions: Higher-level cost functions require decorrelation terms to prevent representation collapse, and decorrelated inputs can avoid bias toward a PCA solution.These requirements are presented as the price of using higher-level objectives.
6 Conclusions: Preliminary experiments verified abstract invariant features and faster learning from extra cost terms, while claims about very deep models and supervised learning need much larger experiments.The reported evidence supports these claims but does not yet verify them conclusively.