Source-linked AI summary
Stacked What-Where Auto-encoders
Junbo Zhao, Michael Mathieu, Ross Goroshin, Yann LeCun
TL;DR
SWWAE addresses the lack of a unified, sampling-free architecture for supervised, semi-supervised, and unsupervised learning. It couples Convnet and Deconvnet pathways through “what”-“where” pooling variables and joint reconstruction losses, achieving good accuracy across varied tasks while improving supervised SVHN error over a vanilla Convnet.
Problem
Existing approaches do not combine supervised, semi-supervised, and unsupervised learning in one procedure, while DBMs rely on sampling with poor convergence and mixing.
Method
SWWAE couples a Convnet encoder with a Deconvnet decoder, using pooled “what” activations, “where” switches, and joint discriminative and reconstruction losses.
Results
SWWAE yields good accuracy across varied semi-supervised and supervised tasks; on supervised SVHN, testing error decreased from 5.89% to 4.94% versus a vanilla Convnet.
Takeaways & Limitations
The Convnet–Deconvnet architecture provides a unified framework for all three learning modalities and may be useful for video tasks with abundant unlabeled samples.
Abstract
from arXiv · showhide
We present a novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training. An instantiation of SWWAE uses a convolutional net (Convnet) (LeCun et al. (1998)) to encode the input, and employs a deconvolutional net (Deconvnet) (Zeiler et al. (2010)) to produce the reconstruction. The objective function includes reconstruction terms that induce the hidden states in the Deconvnet to be similar to those of the Convnet. Each pooling layer produces two sets of variables: the "what" which are fed to the next layer, and its complementary variable "where" that are fed to the corresponding layer in the generative decoder.
1 INTRODUCTION
SWWAE targets a single architecture and learning procedure for supervised, semi-supervised, and unsupervised learning while combining discriminative and generative pathways. It preserves information lost by many-to-one pooling through complementary “where” variables used for reconstruction.
- Existing layer-wise auto-encoder pretraining does not unify supervised and unsupervised learning, while DBMs rely on sampling and suffer poor convergence and mixing.
- SWWAE learns factorized representations encoding invariance and equivariance while leveraging labeled and unlabeled data in one framework.
- Pooling switches provide the complementary “where” information needed to reconstruct inputs after many-to-one Convnet mappings.
- A Convnet encoder and Deconvnet decoder form stacked what-where auto-encoders trained jointly without alternate optimization.
2 RELATED WORK
SWWAE extends prior what-where, invariant-feature, and joint-loss approaches by coupling Convnet and Deconvnet pathways for unified learning with labeled and unlabeled data. Its decoder uses only corresponding-stage “where” states rather than full pre-pooled activations.
- Prior what-where formulations addressed invariance and equivariance, whereas SWWAE learns similar representations without requiring the true latent state.
- Many related methods focus on unsupervised feature learning, and layer-wise greedy training limited the competitiveness of another hierarchical architecture.
- Compared with related joint-loss and semi-supervised methods, SWWAE emphasizes the ease of extending a Convnet with a Deconvnet to use unlabeled data.
- Unlike Ladder networks, SWWAE decoders receive corresponding-stage “where” states and incorporate unpooling rather than reconstructing only the top layer.
3 MODEL ARCHITECTURE
SWWAE combines discriminative and reconstruction losses in a Convnet–Deconvnet architecture, routing pooled “what” upward and “where” laterally for approximate inversion. Hard and soft pooling support different training choices, while joint losses enable supervised, unsupervised, and semi-supervised modes.
- MODEL ARCHITECTURE: SWWAE minimizes discriminative, input-level reconstruction, and intermediate reconstruction losses, weighted by λ parameters.
- MODEL ARCHITECTURE: Pooling separates max activations as “what” and argmax switch positions as “where”; the decoder unpools “what” using “where” before convolutional reconstruction.
- MODEL ARCHITECTURE: Input- and middle-level L2 terms measure discrepancies between the input or Convnet activations and their Deconvnet reconstructions.
- SOFT VERSION “WHAT” AND “WHERE”: Soft pooling makes the pooling operators differentiable through location variables and represents location information more accurately than hard argmax switches.
- TRAINING WITH JOINT LOSSES AND REGULARIZATION: Setting λL2* to 0 yields supervised Convnet training, setting λNLL to 0 yields an unsupervised convolutional auto-encoder, and retaining all losses gives semi-supervised learning.
- TRAINING WITH JOINT LOSSES AND REGULARIZATION: Jointly modeling P(Y | X) and P(X) during training is presented as more effective than using unsupervised pretraining only for initialization.
- TRAINING WITH JOINT LOSSES AND REGULARIZATION: Intermediate reconstruction terms prevent feature-plane shuffling, keep encoder and decoder stages aligned, and discourage upper layers from becoming idle.
4 EXPERIMENTS
Experiments examine SWWAE’s reconstruction behavior, factorized “what-where” representations, and classification across semi-supervised and supervised settings. Results show that preserving “where” improves reconstruction quality, while “what” and “where” exhibit complementary invariant and equivariant behavior.
- 4.1 NECESSITY OF “WHERE”: Unpooling produces clearer MNIST reconstructions than upsampling across pooling sizes 2, 4, 8, and 16, demonstrating the importance of preserving “where” information.The comparison uses a trained SWWAE with hard max-pooling in unsupervised mode.
- 4.2 INVARIANCE AND EQUIVARIANCE: The capsule-emulation experiments reconstruct original and horizontally translated MNIST digits using the factorized representation.Figure 4 compares original inputs, reconstructions, and reconstructions of positive and negative horizontal translations.
- 4.2 INVARIANCE AND EQUIVARIANCE: “What” remains locally stable under input translation, exhibiting strong invariance, whereas “where” responds to translation in a localized and approximately linear manner.The experiments translate MNIST digits horizontally and vertically and visualize feature responses with scatter plots.
- 4.3 CLASSIFICATION PERFORMANCE: SWWAE is evaluated in semi-supervised and supervised classification experiments on MNIST and SVHN, using reconstruction losses alongside the discriminative pathway.The classification experiments use hard pooling, which performs better than soft pooling for classification.
- 4.3 CLASSIFICATION PERFORMANCE: Validation-error curves assess the standalone effect of the combined reconstruction regularization weight λL2* on MNIST and SVHN across different label counts.In this experiment, no other well-known regularizer is applied; λL2M and λL2rec are combined.
- 4.3 CLASSIFICATION PERFORMANCE: 5.89% to 4.94%: SWWAE lowers SVHN testing error versus a vanilla Convnet under the same fully supervised configuration.The paper also evaluates SWWAE on SVHN with hyper-parameters selected using validation error.
5 CONCLUSION AND OUTLOOK
SWWAE pairs a Convnet with a Deconvnet and achieves good accuracy across supervised and semi-supervised tasks.
- SWWAE combines a Convnet feed-forward pathway with a Deconvnet feedback pathway for supervised and semi-supervised learning.
APPENDIX: MORE MNIST
Experiments evaluate SWWAE on MNIST under multiple regularization, semi-supervised, and fully supervised settings. The results include a lower fully supervised testing error than a Convnet baseline, while the authors identify a limitation on MNIST reconstruction.
- MNIST validation experiments compare SWWAE with dropout, convolutional dropout, and L1 sparsity regularization approaches.Dropout is added at fully connected layers by default for these comparisons.
- MNIST semi-supervised evaluation compares SWWAE against other published methods using testing error rate.
- 0.71% testing error versus 0.76% for Convnet under the same configuration in the fully labeled setting.
- MNIST reconstruction can be too easy for SWWAE because binary digits may be reconstructed from pooling switch positions without the “what” representation.
- SWWAE’s reconstruction-based regularization is reported as insufficient on MNIST but effective on natural-image datasets such as CIFAR and STL-10.