Source-linked AI summary
Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification
Yuting Zhang, Kibok Lee, Honglak Lee
TL;DR
Large-scale supervised learning had reduced the role of unsupervised learning, while prior joint-learning approaches had limited large-scale validation. This paper augments classification networks with reconstruction pathways and jointly trains supervised and unsupervised objectives. The authors report improved large-scale classification performance and show that intermediate representations preserve almost all input information except some local spatial detail.
Problem
With large labeled datasets, supervised networks had become dominant, while prior joint supervised–unsupervised methods had not been shown to scale to large amounts of labeled and unlabeled data.
Source: §1 · p. 1Method
The paper augments classification networks with decoding pathways for image reconstruction and trains the augmented architecture using supervised and unsupervised objectives.
Source: §1 · p. 2Results
Takeaways & Limitations
Intermediate activations of large-scale classification networks can support high-quality image reconstruction, and reconstruction objectives can accompany supervised training in large-scale classification.
Source: §1 · p. 2Prior SWWAE evaluations included CIFAR, but its effectiveness for larger-scale supervised tasks had not been demonstrated.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Unsupervised learning and supervised learning are key research topics in deep learning. However, as high-capacity supervised neural networks trained with a large amount of labels have achieved remarkable success in many computer vision tasks, the availability of large-scale labeled images reduced the significance of unsupervised learning. Inspired by the recent trend toward revisiting the importance of unsupervised learning, we investigate joint supervised and unsupervised learning in a large-scale setting by augmenting existing neural networks with decoding pathways for reconstruction. First, we demonstrate that the intermediate activations of pretrained large-scale classification networks preserve almost all the information of input images except a portion of local spatial details. Then, by end-to-end training of the entire augmented architecture with the reconstructive objective, we show improvement of the network performance for supervised tasks. We evaluate several variants of autoencoders, including the recently proposed "what-where" autoencoder that uses the encoder pooling switches, to study the importance of the architecture design. Taking the 16-layer VGGNet trained under the ImageNet ILSVRC 2012 protocol as a strong baseline for image classification, our methods improve the validation-set accuracy by a noticeable margin.
1. Introduction
The introduction revisits unsupervised learning in large-scale classification, where supervised training had become dominant and prior joint-learning approaches lacked large-scale validation. It proposes reconstruction pathways as a way to improve classification while examining what pretrained network representations retain.
1. Introduction
- Earlier joint supervised–unsupervised approaches had not been shown to scale to large labeled and unlabeled datasets, and SWWAE had not been demonstrated on larger-scale supervised tasks.Prior validations were mostly on small datasets such as MNIST; SWWAE results cited here include CIFAR.Source: §1 · pp. 1–2
- The approach augments an existing classification network with a mirrored decoding pathway that reconstructs images, and trains the augmented architecture with supervised and unsupervised objectives.The pathway can be trained separately or together with the encoder and classification pathway using standard stochastic gradient descent.Source: §1 · p. 2
- Reconstructions from intermediate activations indicate that high-capacity classification networks retain nearly all input information, except local spatial details affected by max-pooling.SWWAE produced better reconstructions than an autoencoder using fixed-switch unpooling.Source: §1 · p. 2
- Joint supervised and reconstructive fine-tuning improves large-scale image-classification performance, including for a 16-layer VGGNet.The paper reports a noticeable margin over the VGGNet baseline.Source: §1 · p. 2
- The study compares SWWAE with simpler autoencoder variants to examine the roles of pooling switches and layer-wise reconstruction loss.The models use encoder pooling switches to recover locations during unpooling.Source: §1 · p. 2
2. Related work
Related work connects reconstruction-based representation learning with supervised feature learning and auxiliary-task methods. The paper situates its approach among these lines while distinguishing the scale and generality of prior evaluations.
2. Related work
- Mairal et al. (2009) unified sparse-coding reconstruction and classification losses, while Yang et al. (2010) added max-pooling for translation-invariant local features.Source: §2 · p. 2
- Zeiler et al. (2010) used layer-wise reconstruction in deconvolutional networks, and Zeiler et al. (2011) introduced unpooling with switches for end-to-end training.Source: §2 · p. 2
- Autoencoders learn representations by minimizing reconstruction error, and convolutional stacked autoencoders use pooling-induced sparsity bottlenecks; sparsity was a key regularizer in these studies.The cited background includes Bengio (2009), Makhzani & Frey (2015), and Masci et al. (2011).Source: §2 · p. 2
- Zhao et al. (2015) combined a convolutional autoencoder with classification for semi-supervised learning and reported promising results on MNIST, SVHN, and STL10.Source: §2 · p. 3
- Auxiliary tasks include intermediate-layer classification, layer-wise classification and reconstruction, and objectives based on spatial context or video continuity; autoencoder methods are described as more general-purpose.The cited examples include Lee et al. (2015), Ranzato & Szummer (2008), Doersch et al. (2015), and Wang & Gupta (2015).Source: §2 · p. 3
3. Methods
The method augments a classification network with mirrored decoding pathways and auxiliary reconstruction objectives, training the network with both supervised and unsupervised losses. It defines several decoder variants, including one that uses encoder pooling switches.
3. Methods
- The method attaches auxiliary unsupervised objectives to a classification network’s convolution-pooling layers and adds mirrored decoder pathways for reconstruction.The full augmented network is trained with both classification and reconstruction objectives.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 4
3.1. Unsupervised loss for intermediate representations
- Macro-layers group convolutional layers before max-pooling, and auxiliary objectives address vanishing lower-layer gradients and label-only representation learning.The objective adds an unsupervised loss associated with auxiliary pathways attached to convolution-pooling macro-layers.Source: §3.1 · pp. 3–4
3.2. Network augmentation with autoencoders
- The mirrored decoder uses deconvolutional layers, while SWWAE unpools activations at recorded max-pooling locations to recover local spatial variation.The top-level inner-product layers may be excluded from the autoencoder because they are intended to be more task-relevant.Source: §3.2 · pp. 4–5
- SAE-first measures reconstruction error at the input, whereas SAE-all matches activations at every macro-layer to send auxiliary signals directly to preceding layers.SAE-all replaces the input-only unsupervised loss with one matching activations across macro-layers.Source: §3.2 · p. 4
- SAE-layerwise reconstructs each macro-layer’s input from its clean output activations, allowing parallel layer-wise training and serving as the decoder architecture for efficient pretraining.It uses the same auxiliary loss as SAE-all but replaces its decoding pathway with a layer-wise one.Source: §3.2 · p. 4
- Unlike ladder networks, these models have no lateral connections, so they do not require encoder noise; whether denoising helps at large scale remains open.SWWAE pooling-switch connections do not pass encoder inputs directly to the decoder and therefore are not ladder-style lateral connections.Source: §3.2 · p. 5
4. Experiments
On ImageNet ILSVRC 2012, the authors evaluate reconstruction and classification using networks augmented with autoencoder decoding pathways, primarily based on 16-layer VGGNet. Reconstruction retains substantial input detail, and the augmented networks improve VGGNet classification performance.
4.1. Training procedure
- Training proceeds from a pretrained classification encoder and randomly initialized decoders through layer-wise decoder training, decoder finetuning, and joint encoder-decoder finetuning.The classification pathway is fixed during the initial decoder-training steps; all pathways are jointly finetuned with a reduced learning rate in the final step.Source: §4.1 · p. 5
4.2. Image reconstruction via decoding pathways
- AlexNet reconstructions were almost visually perfect even from the top 1000-way classification layer, with max-pooling spatial invariance accounting for much of the lost input detail.Higher-layer reconstructions mainly differed in local contrast and color saturation.Source: §4.2 · pp. 6–7
- On VGGNet, SWWAE-first recovered pool5 shape details better than fixed-switch SAE-first, and could reconstruct text screenshots, depth maps, and cartoons.The latter examples are reported in Appendix A2.5; the authors suggest high-level representations can adapt to other domains.Source: §4.2 · p. 7
4.3. Image classification with augmented architectures
- All autoencoder-augmented networks outperformed baseline VGGNet, and the reported baseline implementation matched comparable single-scale results while further finetuning alone changed performance little.The comparison uses ImageNet ILSVRC-2012 validation results; the baseline achieved 10.07% single-crop and 8.94% convolution errors in the authors’ experiments.Source: §4.3 · pp. 7–8
- Pooling-switch connections helped but were not critical to classification gains; the authors associate the auxiliary loss more with better supervised local optima than regularization.Layer-wise reconstruction loss was effective for regularizing training, while SAE/SWWAE-first showed lower training but higher validation errors, a slight-overfitting pattern.Source: §4.3 · pp. 7–8
- SWWAE-all reduced VGGNet top-1 errors by 1.66% and 1.18%, and top-5 errors by 1.01% and 0.81%, for single-crop and convolution schemes, respectively.The top-5 reductions correspond to 10% and 9% relative to baseline errors.Source: §4.3 · p. 7, Table 1 · p. 8
5. Conclusion
The paper proposes adding reconstructive decoding pathways to large-scale classification networks and reports improved 16-layer VGGNet performance. Its experiments also find that pooling switches help but are not necessary, and that layer-wise reconstruction loss can regularize joint training.
5. Conclusion
- Augmenting 16-layer VGGNet with reconstructive decoding pathways improved classification performance by a noticeable margin.The authors report that pooling switches helped but were not critical, decoding pathways mainly helped the supervised objective reach a better optimum, and layer-wise reconstruction loss effectively regularized the joint objective.Source: §5 · p. 8
A1. Parameters for VGGNet-based models
The VGGNet-based models use layer-wise training parameters and balancing weights for reconstruction losses.
A1. Parameters for VGGNet-based models
- Learning rates were chosen for the largest decrease in each layer’s reconstruction loss during the first 2,000 iterations.The passage refers to the layer-wise training parameters summarized in Table A-1.Source: A1. Parameters for VGGNet-based models · p. 11, Table A-1 · p. 11
- Reconstruction-loss weights balance layers by making their weighted losses comparable in magnitude.The image reconstruction loss uses RGB values normalized to [0,1], unlike intermediate features, and is not normalized by feature dimensions.Source: A1. Parameters for VGGNet-based models · p. 11
A2. More experimental results and discussions
Finetuning the augmented networks retained pretrained filter shapes while improving reconstruction quality, with changes in filter contrast and reconstructed colors. The appendix also compares model variants, training curves, and reconstructions across network architectures.
A2.1. Learned filters
- ∼35% element-wise relative change in filter weights accompanied retained pretrained visual shapes; a small portion of finetuned filters showed stronger contrast.Figure A-1 visualizes first-layer 3 × 3 filters; SWWAE-all filters had nearly the same patterns as pretrained VGGNet filters, but stronger contrast.Source: A2.1. Learned filters · p. 11, Figure A-1 · p. 11
A2.2. Training curve
- Figure A-2 reports SWWAE-all single-crop validation-accuracy training curves, starting from pretrained VGGNet and decoder networks.Source: A2.2. Training curve · p. 11, Figure A-2 · p. 12
A2.3. Selection of different model variants
- SWWAE variants had comparable performance, and similar training costs with identical testing costs permit task-dependent architecture choice.SWWAE suits spatially corresponding tasks such as reconstruction and segmentation; ordinary SAEs remain an option for objectives such as next-frame prediction, where pooling switches are non-transferrable.Source: A2.3. Selection of different model… · p. 12
- S(WW)AE-first has fewer hyperparameters for quick parameter search and can be switched to *-all for better performance.Source: A2.3. Selection of different model… · p. 12
A2.4. Ladder networks
- Removing batch normalization from the ladder network impaired performance because batch normalization was critical for proper noise injection.The authors suggest their models may be easier to pair with standard convolutional networks and train on large-scale datasets.Source: A2.4. Ladder networks · p. 12
A2.5. Image reconstruction
- Finetuning SWWAE-first/all improved reconstruction quality, mainly color accuracy, while SWWAE-all had slightly higher reconstruction loss and better regularization ability.The decoder found learning harder for SWWAE-all than SWWAE-first.Source: A2.5. Image reconstruction · p. 12
- Figures A-4 and A-5 show input reconstructions from pretrained AlexNet and VGGNet features.Source: A2.5. Image reconstruction · p. 12, Figure A-4 · p. 14, Figure A-5 · p. 17
- Figure A-3 evaluates pool5-to-image reconstructions on ILSVRC2012 validation images using L2 distance; its first two examples are validation images excluding the 100 categories, and the rest are not from ImageNet.Source: Figure A-3 · p. 13