Source-linked AI summary
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
Mehdi Sajjadi, Mehran Javanmardi, Tolga Tasdizen
TL;DR
Limited labeled data and the cost of annotation motivate semi-supervised learning for ConvNets. The paper introduces a stochastic-consistency loss and combines it with mutual-exclusivity loss, reporting significant accuracy improvements across benchmark datasets and ConvNet implementations.
Problem
ConvNets can overfit with limited labeled data, while creating large labeled datasets is costly and unlabeled data are readily available.
Method
The paper minimizes prediction differences across stochastic passes and combines transformation/stability with mutual-exclusivity loss for semi-supervised ConvNet training.
Results
Experiments across multiple benchmark datasets and two ConvNet implementations show significant accuracy improvements, including state-of-the-art error rates on CIFAR10 and CIFAR100.
Takeaways & Limitations
Stochastic consistency regularization provides a competitive semi-supervised learning method when only a small number of labeled examples are available.
Abstract
from arXiv · showhide
Effective convolutional neural networks are trained on large sets of labeled data. However, creating large labeled datasets is a very costly and time-consuming task. Semi-supervised learning uses unlabeled data to train a model with higher accuracy when there is a limited set of labeled data available. In this paper, we consider the problem of semi-supervised learning with convolutional neural networks. Techniques such as randomized data augmentation, dropout and random max-pooling provide better generalization and stability for classifiers that are trained using gradient descent. Multiple passes of an individual sample through the network might lead to different predictions due to the non-deterministic behavior of these techniques. We propose an unsupervised loss function that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network. We evaluate the proposed method on several benchmark datasets.
1 Introduction
ConvNets can overfit when labeled data are limited, while acquiring large labeled datasets is costly and unlabeled data are plentiful. The paper exploits stochastic training behavior by adding an unsupervised loss that stabilizes predictions across multiple passes.
- ConvNets’ large parameter counts can cause overfitting without abundant labeled training data.ImageNet required substantial manual effort and still barely contained enough samples per category to prevent overfitting.
- Unlabeled images and video are comparatively cheap and widely available for semi-supervised learning.
- Randomized transformations, dropout, and pooling make repeated passes of one sample produce different predictions.
- The proposed unsupervised loss minimizes prediction differences across stochastic passes to improve testing-time generalization stability.
- Combining the loss with supervised learning yields a competitive semi-supervised ConvNet method across experiments.
2 Related Work
Prior semi-supervised learning methods include self-training, co-training, generative models, margin-based methods, graphs, embedding, video-based, ladder-network, and surrogate-class approaches. These methods exploit unlabeled data through confidence, feature structure, probability modeling, similarity, representations, temporal correlations, or transformed instances.
- Self-training adds confident classifier predictions to the training set, whereas co-training transfers strong predictions between classifiers trained on disjoint feature subsets.Co-training assumes the two feature sets are conditionally independent.
- Generative methods model the joint distribution of training data and labels, while TSVM and S3VM seek maximum-margin boundaries using labeled and unlabeled data.
- Graph-based methods exploit similarities between samples to incorporate unlabeled data.
- Deep semi-supervised ConvNet approaches pre-train filters, embed representations, use video-frame correlations, or jointly minimize supervised and unsupervised losses.
- A surrogate-class method creates transformed instances of each unlabeled sample, but requires a separate class per sample and scales poorly with unlabeled data.
3 Method
The method regularizes ConvNets by requiring predictions to remain consistent across random input transformations and internal perturbations. A transformation/stability loss is paired with mutual-exclusivity loss to prevent trivial unlabeled predictions, and the combination can be optimized with supervised loss.
- The consistency objective requires a sample’s prediction to remain stable under random data transformations, dropout, and randomized pooling.
- The transformation/stability loss compares predictions from multiple stochastic passes of each training sample.It minimizes the sum of squared differences between every pair of pass predictions using gradient descent.
- The transformation/stability loss can be combined with any supervised loss function.
- Mutual-exclusivity loss forces each prediction vector toward a single non-zero element and prevents trivial unlabeled solutions.
- Combining transformation/stability and mutual-exclusivity losses produces further accuracy improvements in experiments.
4 Experiments
Experiments across MNIST, SVHN, NORB, CIFAR10, CIFAR100, and ILSVRC 2012 evaluate stochastic unsupervised losses with convolutional networks under limited-label settings. Across these evaluations, adding the unsupervised losses improves classifier accuracy, with especially strong results reported for NORB and CIFAR10.
- Experimental setup: Experiments span MNIST, SVHN, NORB, CIFAR10, CIFAR100, and ILSVRC 2012 using two convolutional-network implementations.The implementations are cuda-convnet and sparse convolutional networks with fractional max-pooling.
- MNIST: MNIST uses 100 labeled samples, all remaining training data as unlabeled data, dropout, and randomized fractional max-pooling without data augmentation.The transformation T_j(x_i) is the identity for MNIST, so stochastic regularization comes from dropout and fractional max-pooling.
- SVHN: SVHN experiments compare labeled-only training with mutual-exclusivity, transformation/stability, and combined unsupervised losses across 1%, 5%, 10%, 20%, and 100% labeled data.The experiments use both cuda-convnet and sparse convolutional networks, with five independently sampled 1% labeled subsets in the latter setting.
- NORB: NORB experiments report that unsupervised losses significantly improve accuracy with few labels, and 1% labeled data achieves accuracy close to using 100% labeled data.The sparse-network experiments use dropout and randomized max-pooling without data augmentation.
- CIFAR10: CIFAR10 achieves a 3.18% ± 0.1 mean and standard deviation error rate, compared with 3.47% for fractional max-pooling and 3.00% using a larger 160n model.The authors report surpassing state-of-the-art accuracy by adding unsupervised loss functions, with 100 test-time passes.
- CIFAR100: CIFAR100 achieves a 21.43% ± 0.16 mean and standard deviation error rate, reported as state-of-the-art with 12 test-time passes.Because no data augmentation is used, the proposed loss targets randomness from dropout and max-pooling.
5 Discussion
Across architectures, implementations, and benchmark datasets, the proposed loss improves ConvNet accuracy, including strong error rates on CIFAR100 and CIFAR10.
- The proposed loss improves ConvNet accuracy across different architectures and implementations, including cuda-convnet and sparse convolutional networks.
- For SVHN and NORB, accuracy significantly improves with few labeled samples using either random transformations or dropout and randomized pooling.
- 21.43% error rate is achieved on CIFAR100 using dropout, randomized pooling, and the proposed loss combination.
- 3.00% error rate is achieved on CIFAR10 using data transformation, dropout, and randomized pooling with the proposed method.
6 Conclusion
The paper proposes an unsupervised stability loss for stochastic ConvNet passes and evaluates it with two implementations across benchmark datasets. Combined with mutual-exclusivity loss, it improves accuracy when labeled data are scarce.
- The unsupervised loss minimizes prediction variations across passes caused by stochastic transformations, dropout, and randomized max-pooling.
- The method is evaluated with two ConvNet implementations on multiple benchmark datasets.
- Significant accuracy improvements are obtained with few labeled examples by combining transformation/stability loss with mutual-exclusivity loss.