PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpolation Consistency Training for Semi-Supervised Learning

Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Arno Solin, Yoshua Bengio, David Lopez-Paz

arXiv:1903.03825v5stat.MLcs.AIcs.LG

TL;DR

Semi-supervised learning seeks to use abundant unlabeled data when labels are scarce and expensive. ICT enforces consistency across interpolated unlabeled examples, outperforming or matching state-of-the-art methods on CIFAR-10 and SVHN while offering theoretical insight into its behavior.

  • Problem

    Semi-supervised learning addresses applications where labels are scarce because obtaining them is time-consuming, costly, or requires specialized expertise.

  • Method

    ICT regularizes semi-supervised learning by matching predictions at interpolated unlabeled inputs with interpolations of predictions at the original points.

  • Results

    ICT outperforms or matches state-of-the-art methods on CIFAR-10 and SVHN, improving the best-reported test error by ∼25% with 4,000 labeled samples.

  • Takeaways & Limitations

    ICT provides an efficient semi-supervised approach and offers theoretical guidance that its use is most reliable with high-confidence values.

  • Takeaways & Limitations

    ICT has a predicted and experimentally confirmed failure mode at low confidence values when pseudo-labels are not used.

Abstract

from arXiv · show

We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm. ICT encourages the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points. In classification problems, ICT moves the decision boundary to low-density regions of the data distribution. Our experiments show that ICT achieves state-of-the-art performance when applied to standard neural network architectures on the CIFAR-10 and SVHN benchmark datasets. Our theoretical analysis shows that ICT corresponds to a certain type of data-adaptive regularization with unlabeled points which reduces overfitting to labeled points under high confidence values.

1. Introduction

The introduction motivates semi-supervised learning for settings with scarce labels and abundant unlabeled data, then proposes ICT to enforce consistency between interpolated inputs and predictions. It presents ICT as an efficient alternative to perturbation-based methods, with competitive benchmark performance and a theoretical analysis of its regularization effects.

  • Motivation: Labeling large datasets is often prohibitive, while many applications provide abundant unlabeled data but scarce labels.The stated constraints include time, financial, and expertise costs.
  • Motivation: Semi-supervised learning leverages unlabeled data to improve supervised learning from small labeled datasets by learning structure such as clusters.The cluster assumption suggests that samples in the same cluster share labels.
  • Prior limitations: Consistency regularization encourages invariant predictions on perturbed unlabeled points, enforcing decision boundaries through low-density regions.Random perturbations are inefficient in high dimensions because few perturbations push boundaries toward low-density regions.
  • ICT: ICT enforces f(αu1 + (1 − α)u2) = αf(u1) + (1 − α)f(u2) for interpolations of two unlabeled points.The method is introduced as an efficient consistency-regularization technique for semi-supervised learning.
  • Results: ICT outperforms or matches state-of-the-art methods on CIFAR10 and SVHN while being simpler and more computation efficient than several recent SSL algorithms.The experiments use CNN-13 and WRN28-2 neural network architectures.
  • Theory: The paper additionally provides a novel theory explaining how and when ICT can succeed or fail.The introduction frames this theory as addressing limited understanding of related interpolation-based methods.

2. Interpolation consistency training

ICT trains predictions at interpolations of unlabeled examples to match interpolated teacher predictions, while combining this consistency objective with supervised learning on labeled data. Interpolations are motivated as efficient perturbations that are likely to enter low-density regions near decision boundaries under the cluster assumption.

  • Method: ICT enforces prediction consistency at interpolations of unlabeled points using a student model and a mean-teacher model.The teacher parameters θ′ are an exponential moving average of the student parameters θ.
  • Motivation: Interpolations between randomly selected unlabeled examples provide a more efficient route toward decision boundaries than random perturbations.The directions approaching a decision boundary occupy only a tiny fraction of the ambient space, whereas interpolation toward another unlabeled example can point toward low density.
  • Motivation: Under the cluster assumption, interpolation toward a randomly selected example is most likely to connect different clusters belonging to different classes, producing a low-density region.The passage states that this case has the highest probability, while same-cluster and different-cluster same-class cases are less likely under the stated assumptions.
  • Optimization: ICT combines supervised cross-entropy loss on labeled minibatches with an unlabeled interpolation-consistency regularizer whose weight increases during training.The consistency term uses teacher predictions as fake labels for two unlabeled minibatches.
  • Optimization: The discrepancy between a mixed unlabeled prediction and the interpolation of its fake labels is measured with a loss, using mean squared error in the experiments.The interpolation coefficient λ is sampled from Beta(α, α) on each update.

3. Experiments

Experiments evaluate ICT on CIFAR-10, SVHN, and CIFAR-100 under standardized semi-supervised protocols using CNN-13 and Wide-ResNet-28-2 architectures. ICT improves over supervised and competing consistency-based methods, with competitive SVHN performance and additional CIFAR-100 evaluation.

  • Experimental setup: Experiments use CIFAR-10, SVHN, and CIFAR-100 with only a fraction of training data labeled, following standardized procedures for fair comparison.The remaining training data is treated as unlabeled data.
  • Experimental setup: ICT is evaluated with CNN-13 and Wide-ResNet-28-2, using the latter setup to compare fairly against consistency-based semi-supervised algorithms.CNN-13 is used without additive Gaussian input noise and with Dropout noise removed.
  • CIFAR-10 and SVHN results: ICT often achieves a two-fold reduction in test error compared with state-of-the-art supervised learning algorithms.The supervised baselines are Supervised(Mixup) and Supervised(Manifold Mixup).
  • CIFAR-10 and SVHN results: ~25% improvement is reported over the best-reported method for CIFAR-10 with 4000 labeled samples.The best max-consistency coefficients for 1000, 2000, and 4000 labels are 10.0, 100.0, and 100.0, while the corresponding α values are 0.2, 1.0, and 1.0.
  • CIFAR-10 and SVHN results: ICT improves over other methods on both CIFAR-10 and SVHN with the WRN-28-2 architecture, while remaining competitive with state-of-the-art SSL methods on SVHN.For SVHN, the selected max-consistency coefficient is 100 and α is 0.1 for all reported ICT results.
  • CIFAR-10 and SVHN results: ICT produces test errors of 14.86 ± 0.39, 9.02 ± 0.12, and 8.23 ± 0.22 on CIFAR-10 with 1000, 2000, and 4000 labeled samples.These results use the best hyperparameters from Table 1 without additional hyperparameter search.
  • CIFAR-100 results: Additional CIFAR-100 experiments use CNN-13 with 100 labeled points per class and no hyperparameter search.The reported results are given in Table 4 using the best hyperparameters from the preceding experiments.

4. Theoretical analysis

The theoretical analysis shows that ICT regularizes derivatives of all orders at unlabeled points and can thereby reduce overfitting at labeled points. Its regularization effect strengthens with higher-confidence unlabeled predictions, supporting better generalization without assuming pseudo-label correctness.

  • 4.2 Regularization of higher-order derivatives: ICT regularizes higher-order derivatives at unlabeled points through real analyticity and properties of the Kronecker product.Theorem 1 establishes this for every derivative order K ∈ N+.
  • 4.2 Regularization of higher-order derivatives: The regularization effect is strongest at low confidence and decreases exponentially as prediction confidence increases.The derivative of the sigmoid term is maximized at fθ(u) = 1/2 and approaches zero toward predictions 0 or 1.
  • 4.2 Regularization of higher-order derivatives: ICT is therefore predicted to work well when confidence on unlabeled data is high, a prediction supported by two-moons simulations varying unlabeled confidence.High confidence can be encouraged with pseudo-labels.
  • 4.3 Reducing overfitting: Regularizing all-order derivative norms at unlabeled points helps reduce overfitting at labeled points and improve generalization.Theorem 2 formalizes this relationship through a probabilistic bound, while subsequent results connect classification error at labeled points to expected error.
  • 4.3 Reducing overfitting: The analysis does not assume pseudo-labels are correct, and increasing the number of unlabeled points reduces labeled-to-unlabeled distance while improving pseudo-label quality.The distance decreases because each labeled point can be closer to one of more unlabeled points.

5. Related work

The related work situates ICT at the intersection of consistency regularization for semi-supervised learning and interpolation-based regularization. It distinguishes ICT by applying interpolation consistency between distant points, extending regularization across larger input regions.

  • Consistency regularization: Consistency-regularization methods keep predictions fθ(u) stable under realistic perturbations u + δ of unlabeled samples and move decision boundaries toward low-density regions.These methods are motivated by the low-density separation assumption.
  • ICT’s distinction: ICT’s regularization applies to larger regions of the input space when interpolating between distant points.The broader spatial effect follows directly from using distant interpolation endpoints.
  • Interpolation-based regularization: Interpolation-based regularizers have achieved state-of-the-art performance in supervised learning across varied tasks and architectures, using input-space or hidden-space interpolation.Tokozume et al. and Zhang et al. interpolate inputs, whereas Verma et al. also interpolates hidden representations.

6. Conclusion

The paper concludes that ICT is a simple, efficient semi-supervised learning algorithm motivated by the scarcity and cost of labeled data. Its analysis and experiments support using high confidence values, while future work could extend interpolation to hidden representations.

  • Conclusion: ICT addresses semi-supervised learning settings where labeled data are limited or expensive to obtain.The conclusion motivates semi-supervised learning by its potential for applications constrained by scarce labels.
  • Conclusion: ICT is presented as a simple but efficient semi-supervised learning algorithm.The conclusion describes ICT as having advantages over previous approaches, though the supplied passage truncates their details.
  • Conclusion: High confidence values are recommended because ICT’s low-confidence mode without pseudo-labels failed in experiments.The theoretical prediction of failure was confirmed experimentally, providing practical guidance for ICT use.
  • Conclusion: Future work could extend ICT interpolation from inputs to hidden representations to improve performance further.This proposed direction is associated with interpolating at hidden representations.

Appendix A. Additional lemmas

Appendix A states two auxiliary lemmas used in the theorem proofs, with proofs following established results from prior work. One lemma uses McDiarmid’s inequality for an unlabeled-data-dependent function class, while the other states Talagrand’s contraction lemma.

  • Appendix A. Additional lemmas: The appendix introduces lemmas used in proving the paper’s theorems, whose proofs directly follow previous works.The cited prior works are Bartlett & Mendelson (2002), Mohri & Medina (2014), and Mohri et al. (2012).
  • Lemma 4: Lemma 4 gives a high-probability bound for a C-uniformly bounded loss over a function class depending on an unlabeled dataset.The bound holds for any δ > 0 with probability at least 1 − δ over an i.i.d. draw of m labeled samples.
  • Lemma 4: Lemma 4 follows because changing one sample changes the empirical Rademacher quantity by at most C/m, enabling McDiarmid’s inequality.The proof explicitly attributes the result to this bounded-difference argument.
  • Lemma 5: Lemma 5 states Talagrand’s contraction inequality for μ-Lipschitz functions applied to samples of m points.The lemma is attributed to Ledoux & Talagrand (2013) and Mohri & Medina (2014), with its proof referred to Mohri and Medina (2014, Lemma 8).

Appendix B. Additional experiments · B.1. Kullback–Leibler (KL) divergence

This appendix examines replacing ICT’s usual MSE consistency loss with KL divergence and finds that ICT continues to behave as expected across activation functions. The preliminary results remain consistent with the theoretical importance of the confidence value, motivating more comprehensive future study.

  • B.1. Kullback–Leibler (KL) divergence: ICT’s consistency loss is typically computed with MSE, following prior semi-supervised learning studies.The authors found MSE sufficient for their problems but identify KL divergence as an alternative.
  • B.1. Kullback–Leibler (KL) divergence: The appendix replaces MSE with KL divergence in the experiments reported in Fig. 3.The confidence value is defined by 1 minus the predictor’s boundary value.
  • B.1. Kullback–Leibler (KL) divergence: With KL divergence and softplus activation, ICT produces the decision boundaries shown in Fig. 4.The figure reports decision boundaries for ICT using KL divergence with the softplus activation.
  • B.1. Kullback–Leibler (KL) divergence: With KL divergence and ReLU activation, ICT produces the decision boundaries shown in Fig. 5.The figure reports decision boundaries for ICT using KL divergence with the ReLU activation.
  • B.1. Kullback–Leibler (KL) divergence: The appendix also examines ICT with few data points using 1 or 5 labeled points per class and 10 or 100 unlabeled points.Fig. 6 presents three labeled–unlabeled data configurations.
  • B.1. Kullback–Leibler (KL) divergence: Across updates of 1, 10, 100, and 1000, KL-based ICT works as expected and remains numerically consistent with the theoretical role of confidence.The authors describe these as preliminary results and suggest studying KL divergence in ICT more comprehensively.

B.2. The effect of few data points

ICT can fail to learn the correct decision boundary when labeled or unlabeled data are too scarce. Increasing only the number of unlabeled points improves and further refines the learned boundary, consistent with theoretical predictions.

  • The effect of few data points: ICT may not learn the correct decision boundary with only one labeled point per class or too few unlabeled points.These cases are illustrated in Fig. 6a and Fig. 6b.
  • The effect of few data points: Increasing unlabeled points from 10 to 100 enables ICT to start learning a good decision boundary.Only the number of unlabeled points changes between Fig. 6b and Fig. 6c.
  • The effect of few data points: Further increasing the unlabeled data points refines the decision boundary, consistent with the theoretical prediction.The refinement is reported for Figs. 1 and 3 in the main text.

B.3. Using the ReLU activation in the experiments of Fig. 3

The Fig. 3 experiments used softplus with κ = 100, while an additional ReLU experiment in Fig. 7 produced numerical results consistent with the theoretical prediction. This consistency follows because softplus can approximate ReLU arbitrarily well by varying κ ∈(0, ∞).

  • Activation functions: The Fig. 3 experiments used the softplus activation φ(z) = ln(1 + exp(κz))/κ with κ = 100.The activation parameter was fixed at κ = 100.
  • Activation functions: Replacing softplus with ReLU in an additional experiment produced the results reported in Fig. 7.The ReLU activation was evaluated as an additional experiment.
  • Activation functions: The numerical results with ReLU were consistent with the theoretical prediction because softplus approximates ReLU arbitrarily well as κ varies over κ ∈(0, ∞).This approximation explains why the ReLU results were expected to agree with the theoretical prediction.
Loading 1903.03825v5…