Source-linked AI summary
Smooth Neighbors on Teacher Graphs for Semi-supervised Learning
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, Bo Zhang
TL;DR
Self-ensembling SSL methods enforce prediction consistency under perturbations but overlook connections among data samples. SNTG constructs a teacher-prediction graph and smooths neighboring representations on a low-dimensional manifold. It achieves state-of-the-art benchmark results and robustness to noisy labels.
Problem
Perturbation-based self-ensembling methods consider each unlabeled point separately and ignore connections among samples, underusing cluster and manifold structure.
Method
SNTG builds a graph from teacher-model predictions and enforces similar representations for neighboring points while separating non-neighbors.
Results
SNTG achieves state-of-the-art results across several SSL datasets, improves benchmark performance, and retains over 93% accuracy with 90% noisy labels in TempEns+SNTG.
Takeaways & Limitations
SNTG offers additional benefits in settings with fewer labels and noisy labels, while integrating with existing SSL methods without extra network parameters.
Takeaways & Limitations
The teacher graph may be inaccurate early in training, so convergence depends on a ramp-up that initially lets supervised loss dominate.
Abstract
from arXiv · showhide
The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data samples. In this paper, we propose a novel method, called Smooth Neighbors on Teacher Graphs (SNTG). In SNTG, a graph is constructed based on the predictions of the teacher model, i.e., the implicit self-ensemble of models. Then the graph serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold. We achieve state-of-the-art results on semi-supervised learning benchmarks. The error rates are 9.89%, 3.99% for CIFAR-10 with 4000 labels, SVHN with 500 labels, respectively. In particular, the improvements are significant when the labels are fewer. For the non-augmented MNIST with only 20 labels, the error rate is reduced from previous 4.81% to 1.36%. Our method also shows robustness to noisy labels.
1. Introduction
Deep semi-supervised methods use perturbation consistency, but typically ignore relationships among unlabeled samples. SNTG adds a teacher-derived graph to smooth neighboring representations and improves semi-supervised performance.
- Semi-supervised learning leverages limited labeled data and abundant unlabeled data because fully labeled datasets are expensive and time-consuming.
- Perturbation-based methods regularize predictions around individual data points to encourage smoothness and robustness.
- Existing perturbation methods ignore connections among data points, leaving unlabeled clusters and manifolds underused.
- SNTG constructs a teacher graph from teacher-generated targets and uses neighboring relationships to induce smoothness on the data manifold.
- SNTG makes learned features similar for graph neighbors and dissimilar for non-neighbors in a low-dimensional feature space.
- SNTG requires no extra network parameters, integrates with existing deep SSL methods, and incurs negligible extra time cost.
2. Related work
Related SSL work includes self-training, graph-based, entropy-based, perturbation, and generative approaches. Traditional graph methods generally rely on predefined input-space similarities, whereas newer deep methods address SSL through learned predictions or distributions.
- Self-training uses high-confidence predictions to label unlabeled examples, while co-training iteratively learns from classifiers using disjoint data views.
- Transductive SVMs implement the cluster assumption by keeping unlabeled data away from decision boundaries, while entropy minimization encourages confident class assignments.
- Graph-based SSL methods define similarities with graphs and smooth predictions using graph-based regularization or label propagation.
- Many traditional graph methods construct graphs in advance using prior knowledge or predefined distance metrics, commonly in input space.
- Generative approaches model the input distribution p(x), including Gaussian mixtures, variational auto-encoders, and generative adversarial networks.
3. Background
Semi-supervised classification combines supervised loss with regularization that exploits unlabeled data. Perturbation-based self-ensembling methods generate teacher targets and enforce prediction consistency under noise, with several teacher-update strategies.
- The SSL objective combines labeled-data loss with a regularization term that leverages unlabeled examples.
- The prediction function outputs a K-class distribution, while the regularization weight controls the strength of unlabeled-data regularization.
- Perturbation-based methods: Self-ensembling methods use a teacher to generate training targets and a student trained for consistency under perturbations.
- Perturbation-based methods: The teacher is an implicit ensemble of previous student models, and consistency may be measured with l2 distance or KL divergence.
- Perturbation-based methods: Π model evaluates shared-parameter teacher and student networks under independent perturbations and minimizes their prediction distance.
- Perturbation-based methods: Temporal ensembling averages predictions across epochs with an EMA, while mean teacher averages parameters every iteration to update targets more frequently.
- Perturbation-based methods: VAT defines the consistency regularizer as KL divergence under adversarial input perturbations and assumes worst-case perturbation training improves generalization.
- Perturbation-based methods: Self-generated targets can be incorrect, but better teachers and balanced supervised-consistency weighting support convergence and robustness to incorrect labels.
4. Our approach
SNTG constructs a teacher-prediction graph and uses it to regularize low-dimensional features, making neighboring examples similar and non-neighbors separated. It combines this graph-based smoothness with standard single-point consistency regularization and uses sampling to reduce computational cost.
- 4. Our approach: SNTG addresses the limitation of perturbation methods that enforce only local smoothness around individual points by enforcing smoothness among graph neighbors.The paper characterizes neighbor-based smoothness as stronger regularization because it incorporates cluster structure and data-manifold information.
- 4.1. Learning the graph with the teacher model: SNTG learns a teacher graph from teacher-generated targets because unlabeled examples lack true labels and input-space distances may not capture semantic similarity.The graph treats examples with matching teacher predictions as neighbors; alternative similarities such as KL divergence between soft predictions are also possible.
- 4.2. Guiding the low-dimensional feature mapping: The student decomposes as f = g◦h, and SNTG guides the penultimate-layer mapping h rather than directly comparing class-probability outputs.The resulting feature space is intended to support semantic similarity and linear separation.
- 4.2. Guiding the low-dimensional feature mapping: The contrastive graph loss pulls neighboring features together and pushes non-neighboring features apart by at least margin m.For neighbors it uses Euclidean feature distance; for non-neighbors it penalizes distances below the predefined margin.
- 4. Our approach: The overall objective combines labeled-data cross-entropy with single-point consistency and neighboring-point smoothness, using a ramp-up weight during training.Training uses mini-batches and neural-network optimization; the graph regularizer can be estimated from sampled pairs.
- 4. Our approach: Doubly stochastic pair sampling reduces the cost of evaluating all feature distances, whose full mini-batch computation is O(n^2p).The quadratic dependence on mini-batch size motivates the sampling procedure for large batches.
5. Experiments
Experiments across synthetic datasets and standard SSL benchmarks show that SNTG improves predictions and representations by using neighboring structure, including under scarce or noisy labels.
- Synthetic datasets: With only 12 labeled points per synthetic dataset, SNTG correctly classifies regions where the Π model fails by utilizing neighboring-point structure.The synthetic experiments use two moons and four spins, each with 6000 data points and a 0.002 label ratio.
- Benchmark datasets: SNTG surpasses previous state-of-the-art methods across MNIST, SVHN, CIFAR-10, and CIFAR-100 benchmark settings.Results are averaged over 10 runs with different data-split seeds, using the same architectures and hyperparameters as the baselines.
- Benchmark datasets: 13.15% to 12.49% and 10.55% to 9.89%: adding SNTG improves VAT+Ent error rates on CIFAR-10 without and with augmentation, respectively.The comparison uses VAT with entropy minimization, described as a stronger baseline than the Π model and TempEns.
- Benchmark datasets: When labels are scarce, SNTG provides especially significant benefits on MNIST with 20 labels, SVHN with 250 labels, and CIFAR-10 with 1000 labels.The authors attribute this improvement to reduced overfitting on the small labeled set and better generalization.
- Ablation study: 15.36% versus 16.55%: the neighbor-aware SNTG loss alone outperforms the Π model’s consistency loss in the CIFAR-10 ablation.The two regularizers are reported to be complementary when combined.
- Comparison to other graphs: 13.62% error rate: learning a teacher graph from scratch outperforms Π, input-space k-NN, and a fixed prediction-based graph on CIFAR-10.The corresponding error rates are 16.55% for Π, 16.13% for k-NN in X, and 15.71% for the fixed graph in Y.
- Representations and noisy labels: SNTG produces more concentrated class clusters in learned embeddings and retains over 93% accuracy when 90% of SVHN labels are noisy.TempEns alone obtains 73% accuracy in the same noisy-label setting.
- Feature matching GAN: 14.93% versus 18.63%: applying SNTG to FM GAN reduces CIFAR-10 error and produces samples with improved quality.The resulting error is comparable to the 14.41% state of the art reported for deep generative models.
6. Conclusions and future work
The paper presents SNTG as a simple, effective regularizer that smooths neighboring points on a learned teacher graph. It reports strong empirical performance, including benefits with fewer or noisy labels, and identifies several directions for future work.
- SNTG regularizes neighboring points on a learned teacher graph and learns an invariant mapping on a low-dimensional manifold.
- SNTG outperforms all baselines and achieves new state-of-the-art results on several datasets.
- SNTG offers additional benefits in settings with fewer labels and noisy labels.
- Future work includes theoretical analysis, combinations with generative models, and applications to large-scale datasets such as ImageNet.
A. Experimental setup
The experiments use standard SSL benchmarks and baseline-matched implementations, adding the SNTG loss while largely preserving baseline settings. SNTG adds little computational overhead and no neural-network parameters.
- Experiments cover MNIST, SVHN, CIFAR-10, CIFAR-100, and synthetic benchmarks under specified preprocessing and augmentation settings.
- The implementation uses Theano and Lasagne, with TensorFlow used to match VAT and Mean Teacher settings; reproduction code is available online.
- Π model and TempEns experiments use 300 training epochs, Adam optimization, mini-batches of size n = 100, and mean-only batch normalization with momentum 0.999.
- For Mean Teacher and VAT comparisons, the experiments retain the corresponding baseline architectures and hyper-parameters while adding the SNTG loss.
- The SNTG margin is m = 1, and the regularization coefficient is selected through validation from ratios k in {0.2, 0.4, 0.6, 1.0}.
- SNTG does not increase neural-network parameters and adds only 1-2 seconds per epoch over baselines.
B. Rethinking Π model objective
The Π model consistency objective penalizes prediction variance caused by perturbations. This connects consistency regularization to discouraging unstable predictions, especially near decision boundaries.
- The Π model defines consistency loss as the expected squared prediction norm minus the squared expected prediction, equivalently summing prediction variance across dimensions.
- Minimizing RC penalizes prediction changes under independent random perturbations applied to the model.
- Predictions near a decision boundary are likely to have larger variance because noise can alternate the predicted class.
- Table 7 compares error rates on MNIST with 600 labels across several classical SSL methods.
C. Comparison to classical SSL methods
Classical graph-based SSL methods construct graph-based smoothness objectives, but recent deep SSL methods generally achieve stronger performance and scalability. SNTG is reported to surpass the classical methods on the shared MNIST comparison.
- These methods commonly optimize supervised loss with a graph Laplacian regularizer or propagate labels across graph neighbors.
- Classical graph-based methods include label propagation, LP+kNN, LGC, TSVM, LapRLS, and dynamic label propagation.
- Recent deep-learning SSL methods mostly outperform classical methods in performance and scalability.
- Classical methods lack reported results on challenging natural-image datasets such as SVHN and CIFAR-10, with MNIST being the overlap used for comparison.
- SNTG surpasses the classical SSL methods in the reported MNIST comparison.
D. Significance test of the improvements.
Table 8 reports an independent two-sample t-test comparing the error rates of baselines and SNTG. All p-values are below α = 0.01, indicating statistically significant improvements for SNTG across the reported experiments.
- All p-values are below the significance level α = 0.01 for the independent two-sample t-test comparing SNTG with baselines.The test uses error rates from the baselines and SNTG.
- The reported improvements of SNTG are statistically significant according to Table 8.
- Table 8 separates experiments without augmentation in its top rows from experiments with augmentation in its bottom rows.