Source-linked AI summary
Label Propagation for Deep Semi-supervised Learning
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Ondrej Chum
TL;DR
Deep semi-supervised learning needs ways to exploit abundant unlabeled data because reliable human annotations are scarce. The paper alternates between embedding-based nearest-neighbor label propagation and neural-network training, and reports better performance on standard datasets, especially with few labels.
Problem
Reliable human-annotated visual data are scarce, while transductive semi-supervised methods and the manifold assumption have not been fully exploited in modern deep learning.
Method
The method alternates classifier training with transductive label propagation on a nearest-neighbor graph constructed from the classifier’s embeddings, using inferred labels and certainty weights for unlabeled data.
Results
The method outperforms other semi-supervised approaches on standard datasets, with a more pronounced advantage when fewer labeled data are available.
Takeaways & Limitations
Graph-based pseudo-labels provide a complementary training signal to unsupervised loss terms and are reported to be more valuable than pseudo-labels inferred by the network itself.
Takeaways & Limitations
The label-propagation objective assumes nearby examples should receive the same predictions while preserving predictions on labeled examples.
Abstract
from arXiv · showhide
Semi-supervised learning is becoming increasingly important because it can combine data carefully labeled by humans with abundant unlabeled data to train deep neural networks. Classic methods on semi-supervised learning that have focused on transductive learning have not been fully exploited in the inductive framework followed by modern deep learning. The same holds for the manifold assumption---that similar examples should get the same prediction. In this work, we employ a transductive label propagation method that is based on the manifold assumption to make predictions on the entire dataset and use these predictions to generate pseudo-labels for the unlabeled data and train a deep neural network. At the core of the transductive method lies a nearest neighbor graph of the dataset that we create based on the embeddings of the same network.Therefore our learning process iterates between these two steps. We improve performance on several datasets especially in the few labels regime and show that our work is complementary to current state of the art.
1. Introduction
Deep neural networks perform well but require abundant labeled data, while human annotation remains scarce and costly. The paper addresses this gap by iterating between graph-based label propagation and classifier training, improving semi-supervised performance especially with few labels.
- Deep neural networks offer efficient, high-performing computer vision models but require large amounts of labeled training data.
- Human-annotated visual data are scarce, costly, impractical to obtain at scale, and vulnerable to annotation errors.
- The method uses transductive label propagation to infer pseudo-labels for unlabeled data and train a classifier with them.
- A nearest-neighbor graph is built from the classification network’s embeddings, and the process alternates between network training and label propagation.
- The proposed approach outperforms other semi-supervised methods on standard datasets, with a more pronounced advantage when fewer labels are available.
2. Related work
Deep semi-supervised learning commonly uses either unsupervised loss terms or pseudo-labels, while prior label propagation work largely targets transductive settings. This paper places graph-based pseudo-labeling in deep network training and presents it as complementary to unsupervised-loss methods.
- Prior deep semi-supervised methods mainly add unsupervised loss terms or assign pseudo-labels to unlabeled examples.
- The proposed contribution belongs to pseudo-labeling and is conceptually and implementation-wise orthogonal to unsupervised-loss methods.
- Unsupervised loss in deep SSL: Unsupervised-loss approaches encourage confident or consistent predictions across images, perturbations, or related networks.
- Pseudo-labeling in deep SSL: Pseudo-labeling methods use a network’s most confident class predictions as labels in supervised cross-entropy training.
- Unlike prior online propagation with fixed descriptors, this work performs label propagation offline on the training set while the network is being trained.
3. Preliminaries
This section formulates semi-supervised classification, reviews supervised, pseudo-label, and consistency losses, and introduces transductive label propagation based on graph smoothness and label preservation.
- Problem formulation: Semi-supervised learning uses labeled and unlabeled examples to train a classifier that maps previously unseen samples to class labels.The classifier maps inputs to class confidence scores through parameters θ.
- Supervised and pseudo-label losses: The supervised loss applies only to labeled examples, with cross-entropy given by ℓs(s, y) := −log sy.
- Supervised and pseudo-label losses: Pseudo-labeling assigns labels to unlabeled examples and adds their supervised loss to the training objective.The pseudo-label loss can use a supervised loss such as cross-entropy.
- Consistency loss: Consistency loss trains outputs for original and transformed examples to remain similar under data or network transformations.A simple choice is squared Euclidean distance between the two output vectors.
- Transductive learning: Transductive learning infers labels for a specific dataset rather than targeting only generic generalization to unseen data.
- Label propagation: Diffusion computes an n × c prediction matrix from one-hot labels for labeled examples and zero entries elsewhere, with α ∈ [0, 1).For an unlabeled example, the class prediction is selected from the corresponding row of Z.
- Label propagation: The label-propagation objective balances smoothness among nearby examples with preservation of predictions for labeled examples.Its minimizer is matrix Z, whose rows contain class predictions or scores.
4. Method
The method alternates between constructing a nearest-neighbor graph from the network’s embeddings, propagating labels to obtain pseudo-labels, and retraining the network with weighted supervision. It addresses pseudo-label uncertainty and class imbalance while remaining compatible with unsupervised-loss approaches.
- Overview: The approach iteratively builds a nearest-neighbor graph from network descriptors, propagates labels transductively, and retrains on true and pseudo-labels.The process begins with supervised training on labeled examples, then repeats graph construction, label propagation, and network optimization.
- Weighting: Pseudo-label weights reflect prediction certainty, while inverse class-population weights address imbalance among labeled and pseudo-labeled classes.Entropy is used to assign per-example certainty weights, and class weights are defined from labeled and pseudo-labeled populations.
- Illustration: The toy example visualizes labeled examples as triangles, pseudo-labeled examples as circles, class by color, and certainty weights by circle size.It demonstrates label propagation for different numbers of labeled examples.
- Combination with other approaches: The proposed pseudo-labeling method is orthogonal to unsupervised-loss methods and can be combined with them by adding the corresponding loss term.The paper reports combining the method with Mean Teacher in experiments.
- Algorithm: The iterative procedure extracts descriptors, constructs normalized affinities, solves the diffusion system, assigns pseudo-labels, and optimizes a weighted loss.The algorithm uses conjugate gradients for diffusion and certainty and class weights before the next optimization step.
- Label propagation: Label propagation is computed outside the main neural-network optimization, and the network is trained toward the resulting transductive predictions.This avoids backpropagating through the label-propagation computation while directly incorporating its outputs into training.
5. Experiments
Experiments evaluate the method across CIFAR-10, CIFAR-100, and Mini-ImageNet using reproduced baselines and ablations. Diffusion-based pseudo-labels, uncertainty weighting, and combinations with unsupervised loss generally improve performance, especially with fewer labels.
- Training details: The implementation uses a 13-layer network for CIFAR-10 and CIFAR-100 and ResNet-18 for Mini-ImageNet, with unit-norm embeddings for graph construction.The embedding normalization is inserted after the feature extractor and before the fully connected layer.
- Ablation study: The ablations evaluate uncertainty weights, class weights, and diffusion-based versus network-based pseudo-labeling on CIFAR-10 with 500 labels.The study also compares diffusion with GTG propagation and network predictions.
- Ablation study: Diffusion predictions are consistently more accurate than network predictions and can provide accurate pseudo-labels beyond examples the network already classifies.This comparison is tracked throughout training on CIFAR-10 with 500 labeled images.
- Ablation study: The weighting mechanism estimates prediction certainty, becomes more reliable during training, and is described as robust to incorrect pseudo-labels.Incorrect high-certainty labels commonly involve trucks classified as automobiles or birds classified as frogs.
- Comparison with the state of the art: The method performs best among pseudo-label approaches on CIFAR-10, with a larger advantage as the number of labels decreases.Combining it with Mean Teacher achieves the best CIFAR-10 performance, while on Mini-ImageNet with 4k labels the uncombined method performs best.
6. Conclusions
The paper proposes graph-based label propagation for generating pseudo-labels and forms an additional training set from them. It concludes that the approach is complementary to unsupervised loss terms.
- Conclusions: The method relies on graph-based label propagation to infer pseudo-labels for unlabeled images and use them in network training.These pseudo-labels form an additional training set.
- Conclusions: The paper reports that diffusion-inferred pseudo-labels are more valuable than pseudo-labels inferred directly by the network.The conclusion frames this as the central distinction from recent deep semi-supervised approaches.
- Conclusions: The approach is complementary to unsupervised loss terms, as demonstrated experimentally through combination with such methods.The conclusion specifically identifies complementarity rather than replacement.