PaperlayerView exact revision on arXiv

Source-linked AI summary

SimPLE: Similar Pseudo Label Exploitation for Semi-Supervised Classification

Zijian Hu, Zhengyu Yang, Xuefeng Hu, Ram Nevatia

arXiv:2103.16725v2cs.CVcs.LG

TL;DR

Semi-supervised learning seeks to improve classification from limited labeled data by exploiting abundant unlabeled data, while relationships among different unlabeled samples remain less studied. SimPLE adds Pair Loss to MixMatch-family techniques, and reports gains over prior state-of-the-art methods on several benchmarks and in transfer learning.

  • Problem

    Semi-supervised learning must generalize information from limited labeled data and better exploit abundant unlabeled data, while relationships among different unlabeled samples remain less studied.

  • Method

    SimPLE combines MixMatch-family training with Pair Loss, which minimizes statistical distance between similar unlabeled samples with high-confidence pseudo labels.

  • Results

    SimPLE outperforms prior state-of-the-art methods on CIFAR-100 and Mini-ImageNet, is on par on CIFAR-10 and SVHN, and also outperforms them in transfer learning.

  • Takeaways & Limitations

    Modeling relationships among similar high-confidence unlabeled samples provides the basis for SimPLE’s reported gains across standard benchmarks and transfer learning.

  • Takeaways & Limitations

    In a DomainNet-Real to Mini-ImageNet experiment, pre-trained models had slightly lower test accuracy than training from scratch despite converging approximately 8 to 10 times faster.

Abstract

from arXiv · show

A common classification task situation is where one has a large amount of data available for training, but only a small portion is annotated with class labels. The goal of semi-supervised training, in this context, is to improve classification accuracy by leverage information not only from labeled data but also from a large amount of unlabeled data. Recent works have developed significant improvements by exploring the consistency constrain between differently augmented labeled and unlabeled data. Following this path, we propose a novel unsupervised objective that focuses on the less studied relationship between the high confidence unlabeled data that are similar to each other. The new proposed Pair Loss minimizes the statistical distance between high confidence pseudo labels with similarity above a certain threshold. Combining the Pair Loss with the techniques developed by the MixMatch family, our proposed SimPLE algorithm shows significant performance gains over previous algorithms on CIFAR-100 and Mini-ImageNet, and is on par with the state-of-the-art methods on CIFAR-10 and SVHN. Furthermore, SimPLE also outperforms the state-of-the-art methods in the transfer learning setting, where models are initialized by the weights pre-trained on ImageNet or DomainNet-Real. The code is available at github.com/zijian-hu/SimPLE.

1. Introduction

Semi-supervised learning addresses the cost of labeling by combining limited labeled data with abundant unlabeled data. SimPLE extends consistency-based methods with Pair Loss, which uses relationships among similar, high-confidence unlabeled samples and achieves strong benchmark results.

  • Motivation: Semi-supervised learning combines limited labeled data with abundant unlabeled data to reduce reliance on expensive annotation.It uses information from both labeled and unlabeled examples rather than exclusively exploiting either source.
  • Motivation: Existing semi-supervised methods study consistency across augmentations and relationships between labeled and unlabeled data, but relationships among different unlabeled samples are less studied.This gap motivates directly modeling similarities between unlabeled examples.
  • Method: SimPLE introduces Pair Loss, which minimizes the distance between similar unlabeled samples with high-confidence pseudo labels.The method targets information propagation among unlabeled samples beyond consistency between augmented views of the same sample.
  • Method: SimPLE combines MixMatch-family techniques with supervised loss, unsupervised consistency loss, and the new Pair Loss.Pseudo labels are generated by averaging and sharpening predictions from multiple weakly augmented views, then used with labeled data and strongly augmented samples.
  • Results: SimPLE outperforms state-of-the-art methods on CIFAR-100 and Mini-ImageNet and is on par with them on CIFAR-10 and SVHN.The reported results come from extensive experiments on standard benchmarks.
  • Results: In transfer learning, SimPLE also outperforms current state-of-the-art methods when models are initialized with pre-trained weights.The evaluation uses models pre-trained on ImageNet or DomainNet-Real.

2. Related Work

The related work covers consistency regularization, augmentation anchoring, pseudo-labeling, and label propagation as ways to exploit unlabeled data in semi-supervised learning.

  • Consistency Regularization: Consistency regularization keeps model responses consistent when inputs or models are perturbed.Perturbations include data augmentation, dropout, random max pooling, adversarial transformations, temporal ensembling, and parameter perturbations.
  • Augmentation Anchoring: Augmentation Anchoring aligns predictions for severely perturbed inputs with predictions from slightly perturbed anchor inputs.ReMixMatch introduced the technique, and FixMatch further developed it; the less unstable anchor improves regularization stability.
  • Pseudo-labeling: Pseudo-labeling trains models with artificial labels generated from their own predictions.Confidence filtering rejects pseudo labels when max(p) < τc, while entropy minimization encourages decision boundaries in low-density regions.
  • Label Propagation: Label propagation constructs a similarity-weighted graph over labeled and unlabeled samples to propagate labels.A CNN measures feature similarity, connects each sample to its K nearest neighbors, and alternates model training with graph propagation.

3. Method

SimPLE combines MixMatch-style pseudo-labeling and augmentation consistency with Pair Loss, which propagates information between similar, high-confidence unlabeled samples. Training uses supervised, thresholded unsupervised, and Pair Loss terms over weakly and strongly augmented data.

  • SimPLE targets unlabeled-sample relationships, complementing prior methods that mainly connect labeled to unlabeled data or differently augmented versions of the same sample.
  • Pseudo-labeling: Pseudo labels are formed by averaging predictions from multiple weak augmentations, sharpening the average distribution, and using an exponential moving average model.
  • SimPLE algorithm: The algorithm applies weak augmentation to labeled and unlabeled samples, strong augmentation to unlabeled samples, and uses the resulting labels and pseudo labels in training.
  • Loss: The objective combines supervised cross-entropy, confidence-filtered unsupervised L2 consistency, and Pair Loss.
  • Pair Loss: Pair Loss selects a high-confidence pseudo label as an anchor and aligns strongly augmented samples whose pseudo labels are sufficiently similar to that anchor.
  • Pair Loss: Pair Loss can adaptively extend confidence filtering because a sample below the confidence threshold may be selected through sufficient similarity and pushed toward higher confidence.
  • Pair Loss: The similarity function uses the Bhattacharyya coefficient to measure overlap between probability distributions, while the unsupervised loss retains L2 distance.

4. Experiments

Experiments evaluate SimPLE across standard benchmarks, scalability settings, and transfer learning. SimPLE improves over baselines on CIFAR-100, Mini-ImageNet, and transfer tasks, while matching leading methods on CIFAR-10 and SVHN.

  • Experimental setup: SimPLE uses standard SSL baselines and evaluates CIFAR-10, SVHN, CIFAR-100, Mini-ImageNet, and DomainNet-Real transfer settings.The comparisons include FixMatch, MixMatch, ReMixMatch, VAT, MeanTeacher, and Label Propagation.
  • Standard benchmarks: SimPLE improves over FixMatch by 0.7% on CIFAR-100 while converging in 4.7 hours versus FixMatch’s approximately 8 hours.The comparison uses the same optimizer, hyperparameters, and backbone network as FixMatch.
  • Standard benchmarks: SimPLE is on par with ReMixMatch and FixMatch on CIFAR-10 and SVHN, where the three methods remain within 1% of the fully supervised baseline.Pair Loss contributes less on these datasets because relatively few pseudo-label pairs pass both thresholds.
  • Transfer learning: In transfer learning, SimPLE is 7.57% better than MixMatch and 9.9% better than the supervised baseline for DomainNet-Real-to-Mini-ImageNet adaptation.Pre-trained models converge 5–100 times faster than training from scratch in the reported transfer experiments.
  • Transfer learning: On DomainNet-Real, SimPLE reaches 8% higher accuracy than the supervised baseline, whereas MixMatch is about 7% lower.The reported pattern is consistent across datasets and network architectures, while pre-training does not always improve absolute accuracy because of domain bias.
  • Ablation study: Pair Loss improves performance further with more diverse or numerous augmentations and remains robust to threshold changes.The authors attribute threshold robustness to requiring pairs to pass both thresholds before contributing to the loss.

5. Conclusion

SimPLE is a semi-supervised algorithm that adds Pair Loss to MixMatch-style training. It gains substantially on CIFAR-100 and Mini-ImageNet, matches state-of-the-art methods on CIFAR-10 and SVHN, and performs strongly in transfer learning.

  • Conclusion: SimPLE minimizes statistical distance between similar, high-confidence pseudo labels through a novel Pair Loss objective.The method combines Pair Loss with techniques from the MixMatch family.
  • Conclusion: SimPLE shows significant gains over previous state-of-the-art algorithms on CIFAR-100 and Mini-ImageNet and is on par with them on CIFAR-10 and SVHN.These results are reported across standard semi-supervised learning benchmarks.
  • Conclusion: SimPLE outperforms state-of-the-art methods when models are initialized with weights pre-trained on ImageNet or DomainNet-Real.The conclusion extends the reported benefits to transfer learning settings.

A.1. Hyperparameters

The appendix reports hyperparameter choices designed to remain close to MixMatch and FixMatch settings across backbones and datasets.

  • Hyperparameter choices: SimPLE uses hyperparameters almost identical to MixMatch and FixMatch across CIFAR, Mini-ImageNet, and transfer experiments.The appendix covers WRN 28-8, WRN 28-2, and ResNet-18 configurations.
  • Hyperparameter tables: Tables 7 and 8 provide detailed settings for CIFAR-10, SVHN, CIFAR-100, and Mini-ImageNet experiments.Transfer configurations are listed separately in table 9.

A.2. Optimization

Optimization uses dataset- and backbone-dependent choices, with SGD for several WRN 28-8 experiments and AdamW for WRN 28-2, Mini-ImageNet, and transfer settings.

  • Optimizers: SGD with Nesterov momentum 0.9 and cosine learning-rate decay is used for CIFAR-10, SVHN, and CIFAR-100 with WRN 28-8.The cosine decay rate follows FixMatch settings.
  • Optimizers: AdamW without learning-rate scheduling is used for CIFAR-100 with WRN 28-2, Mini-ImageNet, and transfer experiments.These choices follow the optimization setup used by MixMatch.
  • Augmentations: Augmentations are implemented on GPU with Kornia, while RandAugment follows FixMatch settings.The appendix lists the fixed augmentation transformations and changed parameters.
  • Configuration tables: Tables 8 and 9 document optimization configurations for Mini-ImageNet and the two transfer directions.The configurations distinguish CIFAR-100 and Mini-ImageNet settings from transfer settings.

B.1. Analysis on Confidence Threshold

Theorem 1 gives a lower bound on the maximum coordinate of q when p has sufficient confidence and p, q have sufficient similarity. The proof represents probability vectors on the unit sphere and applies the geodesic triangle inequality.

  • Theorem 1: Theorem 1 lower-bounds max(q) by cos(cos^-1(√τc) + cos^-1(τs))^2 when p satisfies the confidence and similarity thresholds.The condition is ϕτc(max(p)) · ϕτs(fsim(p, q)) > 0.
  • Proof setup: The proof identifies j as the coordinate attaining max(p) and represents its corresponding elementary vector as e_j.The elementary vector has one at coordinate j and zero elsewhere.
  • Proof setup: After square-root mapping, √p, √q, and √e_j lie on the unit n-sphere, where geodesic distance is arccos of the inner product.The unit-sphere representation follows from the vectors’ unit ℓ1 norms.
  • Proof conclusion: Applying the geodesic triangle inequality yields √q_j > cos(cos^-1(√τc) + cos^-1(τs)), which implies the theorem’s squared bound.Here √q_j is the inner product between √q and √e_j.

B.2. More on Pair Loss

The ablation compares SimPLE with and without Pair Loss on CIFAR-100 using confidence, similarity, and false-positive measures. Pair Loss increases qualifying pairs and high-confidence predictions while maintaining nearly zero false positives and improving prediction accuracy.

  • Ablation measures: The ablation measures high-confidence unlabeled samples, unlabeled pairs passing confidence and similarity thresholds, and false-positive unlabeled pairs.It compares SimPLE with Pair Loss enabled against SimPLE without Pair Loss.
  • Pair qualification: 16.67%: the ratio of unlabeled pairs passing both confidence and similarity thresholds increases with Pair Loss, while the false-positive rate remains nearly 0%.The result indicates more consistent and similar predictions for unlabeled samples from the same class.
  • Confidence: 7.5%: the percentage of unlabeled samples with high-confidence labels increases when Pair Loss is enabled.This comparison is shown in the confidence-ratio analysis.
  • Accuracy: 2%: prediction accuracy increases with Pair Loss in the corresponding CIFAR-100 ablation.The accuracy comparison is reported from table 6.
Loading 2103.16725v2…