PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey on Semi-, Self- and Unsupervised Learning for Image Classification

Lars Schmarje, Monty Santarossa, Simon-Martin Schröder, Reinhard Koch

arXiv:2002.08721v5cs.CVcs.LG

TL;DR

Deep learning image classification relies heavily on labeled data, which is often impractical to create at scale. This survey compares 34 reduced-label methods by performance and shared ideas, finding closing performance gaps and distinct idea clusters whose combination can improve performance. It also identifies unresolved real-world limitations and directions for future research.

  • Problem

    Deep learning quality depends strongly on labeled images, but many real-world applications cannot create datasets with millions of labeled examples.

  • Method

    The survey compares 34 semi-, self-, and unsupervised image-classification methods using their performance and shared common ideas rather than a fine-grained taxonomy.

  • Results

    Over 90% Top-5 accuracy on ILSVRC-2012 with only 10% of the labels indicates that semi-supervised methods could be applied to real-world problems.

  • Takeaways & Limitations

    The performance gap is closing, and combining techniques from clusters with few overlapping ideas is identified as beneficial to overall performance.

  • Takeaways & Limitations

    Class imbalance, noisy or fuzzy labels, robustness, and datasets with few unlabeled points are not considered by the presented methods.

Abstract

from arXiv · show

While deep learning strategies achieve outstanding results in computer vision tasks, one issue remains: The current strategies rely heavily on a huge amount of labeled data. In many real-world problems, it is not feasible to create such an amount of labeled training data. Therefore, it is common to incorporate unlabeled data into the training process to reach equal results with fewer labels. Due to a lot of concurrent research, it is difficult to keep track of recent developments. In this survey, we provide an overview of often used ideas and methods in image classification with fewer labels. We compare 34 methods in detail based on their performance and their commonly used ideas rather than a fine-grained taxonomy. In our analysis, we identify three major trends that lead to future research opportunities. 1. State-of-the-art methods are scaleable to real-world applications in theory but issues like class imbalance, robustness, or fuzzy labels are not considered. 2. The degree of supervision which is needed to achieve comparable results to the usage of all labels is decreasing and therefore methods need to be extended to settings with a variable number of classes. 3. All methods share some common ideas but we identify clusters of methods that do not share many ideas. We show that combining ideas from different clusters can lead to better performance.

1. Introduction

Deep learning performs strongly but depends heavily on labeled data, which is often impractical to obtain at scale. This survey reviews reduced-label learning research for image classification, comparing methods, shared ideas, and performance to identify research opportunities.

  • Motivation: Deep learning performance is strongly influenced by the number of labeled images, while real-world applications often cannot create datasets with millions of labeled examples.Transfer learning can help on small specialized datasets, but the fundamental dependence on labeled data remains.
  • Motivation: Unlabeled data has motivated research across unsupervised, semi-supervised, self-supervised, weakly supervised, and metric learning.The survey focuses on how these approaches use unlabeled data for image classification.
  • Scope and contribution: The survey compares similarities and differences among methods while focusing on image classification rather than a broad range of computer vision tasks.Its intended audience is researchers or readers with comparable background who want to track recent developments.
  • Survey organization: The survey reviews methods, organizes their training strategies and common ideas, and compares performance across four common image classification datasets.Its structure moves from definitions and method descriptions to comparisons and discussion of resulting trends.
  • Scope and contribution: The survey excludes several related areas, including metric learning, meta learning, generative adversarial networks, pose estimation, segmentation, videos, sketches, few-shot learning, and zero-shot learning.These exclusions maintain a focus on deep learning approaches for general image classification.

2. Underlying Concepts

The survey distinguishes reduced-supervision methods by how labeled and unlabeled data are used across training stages. It uses this stage-based organization to handle overlapping terminology and compare supervised, semi-supervised, and unsupervised approaches.

  • Definitions: Training strategy denotes the general approach for using unsupervised data during training, while common ideas are concepts or approaches shared across methods.The survey groups methods roughly by training strategy but compares them in detail through their common ideas.
  • Definitions: The survey defines labeled and unlabeled image sets as X_l and X_u, with labels used to distinguish them according to their training-time availability.An image is treated as unlabeled for a training process when its label information is not used.
  • Training strategies: Because semi-supervised, self-supervised, and unsupervised learning have overlapping definitions, the survey classifies methods by training stages and data usage.This taxonomy separates methods using supervised, unsupervised, or semi-supervised learning in different phases.
  • Training strategies: The survey identifies One-Stage-Semi-Supervised, One-Stage-Unsupervised, and Multi-Stage-Semi-Supervised training strategies.One-stage semi-supervised methods use labeled and unlabeled data together; one-stage unsupervised methods use only unlabeled samples; multi-stage methods use unlabeled data first and labeled data later.
  • Training strategies: The strategy using only labeled data in a first stage followed by unlabeled data is excluded because the authors regard it as unusual and lack known methods fitting it.Supervised weight initialization on another dataset is treated as an architectural decision rather than part of reduced-supervision training.

2.2. Common ideas

The survey abstracts recurring method components into common ideas rather than requiring identical implementations. This abstraction enables comparisons across methods despite implementation differences.

  • Common ideas: Common ideas are concepts or approaches used across multiple semi-, self-, and unsupervised methods.The survey distinguishes loss functions from general concepts and orders the ideas alphabetically.
  • Common ideas: A common idea captures an underlying motivation rather than an identical implementation or approximation.This deliberately fuzzy definition makes comparison possible despite many small implementation differences.
  • Common ideas: Abstracting core elements allows the survey to detect similarities among methods while omitting some implementation details, concepts, and motivations.The survey treats this incompleteness as inherent to its abstraction of methods.

Loss Functions

The survey groups several loss-function ideas used in reduced-supervision image classification, including cross-entropy variants, contrastive objectives, and mutual-information-based reasoning.

  • Loss Functions: Cross-entropy measures the difference between network outputs and target labels, while CE* denotes targets such as pseudo-labels or self-supervised outputs.Ground-truth labels are one-hot encoded, making the label entropy generally zero.
  • Loss Functions: Contrastive loss distinguishes positive from negative pairs, often using different views of the same image as positives and other batch combinations as negatives.NT-Xent and InfoNCE are examples based on cross-entropy.
  • Loss Functions: NT-Xent measures pairwise output similarity with a normalized dot product and temperature parameter across a batch of image pairs.The loss is computed across positive pairs in a fixed subset containing N elements.
  • Loss Functions: InfoNCE is treated as combining contrastive loss with mutual information, although evidence suggests mutual-information lower bounds may not explain its success.The survey explicitly counts InfoNCE as a mixture of these common ideas.

Entropy Minimization (EM)

Entropy minimization sharpens semi-supervised predictions, while KL and MSE provide auxiliary ways to compare distributions or outputs; MSE alone can degenerate the network.

  • Entropy Minimization (EM): Entropy minimization encourages sharper, more confident predictions by minimizing the entropy of the network’s output distribution.The approach addresses predictions distributed across many or all classes rather than concentrated on a few classes.
  • Entropy Minimization (EM): KL divergence measures the difference between two distributions and can define an auxiliary loss between network outputs and a secondary class distribution.The secondary distribution may come from another network output, a prior, or another target.
  • Entropy Minimization (EM): MSE measures Euclidean distance between vectors such as two network outputs and pulls those vectors together during minimization.Unlike CE or KL, MSE is not restricted to probability measures and can operate in an arbitrary Euclidean feature space.
  • Entropy Minimization (EM): Using MSE alone on network outputs can degenerate the network, similarly to using entropy minimization alone.The supplied discussion identifies degeneration as a consequence of relying on these losses in isolation.

Mutual Information (MI)

Mutual-information methods relate distributions from images or network outputs, often approximating unknown joint distributions; the survey also describes adversarial invariance as a related consistency idea.

  • Mutual Information (MI): Mutual information is defined as the KL divergence between a joint distribution and its marginal distributions.Reduced-supervision methods often maximize mutual information between input, output, or intermediate network distributions.
  • Mutual Information (MI): For augmented views, mutual-information estimation must be approximated because the conditional or joint distribution is generally unknown.The approximation depends on the method and may use network outputs for different views of the same image.
  • Mutual Information (MI): The survey presents mutual information as a difference between output entropy and conditional entropy, with this balance helping avoid degeneration.Ji et al. argue that this representation illustrates benefits over cross-entropy in unsupervised settings.
  • Mutual Information (MI): VAT minimizes output distance between an image and an adversarially transformed version, selecting the transformation that maximizes that distance within the allowed set.The distance uses a non-negative function such as cross-entropy or KL divergence, with r bounded by ϵ.

Concepts

The survey highlights Mixup, overclustering, pretext tasks, and pseudo-labeling as recurring concepts for exploiting structure beyond ordinary ground-truth supervision.

  • Concepts: Mixup blends images and their labels using the same convex-combination parameterization, encouraging consistent predictions for intermediate interpolations.The approach is used in supervised and several semi-supervised algorithms.
  • Concepts: Overclustering uses more clusters than classes, allowing networks to split noisy, fuzzy, or intermediate data into multiple subregions.The additional separation can be useful when data are assigned to adjacent classes randomly.
  • Concepts: Pretext tasks train networks on a different task, such as rotation prediction, jigsaw solving, contrastive learning, or mutual-information maximization.The learned representations can then be fine-tuned for image classification.
  • Concepts: Pseudo-labeling classifies unseen data with a neural network and reuses its predictions as labels for further training.Additional information is needed to prevent completely random predictions.

3. Methods

The survey organizes semi-supervised methods around shared supervised and unsupervised objectives, augmentation, consistency regularization, pseudo-labeling, and multi-view or multi-stage training. Across methods, differences mainly concern how unlabeled predictions, transformations, optimization, and auxiliary tasks are designed.

  • Overview: The survey summarizes methods chronologically by training strategy and records their shared ideas, usage, and special cases.The authors note that the method list includes many recent approaches but is not claimed to be complete.
  • Pseudo-labeling and consistency: Pseudo-Labels uses network predictions as labels for unknown data while minimizing labeled and unlabeled objectives in parallel.
  • Pseudo-labeling and consistency: Consistency methods combine supervised cross-entropy with an unsupervised consistency loss based on augmented inputs, prior predictions, or an exponential moving average.The π-model uses another augmentation, Temporal Ensembling averages previous predictions, and Mean Teacher averages student weights at each update.
  • Pseudo-labeling and consistency: VAT combines adversarial consistency on unlabeled data with cross-entropy on labeled data, and adding entropy minimization further increased accuracy.
  • Pseudo-labeling and consistency: ICT, MixMatch, and related methods combine supervised cross-entropy with unsupervised losses using interpolated examples, sharpened predictions, or pseudo-labels.ICT interpolates unlabeled points and their pseudo-labels, while MixMatch sharpens predictions from augmented images.
  • Augmentation and optimization: Other approaches modify optimization or training structure, including fast-SWA’s averaging and cycling learning rates, SpamCo’s multi-view co-training, and ReMixMatch’s distribution alignment and augmentation anchoring.ReMixMatch is presented as an extension of MixMatch with the two additional mechanisms.
  • Augmentation and optimization: UDA emphasizes strong augmentation, combining AutoAugment and Cutout with cross-entropy, KL divergence, entropy minimization, training signal annealing, and pseudo-label-based filtering.The method compares predictions for an image and an augmented version and uses regularization to limit labeled-example influence and filter outliers.

Self-Supervised Semi-Supervised Learning (S4L)

Self-supervised and unsupervised methods add auxiliary prediction, contrastive, clustering, or mutual-information objectives to reduce reliance on labels. The survey highlights multi-stage pipelines, direct unsupervised classification, and adaptations for fuzzy labels and unknown cluster counts.

  • S4L and contrastive learning: S4L combines supervised cross-entropy with self-supervised rotation and exemplar-prediction objectives.Its Mix Of All Models framework combines self-supervised and semi-supervised approaches.
  • S4L and contrastive learning: SimCLR maximizes agreement between two augmentations of the same image using normalized temperature-scaled cross-entropy based on positive and negative pair similarity.
  • Clustering and fuzzy labels: FOC extends IIC with overclustering and Inverse Cross-Entropy to subdivide fuzzy labels, achieving 5-10% more consistent predictions than FixMatch on a real-world plankton dataset.FOC produces classifications during its unsupervised stage and can therefore also be viewed as one-stage unsupervised learning.
  • S4L and contrastive learning: Momentum-encoder contrastive learning stores negative examples in a queue updated by an exponential moving average, reducing reliance on large minibatches.
  • S4L and contrastive learning: An online-target-network method trains the online network to predict target representations with MSE and uses slow weight averaging to avoid degeneration.
  • Multi-stage methods: SimCLRv2 uses larger and deeper networks, a contrastive pretext task, fine-tuning with few labels, and self-training or distillation with soft pseudo-labels.The survey reports that self-distillation can improve performance even when performed on the same network.
  • Clustering and fuzzy labels: DAC reformulates unsupervised clustering as pairwise classification by using cluster-prediction distances to identify similar and dissimilar images with a certainty threshold.The threshold is lowered during training to include more images, and RGB features are combined with extracted HOG features.
  • Clustering and fuzzy labels: IMSAT maximizes mutual information between inputs and outputs, using cross-entropy consistency and VAT-based augmentation to address dominant color information.

4. Analysis

The survey compares methods across four image-classification datasets, examining shared ideas, training strategies, and reported performance. The analysis finds distinct clusters of common ideas and substantial variation across datasets and supervision settings.

  • Evaluation setup: The comparison uses four datasets selected because multiple papers reported results on them, supporting a common evaluation basis.The survey compares methods using classification accuracy, or cluster accuracy for unsupervised learning.
  • Common ideas: Common ideas tend to differ between training strategies rather than being used equally across methods.The survey evaluates frequently recurring ideas and excludes special implementation details.
  • Common ideas: One-stage semi-supervised methods center on cross-entropy, while multi-stage methods combine pretext tasks with cross-entropy fine-tuning.The survey reports distinct use of CE and pretext tasks across these strategies.
  • Common ideas: Further clusters associate EM, KL, MSE, and MU with one-stage semi-supervised methods, and CL, MI, and OC with multi-stage methods.The survey hypothesizes that these differences relate to how unlabeled data are used during training.
  • Performance comparison: On CIFAR-10, MixMatch and FixMatch exceed 95% accuracy and remain roughly three percentage points below the fully supervised baseline.On CIFAR-100, FixMatch reaches about 77% versus about 80% for the fully supervised baseline.
  • Performance comparison: On STL-10, most methods outperform the supervised baseline, while SimCLRv2 reaches 80.9% Top-1 and around 96% Top-5 accuracy on ILSVRC-2012.The survey also reports that unsupervised methods remain up to 10% below the supervised baseline but benefit from self-supervised ideas.

1. Trend: Real World Applications?

The strongest reported methods appear potentially applicable to real-world classification, but the survey’s conclusion is limited to balanced, clearly separated classes. Class imbalance, noisy labels, scarce unlabeled data, and robustness issues remain unaddressed.

  • Scalability: Only five methods exceed 90% Top-5 accuracy on high-resolution ILSVRC-2012, unlike the many methods exceeding 90% on CIFAR-10.The survey concludes that most methods are not yet scalable to high-resolution and complex image-classification problems.
  • Scalability: FixMatch and SimCLRv2 are identified as potentially applicable to real-world classification tasks beyond purely scientific use.This conclusion is based on their reported performance and does not extend to all methods.
  • Limitations: The conclusion applies to balanced, clearly separated classes; class imbalance, noisy labels, scarce unlabeled data, and robustness issues remain untreated.The survey notes that strong performance on well-structured datasets does not always transfer completely to real-world datasets.

2. Trend: How much supervision is needed?

The supervision needed for comparable performance is decreasing: reduced-supervision methods can approach or exceed fully supervised results with far fewer labels, but future use depends on broader class settings and real-world scaling.

  • Less than 5% separates reduced-supervised and fully supervised results on CIFAR-10, CIFAR-100, and ILSVRC-2012.
  • About 20% better performance is reported for reduced supervision on STL-10 because of its additional unlabeled data.
  • Comparable results are generally achieved with roughly 10% of the labels, while FixMatch and SimCLRv2 can perform comparably with 1%.
  • Semi-supervised learning is nearing settings with very few labels per class, such as 10 labels for CIFAR-10.
  • Scaling to large datasets remains an open question because the discussed methods must work with few labels per class.
  • Semi-supervised ideas must be extended to datasets with many, unknown, or increasing numbers of classes, where unsupervised methods retain importance.

3. Trend: Combination of common ideas

One-stage and multi-stage semi-supervised methods share few common ideas because they pursue different aims, but combining diverse and unusual ideas can improve performance.

  • One-stage and multi-stage semi-supervised methods share few common ideas.
  • Multi-stage methods emphasize strong representations, whereas one-stage methods target high accuracy with as few labels as possible.
  • SimCLRv2, EnAET, ReMixMatch, and S4L combine broad and uncommon ideas for their training strategies.
  • The survey identifies meaningful combinations of diverse ideas as a promising research direction because many reasonable combinations remain unexplored.

5. Conclusion

The survey analyzes 34 methods and identifies trends in supervision, robustness, and technique combinations. Its conclusions support strong reduced-label performance while defining boundaries for real-world deployment and future research.

  • The survey analyzes 34 semi-, self-, and unsupervised methods through their differences, similarities, and combinations.
  • The analysis uses training-strategy definitions and common ideas, while performance comparisons remain difficult because architectures and implementations differ.
  • Over 90% Top-5 accuracy on ILSVRC-2012 with 10% of the labels indicates potential applicability of semi-supervised methods to real-world problems.
  • Class imbalance and noisy or fuzzy labels are not considered, so more robust methods are needed before real-world application.
  • One-stage and multi-stage training use different sets of common ideas, and combining techniques is identified as beneficial to overall performance.
Loading 2002.08721v5…