PaperlayerView exact revision on arXiv

Source-linked AI summary

CoMatch: Semi-supervised Learning with Contrastive Graph Regularization

Junnan Li, Caiming Xiong, Steven Hoi

arXiv:2011.11183v2cs.LGcs.CV

TL;DR

Semi-supervised learning needs to exploit abundant unlabeled data despite weaknesses in pseudo-labeling and task-agnostic representation learning. CoMatch jointly evolves class probabilities and low-dimensional embeddings through smooth pseudo-labeling and graph-based contrastive learning, achieving strong benchmark and downstream results. Its ImageNet comparison reports 66.0% top-1 accuracy with 1% labels, while the authors caution against direct comparison with SimCLRv2.

  • Problem

    Pseudo-labeling suffers from confirmation bias, task-agnostic contrastive learning can be suboptimal for classification, and graph-based methods have lacked competitive large-dataset performance.

  • Method

    CoMatch co-trains class probabilities and normalized low-dimensional embeddings, using memory-smoothed pseudo-labels to train classification and a pseudo-label graph to train embeddings.

  • Results

    CoMatch outperforms baselines across benchmarks and downstream representation-learning tasks; on ImageNet with 1% labels, it achieves 66.0% top-1 accuracy versus 59.9% for the best baseline.

  • Takeaways & Limitations

    CoMatch unifies consistency regularization, entropy minimization, contrastive learning, and graph-based SSL while improving performance especially in label-scarce settings.

  • Takeaways & Limitations

    CoMatch should not be directly compared with SimCLRv2 because its best model uses 33× larger pre-trained teacher models for distillation.

Abstract

from arXiv · show

Semi-supervised learning has been an effective paradigm for leveraging unlabeled data to reduce the reliance on labeled data. We propose CoMatch, a new semi-supervised learning method that unifies dominant approaches and addresses their limitations. CoMatch jointly learns two representations of the training data, their class probabilities and low-dimensional embeddings. The two representations interact with each other to jointly evolve. The embeddings impose a smoothness constraint on the class probabilities to improve the pseudo-labels, whereas the pseudo-labels regularize the structure of the embeddings through graph-based contrastive learning. CoMatch achieves state-of-the-art performance on multiple datasets. It achieves substantial accuracy improvements on the label-scarce CIFAR-10 and STL-10. On ImageNet with 1% labels, CoMatch achieves a top-1 accuracy of 66.0%, outperforming FixMatch by 12.6%. Furthermore, CoMatch achieves better representation learning performance on downstream tasks, outperforming both supervised learning and self-supervised learning. Code and pre-trained models are available at https://github.com/salesforce/CoMatch.

1. Introduction

CoMatch addresses limitations of pseudo-labeling, task-agnostic contrastive learning, and graph-based SSL by jointly evolving class probabilities and embeddings. It achieves strong gains across benchmarks, especially when labels are scarce.

  • Motivation: Pseudo-labeling can accumulate prediction errors through confirmation bias, while task-agnostic contrastive learning may produce representations suboptimal for classification.Graph-based SSL had also not shown competitive performance on larger datasets such as ImageNet.
  • Method: CoMatch jointly learns class probabilities and low-dimensional embeddings in a co-training framework.Embeddings smooth class probabilities to refine pseudo-labels, while pseudo-labels regularize embeddings through graph-based contrastive learning.
  • Results: 6.11% accuracy improvement over FixMatch on CIFAR-10 with 4 labeled samples per class demonstrates a gain in a label-scarce setting.
  • Results: 13.27% accuracy improvement over FixMatch on STL-10 further demonstrates CoMatch’s performance under limited supervision.
  • Results: 66.0% top-1 accuracy on ImageNet with 1% labels exceeds the best baseline’s 59.9% accuracy.With self-supervised pre-training, CoMatch reaches 67.1%.
  • Results: CoMatch outperforms supervised and self-supervised learning on downstream image classification and object detection representation-learning tasks.

2. Background

Semi-supervised learning combines few labeled examples with many unlabeled examples, while existing approaches use either pseudo-labeling or task-agnostic self-supervised representations. These methods rely on classifier predictions or instance discrimination and can mismatch classification objectives.

  • Self-training: Task-specific self-training uses predicted class probabilities as pseudo-labels for unlabeled samples.
  • Self-supervised learning: Task-agnostic self-supervised learning projects samples into low-dimensional embeddings and uses contrastive learning to discriminate different images.
  • Consistency regularization: Consistency regularization assumes that augmenting an unlabeled image should preserve the classifier’s class probability.Mean Teacher uses an EMA model, while other methods vary the augmentation or prediction-processing strategy.
  • Contrastive learning: Contrastive learning treats differently augmented views of an image as similar while separating different images, using normalized low-dimensional embeddings.SimCLR uses batch images, whereas MoCo maintains an EMA-model embedding queue.
  • Graph-based SSL: Task-agnostic contrastive objectives can conflict with classification because they enforce different representations for images from the same class.Graph-based SSL instead encourages predictions to be smooth over a graph of similar samples.

3. Method

CoMatch jointly learns class probabilities and low-dimensional embeddings, using each representation to refine the other through memory-smoothed pseudo-labeling and graph-based contrastive learning.

  • 3. Method: CoMatch jointly optimizes supervised classification, unsupervised classification, and graph-based contrastive losses while learning the encoder, classification head, and projection head.The unsupervised terms operate on unlabeled data, with scalar weights controlling their contribution.
  • 3. Method: Memory-smoothed pseudo-labels aggregate class probabilities from neighboring memory-bank samples, balancing embedding-based smoothness against preserving the original prediction.The memory bank stores probabilities and embeddings from past weakly augmented labeled and unlabeled samples using FIFO updates.
  • 3. Method: The pseudo-label graph connects samples whose pseudo-label similarity exceeds T and gives each sample a self-loop, then supervises an embedding graph built from strongly augmented images.Both graphs are row-normalized before minimizing their cross-entropy.
  • 3. Method: Self-loops provide same-image augmentation consistency, while connections between similarly pseudo-labeled samples encourage similar embeddings and classwise clustering.The resulting contrastive objective combines self-supervised consistency with pseudo-label-guided clustering and entropy minimization.
  • 3. Method: CoMatch forms a training curriculum in which initially sparse pseudo-label graphs become denser as samples cluster and pseudo-label confidence increases.For large datasets, an EMA model and momentum queue expand graph construction beyond the current batch with limited additional memory and computation.
  • 3. Method: In open-set semi-supervised learning, low-confidence out-of-distribution samples are weakly connected to in-distribution samples and are pushed farther away by contrastive learning.The method's large-scale variant compares current-batch samples with momentum-queue samples using pseudo-label and embedding similarities.

4. Experiment

CoMatch achieves strong semi-supervised performance across CIFAR-10, STL-10, and ImageNet, including substantial gains in label-scarce settings. Ablations and transfer evaluations support the roles of graph regularization, pseudo-label smoothing, and learned representations.

  • CIFAR-10 and STL-10: 93.09% average accuracy on CIFAR-10 with 4 labels per class, versus 86.98% for FixMatch (w. DA).
  • CIFAR-10 and STL-10: 13.27% improvement over FixMatch (w. DA) on STL-10.
  • ImageNet: 66.0% top-1 accuracy on ImageNet with 1% labels, improving 6.1% over the best baseline with 3× less training time.
  • ImageNet: MoCov2 pre-training raises CoMatch performance to 67.1% with 1% labels and 73.7% with 10% labels, while accelerating convergence.
  • Ablation Study: Removing graph-based contrastive learning decreases performance by 2.8%, while α = 1 reduces pseudo-labeling to Mean Teacher and decreases accuracy by 2.1%.
  • Ablation Study: Performance increases as K grows from 10k to 30k, then plateaus; the memory bank and momentum queue add only small computation overhead.

5. Conclusion

CoMatch’s success is attributed to jointly training class probabilities and embeddings, refining pseudo-labels with memory smoothing, and structuring representations through graph-based contrastive learning.

  • CoMatch co-trains class probabilities and image embeddings as interacting representations.
  • Memory-smoothed pseudo-labeling mitigates confirmation bias.
  • Graph-based contrastive learning improves learned representations.

Appendix A. Experiment Details

The appendix documents the hyperparameter configuration for CoMatch’s semi-supervised experiments and specifies strong augmentation details for ImageNet unlabeled data.

  • Table 5 provides the complete hyperparameter set for CoMatch’s semi-supervised learning experiments.
  • ImageNet strong augmentation adds color distortion to the standard crop-and-flip augmentation.

Appendix B. MB and MQ in CoMatch

CoMatch uses an EMA model to maintain a memory bank and momentum queue containing representations and pseudo-label information for labeled and unlabeled samples.

  • The memory bank stores class probabilities and low-dimensional embeddings for weakly augmented labeled and unlabeled samples.
  • The momentum queue stores unlabeled pseudo-labels and strongly augmented embeddings.
  • Figure 5 denotes U and X as unlabeled and labeled batches, while ¯f, ¯h, and ¯g are EMA encoder, classifier, and projection heads.

Appendix C. Pseudo-code of CoMatch

Algorithm 1 summarizes one CoMatch iteration, taking labeled and unlabeled batches alongside model components and memory structures, then updating the network with SGD to minimize the training loss.

  • One iteration takes labeled and unlabeled batches, an encoder, classifier, projection head, and memory bank as inputs.
  • The procedure represents weakly augmented samples and strongly augmented embeddings in its stored training state.
  • The embedding graph assigns positive links when similarity exceeds threshold T, with self-links treated separately.
  • The contrastive formulation uses temperature-scaled exponentiated embedding similarities.
  • The encoder, classifier, and projection head are updated with SGD to minimize L.
Loading 2011.11183v2…