CoMatch: Semi-supervised Learning with Contrastive Graph Regularization

Motivation

• Pseudo-labeling can accumulate prediction errors through confirmation bias, while task-agnostic contrastive learning may produce representations suboptimal for classification.Graph-based SSL had also not shown competitive performance on larger datasets such as ImageNet.Source: §1 · p. 1

Method

• CoMatch jointly learns class probabilities and low-dimensional embeddings in a co-training framework.Embeddings smooth class probabilities to refine pseudo-labels, while pseudo-labels regularize embeddings through graph-based contrastive learning.Source: §1 · p. 1

Results

• 6.11% accuracy improvement over FixMatch on CIFAR-10 with 4 labeled samples per class demonstrates a gain in a label-scarce setting.Source: §1 · p. 1
• 13.27% accuracy improvement over FixMatch on STL-10 further demonstrates CoMatch’s performance under limited supervision.Source: §1 · p. 1
• 66.0% top-1 accuracy on ImageNet with 1% labels exceeds the best baseline’s 59.9% accuracy.With self-supervised pre-training, CoMatch reaches 67.1%.Source: §1 · p. 1
• CoMatch outperforms supervised and self-supervised learning on downstream image classification and object detection representation-learning tasks.Source: §1 · p. 1

Self-training

• Task-specific self-training uses predicted class probabilities as pseudo-labels for unlabeled samples.Source: Figure 1 · p. 2

Self-supervised learning

• Task-agnostic self-supervised learning projects samples into low-dimensional embeddings and uses contrastive learning to discriminate different images.Source: Figure 1 · p. 2, §2 · p. 2

Consistency regularization

• Consistency regularization assumes that augmenting an unlabeled image should preserve the classifier’s class probability.Mean Teacher uses an EMA model, while other methods vary the augmentation or prediction-processing strategy.Source: §2 · p. 2

Contrastive learning

• Contrastive learning treats differently augmented views of an image as similar while separating different images, using normalized low-dimensional embeddings.SimCLR uses batch images, whereas MoCo maintains an EMA-model embedding queue.Source: §2 · p. 2

Graph-based SSL

• Task-agnostic contrastive objectives can conflict with classification because they enforce different representations for images from the same class.Graph-based SSL instead encourages predictions to be smooth over a graph of similar samples.Source: §2 · pp. 2–3

3. Method

• CoMatch jointly optimizes supervised classification, unsupervised classification, and graph-based contrastive losses while learning the encoder, classification head, and projection head.The unsupervised terms operate on unlabeled data, with scalar weights controlling their contribution.Source: §3.1 · p. 3
• Memory-smoothed pseudo-labels aggregate class probabilities from neighboring memory-bank samples, balancing embedding-based smoothness against preserving the original prediction.The memory bank stores probabilities and embeddings from past weakly augmented labeled and unlabeled samples using FIFO updates.Source: §3.2 · p. 3
• The pseudo-label graph connects samples whose pseudo-label similarity exceeds T and gives each sample a self-loop, then supervises an embedding graph built from strongly augmented images.Both graphs are row-normalized before minimizing their cross-entropy.Source: §3.2 · p. 4
• Self-loops provide same-image augmentation consistency, while connections between similarly pseudo-labeled samples encourage similar embeddings and classwise clustering.The resulting contrastive objective combines self-supervised consistency with pseudo-label-guided clustering and entropy minimization.Source: §3.2 · p. 4
• CoMatch forms a training curriculum in which initially sparse pseudo-label graphs become denser as samples cluster and pseudo-label confidence increases.For large datasets, an EMA model and momentum queue expand graph construction beyond the current batch with limited additional memory and computation.Source: §3.2 · pp. 4–5, §3.3 · p. 5
• In open-set semi-supervised learning, low-confidence out-of-distribution samples are weakly connected to in-distribution samples and are pushed farther away by contrastive learning.The method's large-scale variant compares current-batch samples with momentum-queue samples using pseudo-label and embedding similarities.Source: §3.2 · p. 5, §3.3 · p. 5

CIFAR-10 and STL-10

• 93.09% average accuracy on CIFAR-10 with 4 labels per class, versus 86.98% for FixMatch (w. DA).Source: §4.1 · p. 5
• 13.27% improvement over FixMatch (w. DA) on STL-10.Source: §4.1 · p. 6

ImageNet

• 66.0% top-1 accuracy on ImageNet with 1% labels, improving 6.1% over the best baseline with 3× less training time.Source: §4.2 · pp. 6–7
• MoCov2 pre-training raises CoMatch performance to 67.1% with 1% labels and 73.7% with 10% labels, while accelerating convergence.Source: §4.2 · p. 7

Ablation Study

• Removing graph-based contrastive learning decreases performance by 2.8%, while α = 1 reduces pseudo-labeling to Mean Teacher and decreases accuracy by 2.1%.Source: §4.3 · p. 7
• Performance increases as K grows from 10k to 30k, then plateaus; the memory bank and momentum queue add only small computation overhead.Source: §4.3 · p. 8

• CoMatch co-trains class probabilities and image embeddings as interacting representations.Source: §5 · p. 8
• Memory-smoothed pseudo-labeling mitigates confirmation bias.Source: §5 · p. 8
• Graph-based contrastive learning improves learned representations.Source: §5 · p. 8

• Table 5 provides the complete hyperparameter set for CoMatch’s semi-supervised learning experiments.Source: Appendix A · p. 11, Table 5 · p. 11
• ImageNet strong augmentation adds color distortion to the standard crop-and-flip augmentation.Source: Appendix A · p. 11

• The memory bank stores class probabilities and low-dimensional embeddings for weakly augmented labeled and unlabeled samples.Source: Appendix B · p. 11
• The momentum queue stores unlabeled pseudo-labels and strongly augmented embeddings.Source: Appendix B · p. 11
• Figure 5 denotes U and X as unlabeled and labeled batches, while ¯f, ¯h, and ¯g are EMA encoder, classifier, and projection heads.Source: Figure 5 · p. 11

• One iteration takes labeled and unlabeled batches, an encoder, classifier, projection head, and memory bank as inputs.Source: Appendix C · p. 11
• The procedure represents weakly augmented samples and strongly augmented embeddings in its stored training state.Source: Appendix C · p. 11
• The embedding graph assigns positive links when similarity exceeds threshold T, with self-links treated separately.Source: Appendix C · p. 11
• The contrastive formulation uses temperature-scaled exponentiated embedding similarities.Source: Appendix C · p. 11
• The encoder, classifier, and projection head are updated with SGD to minimize L.Source: Appendix C · p. 11