PaperlayerView exact revision on arXiv

Source-linked AI summary

Prototypical Contrastive Learning of Unsupervised Representations

Junnan Li, Pan Zhou, Caiming Xiong, Steven C. H. Hoi

arXiv:2005.04966v5cs.CVcs.LG

TL;DR

Instance-wise contrastive learning can fail to encode semantic structure because semantically similar samples may be treated as negatives. PCL introduces prototypes within an EM framework and optimizes ProtoNCE, outperforming instance-wise methods across multiple benchmarks, especially in low-resource transfer settings.

  • Problem

    Instance-wise contrastive learning can push semantically similar negative pairs apart and may learn embeddings driven by low-level image differences rather than high-level semantics.

  • Method

    PCL introduces prototypes as latent variables in an EM framework, using clustering in the E-step and ProtoNCE-based contrastive optimization in the M-step.

  • Results

    PCL outperforms instance-wise contrastive learning on multiple benchmarks, with substantial improvements in low-resource transfer learning.

  • Takeaways & Limitations

    PCL learns representations that encode semantic structure by bridging contrastive learning and clustering.

  • Takeaways & Limitations

    The EM formulation assumes a uniform prior over cluster centroids and an isotropic Gaussian distribution around each prototype.

Abstract

from arXiv · show

This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL.

1 INTRODUCTION

Instance-wise contrastive learning can push semantically similar samples apart because it treats different instances as negatives. PCL addresses this by clustering embeddings into prototypes and optimizing a ProtoNCE-based EM framework to encode semantic structure.

  • Instance-wise contrastive learning may push semantically similar negative pairs apart, weakening semantic structure in the embedding space.Thousands of negative samples amplify this class-collision problem.
  • PCL assigns each instance multiple prototypes at different granularities to encode semantic structure in the learned embedding space.A prototype is a representative embedding for a group of semantically similar instances.
  • ProtoNCE extends contrastive learning by encouraging sample embeddings to be more similar to assigned prototypes than to other prototypes.The framework bridges clustering and contrastive learning while retaining an instance-based contrastive component.
  • PCL formulates representation learning as EM, estimating prototype distributions through k-means in the E-step and updating network parameters with ProtoNCE in the M-step.ProtoNCE minimization is equivalent to maximizing estimated log-likelihood under an isotropic Gaussian assumption.

2 RELATED WORK

Instance-wise contrastive learning can rely on low-level differences and push semantically similar negatives apart, while clustering methods have lacked demonstrated transferability at large scale. PCL is positioned as a theoretically grounded framework combining representation learning with clustering.

  • Instance discrimination can exploit low-level image differences, so learned embeddings do not necessarily capture high-level semantics.Instance classification can rapidly exceed 90% within 10 epochs, providing limited further informative signals.
  • Large negative sets create semantically similar negative pairs that contrastive loss pushes apart, preserving local smoothness while largely ignoring global semantic structure.This problem is identified as class collision and is reported to hurt representation learning.
  • Existing deep clustering methods jointly learn embeddings and assignments but have not shown the ability to learn transferable representations from large-scale image collections.
  • PCL is presented as a more general and theoretically justified framework than heuristic self-supervised pretext-task designs.

3 PROTOTYPICAL CONTRASTIVE LEARNING

PCL bridges contrastive learning and clustering by treating prototypes as latent variables in an EM framework and optimizing ProtoNCE. Prototype assignments, multiple granularities, and concentration estimates encourage embeddings to capture semantic structure while retaining local smoothness.

  • Framework: PCL replaces instance-based reference embeddings with prototypes and iteratively performs clustering and representation learning at each epoch.Prototypes act as representative embeddings for groups of semantically similar instances; the training framework uses several cluster granularities.
  • ProtoNCE: ProtoNCE generalizes InfoNCE by using prototypes as contrasts and replacing fixed temperature τ with per-prototype concentration φ.Multiple clusterings provide hierarchical structure, while an added InfoNCE term retains local smoothness and helps bootstrap clustering.
  • EM formulation: The EM formulation estimates prototype assignments through k-means in the E-step and updates network parameters by minimizing ProtoNCE in the M-step.Under an isotropic-Gaussian assumption, minimizing ProtoNCE is equivalent to maximizing the estimated log-likelihood.
  • Concentration estimation: The concentration estimate scales prototype similarities so loose clusters are pulled more strongly toward their prototypes and tight clusters are encouraged less.The method reports more balanced clusters with similar concentration and avoids collapse of most embeddings into one cluster.
  • Mutual information analysis: ProtoNCE maximizes mutual information between embeddings and prototypes, whose clustering-based representations have larger mutual information with ImageNet class labels than instance features.Compared with InfoNCE, ProtoNCE increases prototype-label mutual information as training proceeds, indicating more semantically meaningful prototypes.
  • Classifier interpretation: PCL can also be interpreted as optimizing cluster-assignment probabilities with prototypes serving as weights for a non-parametric linear classifier.With k-means, the classifier weights are the mean representation vectors of the clusters.

4 EXPERIMENTS

PCL is evaluated across transfer, classification, clustering, and detection benchmarks, generally outperforming prior methods and improving low-resource transfer performance.

  • Evaluation setup: PCL is evaluated on transfer learning tasks using settings matched to MoCo, enabling direct comparison of the prototypical contrastive loss.Experiments cover low-shot and semi-supervised classification, linear and kNN classifiers, clustering, and object detection.
  • Transfer learning: PCL substantially outperforms MoCo and SimCLR in low-shot classification on Places205 and PASCAL VOC2007 using fixed representations.Linear SVMs are trained with varying numbers of samples per class, averaging results across five runs.
  • Transfer learning: PCL sets a new state-of-the-art in semi-supervised ImageNet classification with 1% or 10% labeled data under 200 training epochs.It outperforms self-supervised and semi-supervised methods, with standard deviation below 0.6 for the 1% setting.
  • Classification benchmarks: PCL outperforms MoCo with linear classifiers and substantially outperforms previous methods with kNN classification on ImageNet.Linear evaluation covers ImageNet, VOC07, and Places205; kNN evaluation uses ResNet-50 features.
  • Clustering evaluation: PCL achieves substantially higher adjusted mutual information in ImageNet k-means clustering than representations learned by other methods.The reported comparison uses AMI for clustering performance.
  • Object detection: PCL substantially closes the gap between self-supervised and supervised training for frozen-backbone VOC object detection, and outperforms both on COCO fine-tuning experiments.Frozen-backbone evaluation uses Faster R-CNN; the COCO result is reported in the appendix.

5 VISUALIZATION OF LEARNED REPRESENTATION

t-SNE visualization compares MoCo and PCL representations for ImageNet images from the first 40 classes, showing more separated PCL clusters.

  • Observed structure: Compared with MoCo, PCL forms more separated clusters, suggesting a representation with lower entropy.The comparison is based on the t-SNE visualization of the learned representations.
  • Visualization setup: The visualization compares MoCo and PCL embeddings for ImageNet training images from the first 40 classes, with colors denoting classes.MoCo appears on the left and PCL on the right.

6 CONCLUSION

The paper concludes that PCL combines prototypes, clustering, and contrastive learning in an EM-based framework to encode semantic structure and perform strongly across benchmarks.

  • Conclusion: PCL is a generic unsupervised framework that maximizes observed-data log-likelihood through iterative clustering and representation learning.Prototypes serve as latent variables, and ProtoNCE trains the embedding space.
  • Conclusion: Extensive experiments across multiple benchmarks demonstrate PCL’s advantage for unsupervised representation learning.The conclusion summarizes the method’s broad empirical performance without specifying individual benchmark values.

APPENDIX A ABLATION ON PROTONCE

The ablation shows that the prototypical contrastive term is especially important for low-resource fine-tuning, while warm-up improves clustering initialization. The appendix also details PCL’s iterative clustering and representation-learning procedure.

  • Ablation findings: The prototypical contrastive term plays an important role, especially in low-resource fine-tuning.The ablation evaluates low-resource fine-tuning and linear classification on ImageNet.
  • Ablation findings: Warm-up improves results by bootstrapping clustering with better representations.
  • Algorithm: PCL alternates an E-step that obtains momentum features and clusters them into prototypes with an M-step that processes minibatches through the encoders.The algorithm specifies k-means clustering during the E-step and minibatch forward passes during the M-step.
  • Algorithm: The method uses a momentum encoder updated as θ′ = 0.999 ∗ θ′ + 0.001 ∗ θ.

APPENDIX D COCO OBJECT DETECTION AND SEGMENTATION

The COCO appendix evaluates fine-tuned object detection and instance segmentation with Mask R-CNN. PCL outperforms both MoCo and supervised pre-training across all reported metrics.

  • COCO evaluation: PCL outperforms both MoCo and supervised pre-training in all COCO detection and segmentation metrics.The evaluation uses Mask R-CNN with a C4 backbone on COCO train2017 and val2017.
  • Metrics: The evaluation reports bounding-box AP and mask AP on the COCO val2017 set.

APPENDIX E TRAINING DETAILS FOR TRANSFER LEARNING EXPERIMENTS

The transfer-learning appendix specifies preprocessing, classifiers, optimization, and detector settings for linear classification, semi-supervised learning, and VOC object detection.

  • Linear classification: Linear classification uses resized center-cropped images and global average pooling or conv5 features, with task-specific linear SVM or logistic regression classifiers.Places uses approximately 9000-D pooled conv5 features, while ImageNet and VOC use 2048-D global average pooling features.
  • Semi-supervised learning: Semi-supervised learning fine-tunes ResNet-50 on 1% or 10% labeled ImageNet subsets using separate learning rates for the ConvNet and classifier.Training runs for 20 epochs with scheduled learning-rate drops.
  • Object detection: VOC object detection uses a frozen R50-FPN backbone with fixed BatchNorm parameters and 15 epochs of SGD fine-tuning.The learning rate drops by 0.1 at epoch 12.

APPENDIX F EVALUATION OF CLUSTERING

PCL cluster quality is evaluated by adjusted mutual information between ImageNet training-data clusterings and ground-truth labels. AMI corrects for chance and ranges from an identical-partition value of 1 to an expected random-partition value of 0.

  • Evaluation: PCL cluster quality is measured by adjusted mutual information between generated clusterings and ImageNet ground-truth labels.
  • Metric interpretation: AMI equals 1 for identical partitions and has expected value 0 for random independent partitions, while correcting mutual information’s cluster-count bias.

APPENDIX G CONVERGENCE PROOF

PCL’s EM procedure alternates prototype-distribution estimation and parameter optimization, yielding a monotonically increasing objective and convergence. Its clusters also reveal both class-level and fine-grained semantic structure.

  • EM convergence: PCL’s convergence proof uses an E-step that estimates prototype distributions as Qt(ci) = p(ci; xi, θt).The equality condition for the auxiliary bound is Q(ci) = p(ci; xi, θ).
  • EM convergence: During the M-step, PCL fixes Qt(ci) and trains θ to maximize Equation 14.This optimization establishes the next iteration’s objective bound.
  • EM convergence: F(θt) monotonously increases with more iterations, so the algorithm converges.The proof attributes convergence to the monotonic increase of the objective across EM iterations.
  • Semantic structure: PCL clusters images from the same class together while finding fine-grained patterns that distinguish subclasses.Figure 6 illustrates this hierarchy, including horse-and-man images within a broader horse cluster.
Loading 2005.04966v5…