Source-linked AI summary
Weakly Supervised Contrastive Learning
Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, Chang Xu
TL;DR
Instance-discrimination contrastive learning can mistake semantically similar images for negatives, creating class collisions that may hurt representations. WCL combines regular instance discrimination with graph-derived weak labels, supervised contrastive learning, and K-Nearest Neighbor multi-crops. It reports state-of-the-art performance, including 65% and 72% ImageNet Top-1 Accuracy with 1% and 10% labeled samples using ResNet50.
Problem
Instance discrimination ignores relationships between different images and can create class collisions when semantically similar instances are treated as different classes.
Method
WCL uses two projection heads: one for instance discrimination and one for graph-based weak-label generation followed by supervised contrastive learning, with K-Nearest Neighbor multi-crops to expand positives.
Results
65% and 72% ImageNet Top-1 Accuracy are achieved with 1% and 10% labeled samples using ResNet50, exceeding SimCLRv2 with ResNet101.
Takeaways & Limitations
WCL improves self-supervised representations across datasets and reports state-of-the-art results for semi-supervised and transfer learning.
Takeaways & Limitations
K-Nearest Neighbor similarities are unreliable early in training, so the method requires standard multi-crops as a warm-up before switching strategies.
Abstract
from arXiv · showhide
Unsupervised visual representation learning has gained much attention from the computer vision community because of the recent achievement of contrastive learning. Most of the existing contrastive learning frameworks adopt the instance discrimination as the pretext task, which treating every single instance as a different class. However, such method will inevitably cause class collision problems, which hurts the quality of the learned representation. Motivated by this observation, we introduced a weakly supervised contrastive learning framework (WCL) to tackle this issue. Specifically, our proposed framework is based on two projection heads, one of which will perform the regular instance discrimination task. The other head will use a graph-based method to explore similar samples and generate a weak label, then perform a supervised contrastive learning task based on the weak label to pull the similar images closer. We further introduced a K-Nearest Neighbor based multi-crop strategy to expand the number of positive samples. Extensive experimental results demonstrate WCL improves the quality of self-supervised representations across different datasets. Notably, we get a new state-of-the-art result for semi-supervised learning. With only 1\% and 10\% labeled examples, WCL achieves 65\% and 72\% ImageNet Top-1 Accuracy using ResNet50, which is even higher than SimCLRv2 with ResNet101.
1. Introduction
Contrastive learning commonly uses instance discrimination, but treating every image as a separate class can create class collisions between semantically similar images. WCL addresses this with weak similarity supervision, a two-head design, and K-Nearest Neighbor multi-crops.
- Instance discrimination pulls augmented views together while pushing other images apart, but can produce class collisions between semantically similar instances.The problem arises because different instances may nevertheless share similar semantic information.
- WCL treats instance similarities as weak supervision and aggregates samples assigned the same weak label.Weak labels are determined from a nearest-neighbor graph and connected components.
- The two-head framework combines regular instance discrimination with a second head that attracts similar samples through supervised contrastive learning.The separate heads help accommodate noisy or unreliable mined similarities.
- A graph-based, parameter-free method adaptively identifies similar samples for weak-label generation.The graph is formed from nearest-neighbor relations and connected components are used to identify reachable similar samples.
- K-Nearest Neighbor multi-crops expand the number and diversity of positive samples beyond standard multi-crops.The strategy propagates weak labels through additional related views.
- 65% and 72% ImageNet Top-1 Accuracy are achieved with 1% and 10% labeled samples using ResNet50, exceeding SimCLRv2 with ResNet101.The authors report these results as a new state-of-the-art performance for contrastive-learning methods.
2. Related Work
Prior self-supervised learning developed diverse pretext tasks, while contrastive learning became a leading approach by shaping representations through positive and negative examples. The paper’s framework extends this line of work with a two-head design that combines instance discrimination and similarity-based supervision.
- Self-Supervised Learning: Early self-supervised methods used pretext tasks including patch-offset prediction, jigsaw solving, colorization, inpainting, rotation prediction, clustering, and reconstruction.
- Contrastive Learning: Contrastive learning became one of the most successful self-supervised learning approaches by improving positive augmentations and negative-sample exploration.Examples include SimCLR’s augmentation composition and MoCo’s momentum encoder with a queue of negatives.
- Contrastive Learning: The proposed two-head structure pairs regular instance discrimination with a head that generates weak labels through connected-component labeling for supervised contrastive learning.
- Contrastive Learning Without Negatives: Negative-free approaches such as BYOL learn representations by predicting a target-network representation from another augmented view and use a predictor to avoid collapse.SimSiam extends the siamese structure described for contrastive learning without negatives.
3. Method
WCL extends instance-discrimination contrastive learning with weak supervision from graph-mined similarities, using separate projection heads and multi-crop positives to address class collision.
- 3.1. Revisiting Contrastive Learning: Contrastive learning pulls augmented views of one image together while pushing away other images, but this instance-discrimination setup ignores relationships between different images and induces class collision.The paper frames class collision as a mismatch between treating every instance as a separate class and the existence of similar images.
- 3.2. Instance Similarities as Weak Supervision: WCL uses a primary head for instance discrimination and an auxiliary head to mine similar samples and attract them in a separate embedding space.The two heads separate instance-level and similarity-based objectives, while the primary head helps maintain convergence when weak labels are noisy.
- 3.2. Instance Similarities as Weak Supervision: The weak label marks pairs in the same connected component as similar, and supervised contrastive learning uses those marked pairs as positives.The weak-label indicator is inserted into the contrastive objective, replacing augmentation-only positive-pair selection for the auxiliary task.
- 3.3. Weak Label Generation: A 1-nearest-neighbor graph connects each sample with its closest sample, and connected components define which samples receive the same weak similarity label.The graph is sparse and symmetric; connected components are computed with a disjoint-set procedure using makeSet, union, and find.
- 3.3. Weak Label Generation: The weak-label process is parameter-free, reciprocal, and deterministic, avoiding hyperparameter optimization and dependence on an initial state.Reciprocity means yij = yji because the graph is undirected and labels are assigned by connected components.
- 3.4. Label Propagation with Multi-Crops: WCL adds K-Nearest Neighbor multi-crops to expand positive samples, first warming up with standard multi-crops because mined similarities are unreliable early in training.The strategy is intended to provide more effective information than overlapping crops of the same image, while low-resolution crops reduce computational cost.
4. Experimental Results
Experiments across CIFAR, ImageNet, and VOC evaluate WCL’s weak supervision, two-head design, K-NN multi-crops, and representation transfer. WCL generally improves over contrastive baselines, with especially strong semi-supervised and low-shot results.
- Ablation Studies: WCL outperforms SimCLR across CIFAR-10 and CIFAR-100 batch-size and training-epoch settings.Improvements range from 0.98% to 2.91% on CIFAR-10 and 0.73% to 1.80% on CIFAR-100.
- Ablation Studies: Separating instance discrimination and similar-sample attraction across two heads improves ImageNet-100 performance by 1.72% over the baseline.Applying both losses on one head causes a 0.53% drop, while multi-crops further improves performance by 2.26%.
- Ablation Studies: 80.78% Top-1 accuracy is achieved with K-NN multi-crops, improving 5% over SimCLR’s 75.79% baseline.The K-NN strategy gives a 1% improvement over standard multi-crops after 50 warm-up epochs.
- Ablation Studies: WCL’s t-SNE visualization shows greater intra-class compactness and inter-class discrepancy than SimCLR.The visualization uses representations from 10 randomly selected classes.
- Comparison on ImageNet-1K Dataset: With negligible additional computational cost, WCL surpasses SimCLR by 1.7% and matches FNCancel on ImageNet-1K.With K-NN crops, WCL reaches 71.0 Top-1 accuracy at 31% additional cost, twice faster than FNCancel and 0.6% better in linear evaluation.
- Comparison on ImageNet-1K Dataset: WCL achieves state-of-the-art linear evaluation after 200 pretraining epochs and remains above most recent methods after 800 epochs, though slightly below SwAV.Semi-supervised evaluation improves over SwAV by 4.4% with 1% labels and 0.9% with 10% labels under the PCL setting.
5. Conclusion
WCL combines two projection heads with a graph-derived weak-label objective and introduces K-NN multi-crops to improve unsupervised representations. Experiments report effectiveness across datasets and state-of-the-art performance in semi-supervised and transfer learning.
- 5. Conclusion: WCL uses one projection head for instance discrimination and another for graph-based weak-label supervised contrastive learning.The second head generates weak labels with Connected Components Labeling and swaps them across augmentations.
- 5. Conclusion: The K-NN multi-crops strategy expands the number of positive samples to K times and provides more effective information.
- 5. Conclusion: Experiments on CIFAR10, CIFAR-100, and ImageNet-100 demonstrate the effectiveness of each component.
- 5. Conclusion: Semi-supervised and transfer-learning results demonstrate state-of-the-art performance for unsupervised representation learning.