Source-linked AI summary
Boosting Contrastive Self-Supervised Learning with False Negative Cancellation
Tri Huynh, Simon Kornblith, Matthew R. Walter, Michael Maire, Maryam Khademi
TL;DR
False negatives arise when contrastive learning treats semantically similar images from different images as negatives, which can discard semantic information and slow convergence. The paper identifies potential false negatives without labels and mitigates them through elimination and attraction, achieving consistent improvements, including a 5.8% absolute Top-1 accuracy gain with 1% ImageNet labels.
Problem
False negatives are difficult to resolve without labels and can discard semantic information while slowing contrastive-learning convergence.
Method
The paper identifies potential false negatives from augmented views and applies false negative elimination or attraction to the contrastive objective.
Results
The method consistently improves existing contrastive approaches, including a 5.8% absolute Top-1 accuracy improvement with 1% labels on ImageNet.
Takeaways & Limitations
False-negative cancellation provides a practical way to improve contrastive self-supervised learning across varied model and training configurations.
Takeaways & Limitations
The approach assumes false negatives share similarity with augmented views of the anchor, although they may not be as similar to the anchor itself.
Abstract
from arXiv · showhide
Self-supervised representation learning has made significant leaps fueled by progress in contrastive learning, which seeks to learn transformations that embed positive input pairs nearby, while pushing negative pairs far apart. While positive pairs can be generated reliably (e.g., as different views of the same image), it is difficult to accurately establish negative pairs, defined as samples from different images regardless of their semantic content or visual features. A fundamental problem in contrastive learning is mitigating the effects of false negatives. Contrasting false negatives induces two critical issues in representation learning: discarding semantic information and slow convergence. In this paper, we propose novel approaches to identify false negatives, as well as two strategies to mitigate their effect, i.e. false negative elimination and attraction, while systematically performing rigorous evaluations to study this problem in detail. Our method exhibits consistent improvements over existing contrastive learning-based methods. Without labels, we identify false negatives with 40% accuracy among 1000 semantic classes on ImageNet, and achieve 5.8% absolute improvement in top-1 accuracy over the previous state-of-the-art when finetuning with 1% labels. Our code is available at https://github.com/google-research/fnc.
1. Introduction
Contrastive self-supervised learning reliably generates positive pairs but can misclassify semantically similar images as negatives, producing false negatives that discard semantic information and hinder convergence. The paper proposes identifying and canceling false negatives, reporting consistent improvements and strong ImageNet results.
- Contrastive learning setup: Positive pairs use different views of one image, whereas negatives are sampled from different images without considering semantic content.This makes negative-pair construction substantially less reliable than positive-pair generation.
- False negatives: False negatives are negative pairs from the same semantic category, such as two dog images treated as negatives.Without labels, automatically selected negative pairs can belong to the same semantic category.
- False-negative effects: False negatives discard shared semantic features and create contradictory attraction and repulsion objectives that hinder convergence.A feature may be attracted to a positive pair while being repelled from similar content in another image.
- Proposed framework: The paper proposes unlabeled false-negative identification and two cancellation strategies: elimination and attraction.The framework first identifies potential false negatives, then modifies contrastive learning through these two strategies.
- Results: Without labels, the method finds false negatives with ∼40% accuracy among 1000 ImageNet semantic categories.The paper also reports consistent improvements over existing approaches across varied settings.
- Results: 5.8% absolute improvement in Top-1 accuracy is achieved in the semi-supervised ImageNet setting with 1% labels.This result is reported for downstream fine-tuning.
2. Related Work
Related work spans proxy-task, clustering-based, contrastive, and negative-sampling approaches to self-supervised representation learning. The paper distinguishes false-negative cancellation from methods focused on hard negatives, reweighting, or non-contrastive positive-view prediction.
- Proxy tasks: Early self-supervised methods use proxy tasks to guide embeddings, but these tasks are described as heuristic and lacking generality.
- Clustering-based methods: Clustering-based methods iteratively form clusters as pseudo-labels and train networks to classify samples using those pseudo-labels.The paper contrasts its elimination strategy with clustering and notes that its attraction strategy also groups visually connected samples in a different formulation and context.
- Contrastive learning: Contrastive methods improve representation learning through large negative pools, using memory banks, momentum encoders, or large batches with projection and augmentation changes.Examples include PIRL, MoCo, and SimCLR.
- Negative sampling: Most negative-sampling research targets hard negatives, which are true negatives close to the anchor, rather than false negatives.Other work considers hard positives or reweights positive and negative terms, but these approaches generally do not identify false negatives.
- Non-contrastive methods: SwAV and BYOL predict or associate positive views without contrasting against negative samples, so they are not categorized as contrastive learning.SwAV uses online clustering, whereas BYOL uses a momentum encoder as a prediction target.
3. Method
The method identifies potential false negatives by comparing negative samples with additional augmented support views, then mitigates them through elimination or attraction strategies. It also evaluates similarity aggregation and screening choices for detecting candidates.
- 3.1. Contrastive Learning: Contrastive learning attracts positive pairs and repels negatives, but negatives from different images can share semantic content with the anchor.Such false negatives can discard semantic information and slow convergence.
- 3.2.1 False Negative Elimination: False negative elimination removes detected false negatives from the contrastive objective instead of contrasting against them.The set F_i denotes detected false negatives for anchor i.
- 3.2.2 False Negative Attraction: Attraction incorporates true positives from different images to increase training-data diversity, rather than merely ignoring false negatives.The approach is motivated by the potential for greater positive diversity to improve learned embeddings.
- 3.2.3 Finding False Negatives: Support views identify candidate false negatives because a negative may resemble another augmentation of the anchor more than the anchor itself.The procedure generates support views, computes negative-to-support similarities, aggregates them, and selects the most similar samples.
- 3.2.3 Finding False Negatives: Candidate screening uses cosine similarity with top-k, threshold, or combined top-k-and-threshold rules.Mean aggregation is also considered, while thresholding supports dynamic adaptation and top-k uses an approximate false-negative count.
- 3.2.3 Finding False Negatives: False negative elimination outperforms SimCLR across the full tested crop-ratio range, whereas attraction is sensitive to false-negative reliability.Attraction works with very high-similarity candidates, while elimination is less sensitive to thresholds.
4.1. Ablation Studies
The ablations show that false negative cancellation improves contrastive learning, with performance depending on support sets, aggregation, filtering, and cancellation strategy. The method also complements multi-crop and momentum encoders while improving accuracy, detection, class separation, and computational efficiency.
- False Negative Cancellation Strategies: False negative elimination consistently improves top-1 accuracy across crop ratios, despite eliminating only two potential false negatives among 8190 negatives per anchor.The improvement reaches as much as 1%, supporting the significant effect of false negatives.
- False Negative Cancellation Strategies: Support sets improve both cancellation strategies, producing approximately 2% gains for attraction versus approximately 0.2% for elimination.Attraction without a support set can hurt performance, whereas elimination improves over SimCLR with or without one.
- False Negative Cancellation Strategies: Attraction is more sensitive to false-negative quality, working best with highly similar detections, while elimination is less sensitive to filtering thresholds.This sensitivity motivates more reliable false-negative identification for attraction.
- False Negative Cancellation Strategies: Max aggregation consistently outperforms mean aggregation for attraction across support sizes and top-k values, with some gaps exceeding 1%.The difference is less pronounced for elimination because averaging can corrupt scores when false negatives resemble only a subset of the support set.
- False Negative Cancellation Strategies: Top-k filtering generally outperforms threshold filtering, while combining both achieves the best accuracy except for a negligible degradation in attraction at top-4.The comparison covers both elimination and attraction strategies.
- False Negative Cancellation Strategies: False negative attraction improves over SimCLR by 1.75%, exceeding elimination’s 1.02% improvement when detected false negatives are valid.These results use max aggregation, support size eight, top-4 attraction filtering, and top-8 elimination filtering.
4.2. Comparison with State-of-the-Art
Across ImageNet evaluation and transfer tasks, false negative cancellation improves contrastive learning performance over existing contrastive baselines. The method reaches strong results in linear, semi-supervised, classification-transfer, and object-detection evaluations.
- ImageNet evaluation: 74.4% top-1 accuracy is achieved in ImageNet linear evaluation, improving 2.7% over SimCLR v2 and 1.4% over the previous best.The method is second only to SwAV among all approaches and is competitive with BYOL.
- Transfer learning: The method significantly improves performance over SimCLR v1 and SimCLR v2 across 12 classification transfer datasets in both evaluation settings.The evaluation covers datasets including Food, CIFAR10, CIFAR100, Birdsnap, SUN397, Cars, Aircraft, VOC2007, DTD, Pets, Caltech-101, and Flowers.
- Transfer learning: The proposed method outperforms baseline approaches when transferred to PASCAL VOC object detection.The transfer evaluation uses Faster R-CNN with a ResNet-50 C4 backbone on VOC test2007.
5. Conclusion
The paper studies false negatives in contrastive self-supervised learning and introduces strategies for identifying and using them to improve contrastive frameworks. Its experiments provide novel insights and report substantially stronger performance for contrastive learning methods.
- Conclusion: The work addresses false-negative identification, a fundamental problem that had not been adequately studied in contrastive self-supervised learning.It focuses on identifying negative pairs that share semantic content despite coming from different images.
- Conclusion: The proposed strategies use false-negative identification to improve existing contrastive learning frameworks.The paper presents this contribution alongside in-depth experimental analysis.
- Conclusion: The method significantly boosts existing models and sets new performance standards for contrastive self-supervised learning methods.This conclusion is based on the paper’s reported experimental evaluations.