PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovering Global False Negatives On the Fly for Self-supervised Contrastive Learning

Vicente Balmaseda, Bokun Wang, Ching-Long Lin, Tianbao Yang

arXiv:2502.20612v2cs.LGcs.AIcs.CVstat.ML

TL;DR

In self-supervised contrastive learning, semantically similar samples can be treated as negatives, pushing their embeddings apart. GLOFND learns dynamic thresholds for each anchor to find false negatives globally, and experiments report improvements across unimodal and bimodal contrastive tasks with minimal computational overhead.

  • Problem

    Instance-discrimination negatives can be semantically similar, pushing their embeddings apart and discarding crucial semantic information; false negatives were about 1% of ImageNet100 negative pairs.

    Source: §1 · pp. 1–2

  • Method

    GLOFND learns a dynamic threshold for each anchor to identify the top-α% most similar negatives across the dataset on the fly.

    Source: §1 · p. 2

  • Results

    GLOFND improves existing contrastive-learning methods on unimodal and bimodal tasks with minimal computational overhead.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Qualitative results show better separation between learned representations of different classes when false negatives are identified and removed.

    Source: §1 · p. 2

    The downstream benefits depend on the pretraining dataset’s false-negative proportion and how false negatives are defined for the downstream task.

    Source: §5 · p. 9

Abstract

from arXiv · show

In self-supervised contrastive learning, negative pairs are typically constructed using an anchor image and a sample drawn from the entire dataset, excluding the anchor. However, this approach can result in the creation of negative pairs with similar semantics, referred to as "false negatives", leading to their embeddings being falsely pushed apart. To address this issue, we introduce GloFND, an optimization-based approach that automatically learns on the fly the threshold for each anchor data to identify its false negatives during training. In contrast to previous methods for false negative discovery, our approach globally detects false negatives across the entire dataset rather than locally within the mini-batch. Moreover, its per-iteration computation cost remains independent of the dataset size. Experimental results on image and image-text data demonstrate the effectiveness of the proposed method. Our implementation is available at https://github.com/vibalcam/GloFND.

1. Introduction

False negatives arise when semantically similar images are treated as negative pairs, pushing their embeddings apart. GLOFND learns dynamic, dataset-wide thresholds for each anchor to identify these negatives during training.

1. Introduction

  • Instance discrimination samples negatives irrespective of semantics, so augmented views of semantically similar images can be pushed apart and crucial semantic information discarded.Figure 1 shows examples encountered during ImageNet training.Source: §1 · p. 1, Figure 1 · p. 1
  • Approximately 1% of ImageNet100 negative pairs were false negatives; a linear classifier achieved up to 10% lower accuracy with such representations in a semi-supervised setting.This corresponds to around 20,000 false negatives per batch of 1,024 or about 325 per batch of 128.Source: §1 · pp. 1–2
  • Identified false negatives can be filtered from training or incorporated as additional positive pairs, but confidently identifying them without labels is challenging.Some are difficult to identify, especially after data augmentation.Source: §1 · p. 2
  • Local discovery assesses similarities within a mini-batch, while the global IFND approach clusters the dataset at specific epochs, which can be computationally expensive at scale.A mini-batch's most similar item may not be semantically similar in the full data space, especially when batches are small.Source: §1 · p. 2
  • GLOFND learns a dynamic threshold for each anchor to select the top-α% most similar negatives across the dataset on the fly.It alternates threshold updates by SGD with encoder updates using a modified contrastive loss that can exclude identified false negatives.Source: §1 · p. 2
  • GLOFND integrates with multiple contrastive-learning techniques and shows empirical success on unimodal, bimodal, and semi-supervised tasks, with minimal computational overhead.The reported qualitative comparison shows better separation between learned representations of different classes.Source: §1 · p. 2

2. Related Work

Prior work spans self-supervised pretext tasks, instance-discrimination contrastive learning, semantic-aware methods, and dataset-wide clustering. The paper situates its approach amid limits in semantic awareness, small-batch reliability, and clustering cost.

2. Related Work

  • Early self-supervised methods used crafted pretext tasks, such as patch-offset prediction or jigsaw solving, which may not generalize across diverse domains.Other examples include colorization and unsupervised deep clustering.Source: §2 · p. 2
  • Contrastive methods such as MoCo, SimCLR, and SogCLR differ in how they generate negatives, but can still treat semantically similar images as negative pairs.SimCLR performance degrades as batch size decreases; SogCLR does not rely on large batch size.Source: §2 · pp. 2–3
  • Semantic-aware methods include individualized temperatures, supervised or weakly supervised objectives, and selecting top-k similar samples within a batch.For small mini-batches, the most similar negatives may not be reliable false negatives; GLOFND instead detects each sample’s false negatives.Source: §2 · p. 3
  • Dataset-wide clustering identifies pairs within the same cluster as false negatives, but requires computing all dataset embeddings and applying k-means, making it costly at scale.The paper motivates a global approach that is agnostic to batch size and scalable to large datasets.Source: §2 · p. 3

3. CL with Global False Negative Identification

GLOFND learns dynamic thresholds for each anchor to identify globally similar negatives, then filters identified false negatives from the contrastive loss. The approach extends from unimodal contrastive learning to image-text learning.

3.1. Learning Dynamic Per-Anchor Thresholds

  • GLOFND treats the top α% of an anchor’s most similar negatives as false negatives, with α adjustable to the desired semantic granularity.The threshold is the (1 − α)-quantile of similarity scores; the paper notes that the appropriate α depends on the task’s granularity.Source: §3.1 · pp. 3–4
  • An optimization-based quantile threshold selects the top α% of negatives: its solution is the k-th largest similarity or a value between the k-th and (k + 1)-th largest.Here k = ⌈α|S−_i|⌉ for anchor x_i.Source: §3.1 · p. 4

3.2. GLOFND for Unimodal CL

  • For unimodal learning, each iteration updates thresholds for sampled anchors, filters identified false negatives from the loss, and updates the encoder; threshold computation is independent of the contrastive loss.The loss determines how filtered false negatives are handled during training.Source: §3.2 · p. 4

3.2.1. UPDATING THE THRESHOLD λ

  • Thresholds are updated for sampled anchors using stochastic subgradients, while thresholds for other anchors remain unchanged, keeping computation mini-batch-wise while tracking thresholds across the dataset.The update projects thresholds onto [−1, 1], the range of similarity scores.Source: §3.2.1 · p. 4

3.2.2. UPDATING ENCODER NETWORK Ew

  • The encoder update excludes negatives above each anchor’s threshold, and a moving-average estimator is used to alleviate the requirement for a large batch size.The algorithm also reports O(B2) overhead for basic matrix computations, relatively negligible compared with similarity and forward/backward computations.Source: §3.2.2 · p. 5

3.3. Extension to Bimodal CL

  • GLOFND extends to image-text contrastive learning by learning separate thresholds for image and text anchors and applying the same general procedure.For image anchors, the negatives are texts; for text anchors, the negatives are images.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate GLOFND in unimodal, semi-supervised unimodal, and bimodal contrastive learning, alongside ablations of its thresholds and comparison with a mini-batch method. Across these settings, the reported results assess false-negative identification, representation performance, and computational cost.

4.1. Unimodal and Semi-supervised Experiments

  • GLOFND raises mean precision by 20.83% and recall by 5.14%, reaching 53.10% F1, 16.68% above FNC.The reported final-epoch false-negative identification results accompany the linear evaluation results.Source: Table 1 · p. 6, §4.1 · pp. 6–7

4.2. Bimodal Experiments

  • GLOFND improves the overall Datacomp score for both SogCLR and FastCLIP and enhances both models across most reported bimodal metrics.Datacomp averages 38 zero-shot tasks; the table also reports retrieval and ImageNet-family task averages.Source: Table 3 · p. 7, §4.2 · p. 7

4.1. Unimodal and Semi-supervised Experiments

  • Removing GLOFND-identified false negatives improves semi-supervised performance by 1.04%-2.44% and transfer-learning average by 1.64%, with a reported 2% per-epoch training-time increase.Table 2 reports transfer-learning averages and improvement relative to the SogCLR baseline.Source: Table 2 · p. 6, §4.1 · p. 7

4.3. Ablation Study

  • ImageNet100 learned λ_i values span 0.1-0.6, and per-anchor thresholds consistently outperform the Single λ variant in semi-supervised evaluation.Figure 4 compares SogCLR without false-negative identification, FNC, a single learned threshold, and GLOFND.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4. Comparison with Mini-batch Top-k Method

4.3. Ablation Study

  • GLOFND reaches and maintains the target α false-negative share after a few epochs; its estimated λ_i error is below half of FNC's.For estimated optimal-threshold error, GLOFND reports MAE 0.1 and RMSE 0.13, versus FNC's MAE 0.21 and RMSE 0.28.Source: Figure 3 · p. 7, §4.3 · p. 8
  • On ImageNet100, semi-supervised performance peaks when GLOFND starts at epoch 70 and declines with later starts.At batch size 128, SogCLR + GLOFND adds 2% per-epoch training time (435.19 s vs 426.67 s), comparable to FNC (434.06 s).Source: §4.3 · p. 8

4.4. Comparison with Mini-batch Top-k Method

  • Across example mini-batches, GLOFND's dynamic threshold selects varying numbers of false negatives, while FNC's fixed top-k can select less-similar negatives.GLOFND identifies more false negatives than FNC in the first and last illustrated rows.Source: Figure 6 · p. 9, §4.4 · p. 9

5. Conclusions

GLOFND learns per-anchor thresholds to identify global false negatives on the fly, and experiments show improvements to unimodal and bimodal contrastive learning with minimal computational overhead. The paper limits its approach to filtering and notes that downstream benefits depend on false-negative prevalence and task definitions.

5. Conclusions

  • GLOFND learns a threshold for each anchor to identify false negatives whose similarity exceeds the desired quantile across the dataset.The method is optimization-based and identifies false negatives on the fly.Source: §5 · p. 9
  • Experiments show improvements to unimodal and bimodal contrastive learning methods with minimal computational overhead.Source: §5 · p. 9
  • The approach addresses false negatives through filtering, which the authors report is effective in their settings.They suggest future work could explore more advanced methods to improve downstream performance.Source: §5 · p. 9
  • Downstream benefits depend on the pretraining dataset’s false-negative proportion and how false negatives are defined for the downstream task.Source: §5 · p. 9

A. Proof of Lemma 3.1

The proof characterizes threshold solutions using ordered similarity scores and subgradient conditions. It identifies the score at the relevant rank as a solution and notes that a range of values can also solve the optimization in a specified case.

A. Proof of Lemma 3.1

B. More Details on Extension to Bimodal CL

The bimodal extension learns separate thresholds for image and text anchors from negatives drawn across the dataset. It uses those thresholds to exclude identified false negatives and update the image-text encoders.

B. More Details on Extension to Bimodal CL

C. High-level Intuition for α Hyperparameter

The α hyperparameter sets how many similar negatives GLOFND treats as false negatives, allowing its definition to reflect the desired semantic granularity. Its setting also affects how many negatives are removed and can influence performance.

C. High-level Intuition for α Hyperparameter

  • α controls the percentage of negatives treated as false negatives, allowing the definition to match the task’s semantic granularity.For example, different dog breeds can count as false negatives for coarse car-versus-animal classification but not for breed classification.Source: C. High-level Intuition for α… · p. 14
  • α can be set using prior knowledge about the expected false-negative rate or desired representation granularity, or tuned like temperature.Source: C. High-level Intuition for α… · p. 14
  • If α is too low, GLOFND may identify too few false negatives and have minimal impact; if too high, filtering can leave too few negatives and potentially harm performance.The authors recommend starting low and increasing α until performance gains stop.Source: C. High-level Intuition for α… · p. 14

D. More Details on Experiments

Unimodal experiments ran on one NVIDIA A30, while bimodal experiments used a two-node setup with two NVIDIA A100 GPUs per node.

D. More Details on Experiments

D.1. Additional Details for Unimodal Experiments

The unimodal experiments evaluate frozen pretrained encoders through linear and transfer-learning protocols, with semi-supervised evaluation across four labeled-data fractions.

D.1. Additional Details for Unimodal Experiments

  • Pretraining uses a ResNet-50 with a two-layer projection head, 200 epochs, batch size 128, and LARS; GLOFND updates λ_i with Adam after a tuned start epoch.The GLOFND starting epoch is selected from 70, 90, and 110; α = 0.01.Source: §D.1 · p. 15
  • Linear evaluation freezes the final pretrained encoder, removes its projection head, and trains a single fully connected classifier on the encoder output.Source: §D.1 · p. 15
  • Semi-supervised linear evaluation averages validation top-1 accuracy across training subsets containing 100%, 10%, 1%, and 0.1% of labeled data.Source: §D.1 · p. 15
  • Linear evaluation trains for 90, 285, 900, and 900 epochs at 100%, 10%, 1%, and 0.1% labeled data, respectively, stopping if validation accuracy fails to improve for 100 epochs.L-BFGS is employed, and best test performance is reported across 10 logarithmically spaced ℓ2 regularization values from 10^-6 to 10^5.Source: §D.1 · p. 15
  • The evaluation covers eight transfer datasets, reporting top-1 accuracy for five and mean per-class accuracy for Pets, Caltech-101, and Oxford 102 Flowers.DTD results use only the first split; Caltech-101 test images are randomly selected as 20% per class.Source: §D.1 · p. 15
  • Transfer evaluation fits an ℓ2-regularized multinomial logistic regression classifier on frozen features, selecting each method’s model by semi-supervised average performance.Source: §D.1 · p. 15

D.2. Additional Details for Bimodal Experiments

The appendix reports statistically significant improvements for GLOFND in unimodal evaluations and describes additional bimodal, SimCLR, and CLIP experiments. It also illustrates dynamically identified false negatives as semantically similar to their anchors.

D.2. Additional Details for Bimodal Experiments

  • Bimodal evaluation uses 2,723,840 CC3M image-text pairs and averages performance across 38 Datacomp tasks, including ImageNet variants and image-text retrieval.The selected model is the one with the best Datacomp average.Source: §D.2 · p. 16
  • Bimodal pretraining runs for 37 epochs at global batch size 1024; GLOFND uses Adam-updated λ_i, initialized to 1, with α and start epoch tuned by loss.For SogCLR, GLOFND starts after 15 epochs with α = 5e-4; for FastCLIP, after 20 epochs with α = 1e-3.Source: §D.2 · p. 16, Table 4 · p. 16

D.3. Additional Experimental Results Statistical Significance

  • GLOFND’s semi-supervised improvements over the SogCLR baseline were significant across all scenarios, with p-values below 0.018 and below 1% for the 100% and 1% scenarios.The comparisons use paired t-tests across multiple runs; the reported tables contain the p-values.Source: §D.3 · p. 16, Table 5 · p. 17
  • GLOFND showed significant transfer-learning improvements on 6 of 8 datasets, compared with 2 of 8 for FNC.The reported comparisons are against the SogCLR baseline.Source: §D.3 · p. 17, Table 6 · p. 17

D.2. Additional Details for Bimodal Experiments

  • Additional experiments evaluate GLOFND with SimCLR at batch size 512 and fine-tune OpenAI’s ResNet-50 CLIP on CC3M for 15 epochs, reporting retrieval results.The SimCLR linear-evaluation results and CLIP retrieval results are presented in Tables 7 and 8, respectively.Source: §D.3 · p. 17, Table 7 · p. 17, Table 8 · p. 17
  • For ImageNet100 at α = 0.01, the number of false negatives found varies by anchor under GLOFND’s dynamic thresholds, and the examples are semantically similar to their anchors.Figure 7 depicts the anchor in the left column and identified false negatives in the remaining columns.Source: §D.3 · pp. 17–18, Figure 7 · p. 18

Related papers

Loading 2502.20612v2…