PaperlayerView exact revision on arXiv

Source-linked AI summary

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

Andreas Fürst, Elisabeth Rumetshofer, Johannes Lehner, Viet Tran, Fei Tang, Hubert Ramsauer, David Kreil, Michael Kopp, Günter Klambauer, Angela Bitto-Nemling, Sepp Hochreiter

arXiv:2110.11316v4cs.LGcs.CV

TL;DR

CLIP’s explaining-away problem involves focusing on a few features while neglecting others, linked to insufficient extraction of feature co-occurrences and covariance structure. CLOOB combines modern Hopfield networks with InfoLOOB, and consistently outperforms CLIP in zero-shot transfer learning across the considered architectures.

  • Problem

    CLIP focuses on one or a few features while neglecting others, a problem linked to insufficient extraction of feature co-occurrences and covariance structure.

    Source: §1 · p. 2

  • Method

    CLOOB combines modern Hopfield networks, which amplify covariance structure, with InfoLOOB, which avoids InfoNCE saturation.

    Source: §1 · p. 2, §3 · p. 3, §4 · p. 6

  • Results

    CLOOB consistently outperforms CLIP in zero-shot transfer learning across the considered architectures.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors theoretically justify InfoLOOB for contrastive learning and suggest it as an alternative to InfoNCE.

    Source: §6 · p. 9

    The CLIP training samples used by Radford et al. were unavailable, and limited computational resources prevented comparison with CLIP on other datasets of that size.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3. CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on particular tasks. Though CLIP excels at zero-shot transfer learning, it suffers from an explaining away problem, that is, it focuses on one or few features, while neglecting other relevant features. This problem is caused by insufficiently extracting the covariance structure in the original multi-modal data. We suggest to use modern Hopfield networks to tackle the problem of explaining away. Their retrieved embeddings have an enriched covariance structure derived from co-occurrences of features in the stored embeddings. However, modern Hopfield networks increase the saturation effect of the InfoNCE objective which hampers learning. We propose to use the InfoLOOB objective to mitigate this saturation effect. We introduce the novel "Contrastive Leave One Out Boost" (CLOOB), which uses modern Hopfield networks for covariance enrichment together with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after pre-training on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.

1 Introduction

CLIP learns expressive image embeddings from raw text and performs strongly in zero-shot transfer, but it can focus on a few features while neglecting others. CLOOB combines modern Hopfield networks to reinforce covariance structure with InfoLOOB to avoid InfoNCE saturation.

1 Introduction

  • CLIP’s explaining-away problem is its focus on one or a few features while neglecting others, linked to insufficient extraction of multimodal co-occurrences and covariance.The proposed analogy is that modern Hopfield networks can amplify such structure by associating current perceptions with stored memories.Source: §1 · p. 2
  • Modern Hopfield networks are proposed to amplify co-occurrences and covariance structure in the original multimodal data.The paper describes continuous associative memories with high storage capacity that can retrieve samples in one update.Source: §1 · p. 2
  • Hopfield retrieval increases sample similarity, worsening InfoNCE saturation; InfoLOOB avoids saturation and continues reducing the uniformity penalty.InfoNCE contains terms of the form a/(a + b), whereas InfoLOOB contains a/b terms.Source: §1 · p. 2
  • CLOOB combines modern Hopfield networks with InfoLOOB as a new contrastive learning method.The stated contributions also include theoretical justifications for optimizing InfoLOOB.Source: §1 · p. 2

2 CLOOB: Modern Hopfield Networks with InfoLOOB

CLOOB replaces CLIP’s original embeddings in the contrastive objective with normalized embeddings retrieved from modern Hopfield networks. The networks store image and text embeddings, typically from the minibatch, and retrieved embeddings can also be used for downstream tasks.

2 CLOOB: Modern Hopfield Networks with InfoLOOB

  • Image and text queries retrieve embeddings from separate memories of stored image and text embeddings, respectively.The architecture contrasts a retrieved anchor with positive and negative retrieved samples in each direction.Source: §2 · p. 2, Figure 1 · p. 3
  • CLOOB applies InfoLOOB to normalized Hopfield-retrieved embeddings rather than the original image and text embeddings.The loss uses image-memory retrievals in one term and text-memory retrievals in the other.Source: §2 · pp. 2–3
  • By default, each Hopfield network stores the minibatch embeddings, avoiding encoder computations for additional samples.The memories can instead store prototypes, templates, or proprietary samples to amplify particular embedding features.Source: §2 · p. 3
  • The paper presents CLOOB as a CLIP replacement usable in methods built upon CLIP.Its two major components are Hopfield networks for covariance structure and InfoLOOB to avoid InfoNCE saturation.Source: §2 · p. 3

3 Modern Hopfield Networks for Enriching the Covariance Structure

Modern Hopfield retrievals are intended to reinforce co-occurring features and covariance structure while averaging out sample-specific spurious co-occurrences. The paper reports that these networks increase the effective eigenvalues of embedding covariance matrices during learning.

3 Modern Hopfield Networks for Enriching the Covariance Structure

  • Replacing original embeddings with retrieved embeddings reinforces features that frequently co-occur in stored embeddings and averages out sample-specific spurious co-occurrences.The paper presents this as a mechanism for reinforcing the data’s covariance structure.Source: §3 · p. 3
  • A weighted covariance uses probabilities pi for stored patterns rather than uniform sampling with probability 1/M.The softmax Jacobian is described as the weighting mechanism used to form these covariances.Source: §3 · p. 4
  • The analysis relates agreement in image and text retrievals to similar weighted covariance matrices and large dot products.When subsets of stored embeddings have strong covariance structure, the paper says that structure can contribute to large weighted covariances and dot products.Source: §3 · p. 4
  • Modern Hopfield networks consistently increase the number of effective eigenvalues during learning, indicating greater extraction of covariance structure and mitigation of explaining away.Effective eigenvalues are counted as those needed to account for 99% of the total eigenvalue sum; Figure 2 reports relative changes from a reference epoch.Source: §3 · p. 4

4 InfoLOOB for Contrastive Learning

InfoLOOB avoids the saturation that can hinder InfoNCE when positive-pair similarity is high, allowing the uniformity penalty to keep decreasing. The paper reports increased uniformity for image and text embeddings trained with InfoLOOB and provides theoretical discussion of the objective.

4 InfoLOOB for Contrastive Learning

  • InfoLOOB excludes the positive sample from the negative-sample matrix, unlike InfoNCE.The losses use the same positive pair, but InfoLOOB’s negative set omits the positive sample.Source: §4 · p. 5
  • InfoNCE’s gradient is scaled by (1 − p1) relative to InfoLOOB, so it saturates as similarity between the anchor and positive sample increases.The paper identifies p1 as the softmax similarity between the anchor and positive sample.Source: §4 · p. 6
  • InfoLOOB does not saturate and keeps decreasing the uniformity penalty, with image and text embeddings becoming more uniform on the sphere.Figure 3 uses Ajne uniformity test statistics; higher uniformity correlates with a lower uniformity penalty.Source: §4 · p. 6, Figure 3 · p. 6
  • The paper discusses InfoLOOB’s theoretical properties but states that with neural networks it is not an upper bound on mutual information.It is used as an objective because it does not suffer from InfoNCE’s saturation effects.Source: §4 · p. 6

5 Experiments

The experiments compare CLOOB and CLIP on zero-shot transfer across Conceptual Captions and YFCC, then examine how the method’s components affect learning. CLOOB outperforms CLIP across the reported YFCC architectures and on most or all reported tasks in the other comparisons.

5 Experiments

  • The experiments pre-train on Conceptual Captions and YFCC and evaluate zero-shot transfer on seven image-classification datasets.YFCC uses 15 million image-text pairs; the Conceptual Captions description notes its rich captions and smaller dataset.Source: §5 · p. 6, §5.2 · p. 7

5.1 Conceptual Captions Pre-training

  • On Conceptual Captions, CLOOB significantly outperforms CLIP on most tasks or matches its performance at both reported checkpoints.The checkpoints cover 31-epoch and 128-epoch training; Table 1 reports mean accuracy over five runs.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 YFCC Pre-training

  • On YFCC, CLOOB beats original CLIP on all 7 reported zero-shot tasks and the CLIP reimplementation on 7 of 8 tasks with ResNet-50 and all tasks with larger ResNets.CLOOB performance increases with encoder size.Source: Table 3 · p. 8, §5.2 · p. 8, Table 2 · p. 8

5.3 Ablation studies

  • Modern Hopfield retrievals increase effective covariance eigenvalues, but their higher sample similarity intensifies InfoNCE saturation and is associated with low uniformity.The reported ablation also finds a small number of effective eigenvalues under this regime.Source: §5.3 · p. 9
  • Combining modern Hopfield networks with InfoLOOB counters InfoNCE saturation, increases embedding uniformity, and shows a strong synergistic effect.The uniformity increase is reported for image and text embeddings on the sphere.Source: §5.3 · p. 9
  • InfoLOOB without Hopfield can overfit alignment, with high similarities for unmatched pairs that may reduce zero-shot performance.Figure 4 compares matched pairs with the ten highest-similarity unmatched pairs.Source: §5.3 · p. 9, Figure 4 · p. 9
  • CLOOB balances InfoLOOB overfitting against modern Hopfield underfitting and remains in effective learning regimes.Further ablations are reported in the appendix.Source: §5.3 · p. 9

6 Conclusion

CLOOB combines modern Hopfield networks with InfoLOOB, and the paper reports consistent zero-shot transfer gains over CLIP across the considered architectures and datasets.

6 Conclusion

  • CLOOB consistently outperforms CLIP on zero-shot transfer across all considered architectures and datasets, after pre-training on Conceptual Captions and YFCC.The comparison covers seven zero-shot transfer tasks.Source: §6 · p. 9
  • Modern Hopfield networks add covariance structure to embeddings, helping build more relevant features and mitigate explaining away.The conclusion links covariance enrichment with addressing CLIP’s focus on a few features.Source: §6 · p. 9
  • InfoLOOB avoids InfoNCE’s saturation problem, and the paper theoretically justifies it as an alternative contrastive-learning objective.Source: §6 · p. 9

Checklist

The checklist reports the paper’s scope, data considerations, reproducibility details, and potential societal impacts.

Checklist

  • The method is limited to natural images and short text prompts; its use with medical or biological images remains unexplored.The authors hypothesize possible use in similar data domains but say this has not been assessed.Source: Checklist · p. 18
  • Error bars were added where computationally feasible, and the experiments used roughly 11,000 GPU hours with A100s.Source: Checklist · p. 18
  • The paper reports providing code and a GitHub repository, and says training details are documented in the main text and appendix.Source: Checklist · p. 18
  • The checklist says public research datasets were used, including YFCC under a Creative Commons licence, and reports discussing data consent and assets.Source: Checklist · p. 18
  • The datasets have biases, including social biases, and responsible use requires careful training-data selection and awareness of potential biases.The method relies on human-annotated data and human decisions, which the authors describe as usually strongly biased.Source: Checklist · p. 18

A Appendix

The appendix covers theoretical properties and analysis of InfoLOOB and InfoNCE, experimental details, modern Hopfield networks, and related work.

A Appendix

  • Section A.1 derives InfoNCE as a lower bound and InfoLOOB as an upper bound on mutual information, and discusses the loss and its gradients.Source: A Appendix · p. 20
  • Section A.2 provides experimental details, while Section A.3 reviews continuous modern Hopfield networks.Source: A Appendix · p. 20
  • Section A.4 discusses further related work.Source: A Appendix · p. 20

Contents of the appendix

The appendix contents list theoretical analyses, experiments, a review of modern Hopfield networks, and further related work.

Contents of the appendix

List of theorems

The theorem list covers mutual-information bounds, covariance structure, and modern Hopfield network retrieval and storage.

List of theorems

List of definitions

The definition list contains an entry for patterns that are stored and retrieved.

List of definitions

A.1.1 InfoNCE: Lower Bound on Mutual Information

This section derives InfoNCE as a lower bound on mutual information and describes its sampled-pair formulation. It also notes a limitation in a tighter approximation.

A.1.1 InfoNCE: Lower Bound on Mutual Information

  • The section says the lower bound can be loose when a becomes small, and that an approximation requires an assumption whose enforceability is unclear.Source: §A.1.1 · p. 22
  • For an anchor y, the sampling setup uses one positive x1 drawn from p(x1 | y) and N−1 negatives drawn from p(x).The section also describes sampling N pairs independently from p(x, y).Source: §A.1.1 · pp. 21–22
  • InfoNCE with a score function, and its probability-based form, are stated to lower-bound mutual information.Source: §A.1.1 · p. 23
  • The probability-based bound is a special case of the score-function bound.Source: §A.1.1 · p. 23

A.1.2 InfoLOOB: Upper Bound on Mutual Information

This section examines InfoLOOB’s objective, bounds, estimators, and gradients, alongside modern Hopfield retrieval and its relation to contrastive learning. It also describes analyses and evaluations of their combination, including effects on covariance, similarity, downstream performance, and computational cost.

A.1.2 InfoLOOB: Upper Bound on Mutual Information

  • InfoLOOB is an upper bound on mutual information when the main assumption holds; the probability formulation with negatives drawn from p(x) is a special case.The theorem also establishes the bound for iid negatives drawn from the undersampling distribution, subject to the stated assumption.Source: §A.1.2 · p. 27

A.1.3 InfoLOOB: Analysis of the Objective

  • Without undersampling, InfoLOOB with a neural scoring function is not an MI upper bound, but it is bounded above for bounded similarities and differs from MI by KL and DE terms.The KL term compares the variational distribution with the posterior; DE is the difference between the mean score and its empirical mean.Source: §A.1.3 · pp. 29, 31, 32
  • Maximizing InfoLOOB most effectively reduces KL by aligning q(x|y) with p(x|y), yielding f(y|x) ≈ p(y|x), while DE is difficult to change.The analysis identifies KL as the dominant term; DE depends only on the marginals and is not changed by scaling the score function.Source: §A.1.3 · pp. 29, 34, 35
  • For large N, Hoeffding’s bound makes DE small with high probability, so the KL mismatch is the relevant term in maximizing InfoLOOB.The analysis assumes the score or similarity satisfies the stated concentration conditions.Source: §A.1.3 · pp. 29, 33
  • When the posterior and variational distributions match, the derivative vanishes; when the expectation is approximated by the negative-sample average, both DE and its gradient are small.The derivative also becomes small for small b − a and large N.Source: §A.1.3 · pp. 35–37

A.1.4 InfoNCE and InfoLOOB: Gradients

  • InfoLOOB differs from InfoNCE in excluding the positive example from the negatives; the cited text also states that the learning rate is scaled by (1 − p1).Source: §A.1.4 · pp. 37–38

A.1.5 InfoLOOB and InfoNCE: Probability Estimators

  • Both losses can be viewed as cross-entropy losses, but separately estimating the normalizing constant is biased for InfoNCE and unbiased for InfoLOOB.Source: §A.1.5 · pp. 39–40

A.1.6 InfoLOOB and InfoNCE: Losses

  • The cross-modal retrieval dot product requires M + K stored patterns rather than M or K for the other dot products, and has higher variance.Source: §A.1.6 · p. 41

In particular everywhere holds

  • The dot product of two U-retrieved embeddings is expressed using weighted covariances, whose mean Jacobian is symmetric, positive semi-definite, and bounded in spectral norm by 0.5β.Source: In particular everywhere holds · p. 43

A.2.1 Ablation studies

  • InfoLOOB with Hopfield retrieval outperforms the other ablation models on 5/8 datasets at CC epoch 31, 7/8 at CC epoch 128, and 6/8 at YFCC epoch 28.By contrast, InfoLOOB alone and InfoNCE with Hopfield retrieval decrease performance versus InfoNCE on most tasks.Source: Table A1 · p. 44, Table A2 · p. 44, §A.2.1 · p. 44
  • With learning-rate restarts, most downstream-task performance either increases or remains roughly the same for both CLIP and CLOOB.The scheduler enables flexibility in total training epochs and training up to a plateau.Source: §A.2.1 · pp. 44–45, Table A3 · p. 45

A.2.4 Zero-shot evaluation

  • The study evaluates its method on downstream classification tasks, adding zero-shot results for multiple datasets and reporting zero-shot performance on additional datasets beyond the main paper.Pretraining considered Conceptual Captions (2.9 million images with corresponding high-quality captions) and a filtered YFCC subset yielding 15 million image-caption pairs. Zero-shot evaluation uses the prompt published in CLIP; when multiple prompts are used, their embeddings are averaged.Source: §A.2.3 · p. 45, §A.2.4 · p. 47

A.2.5 Linear probing

  • YFCC linear-probing results are mixed across ResNet encoders, but increasing CLOOB encoder capacity improves accuracy.The authors suggest task dependence and the mismatch between multimodal pattern restrictions and single-modality evaluation as possible explanations for mixed results.Source: §A.2.5 · p. 48, Table A7 · p. 48

A.2.6 Image-Text retrieval

  • On the 13,330-pair CC validation set, CLOOB significantly outperforms CLIP in both image-to-text and text-to-image retrieval.The task is to find the matching image for a text or the matching text for an image.Source: §A.2.6 · p. 49, Table A8 · p. 49

A.2.7 Analysis of the image and text embeddings

  • InfoLOOB with Hopfield starts with many effective eigenvalues and increases them during training, while both methods without Hopfield struggle to increase their effective eigenvalue counts.InfoNCE with Hopfield starts with few effective eigenvalues, attributed to InfoNCE saturation impeding covariance extraction, but increases them steadily during training.Source: §A.2.7 · p. 49, Figure A1 · p. 50
  • InfoLOOB does not saturate: with Hopfield retrieval, it raises matched-pair similarity while reducing average similarity for the top-1,000 unmatched pairs versus InfoNCE without Hopfield.InfoLOOB alone can overfit, assigning cosine similarity one to many pairs; InfoLOOB with Hopfield does not show the top-10 unmatched-pair overfitting pattern.Source: §A.2.7 · pp. 49–50, Figure A2 · p. 51, Figure A3 · p. 51

A.2.8 Training time and memory consumption

  • CLOOB uses the same memory as CLIP, incurs approximately 5% more training time, and adds O(N) retrieval complexity per sample.Modern Hopfield networks are continuous associative memories that retrieve with one update; their update rule coincides with transformer attention.Source: §A.2.8 · p. 50, Table A9 · p. 50, §A.3 · p. 52

A.3 Review of Modern Hopfield Networks

  • With well-separated patterns, a modern Hopfield network typically converges after one update, with retrieval error exponentially small in the separation.The theorem describes retrieval of a pattern from a query using the network update rule.Source: §A.3 · p. 52
  • Modern Hopfield networks can store exponentially many random patterns, supporting retrieval of hundreds of thousands of embeddings when their dimension is large enough.The stated storage-capacity result is for randomly chosen patterns on a sphere; the paper gives example proven parameter settings.Source: §A.3 · p. 53

A.4 Further Related Work

  • Related InfoNCE variants include multi-positive JCL, prototype-based ProtoNCE, and sentence-embedding SimCSE.Other cited extensions use sampled negatives, conditional objectives, or video representations.Source: §A.4 · p. 53
  • CLOOB uses InfoLOOB as a mutual-information upper bound; maximizing CLUB failed, consistent with the embedding distribution not being uniform as contrastive learning requires.CLUB was otherwise used to minimize mutual information, with analysis by Wang & Liu (2021).Source: §A.4 · pp. 53–54
  • CLIP has been applied to vision-and-language tasks, generative models, clustering, robustness and out-of-distribution studies, zero-shot prompting, and text-to-video retrieval.The cited works also discuss adversarial attacks on uncurated datasets and broader evaluation schemes.Source: §A.4 · p. 54

Related papers

Loading 2110.11316v4…