PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models

Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, Thomas Brox

arXiv:2404.07983v3cs.CVcs.LG

TL;DR

The paper examines the effects and origins of the modality gap and object bias in contrastive VLMs. It analyzes pretrained models and manipulates information imbalance in experiments, finding that imbalance triggers both phenomena and that the gap’s performance effect is often overshadowed by other factors.

  • Problem

    The effects and origins of the modality gap and hypothesized object bias in contrastive VLMs remained unclear, limiting understanding of their learned representations.

    Source: §1 · p. 1

  • Method

    The authors analyze pretrained VLMs, define MOAD to measure object bias, and manipulate information imbalance using synthetic and real data.

    Source: §1 · p. 2, §5 · p. 7

  • Results

    Information imbalance triggers both the modality gap and object bias, while the gap’s performance effect is typically overshadowed by other factors.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The findings can inform data filtering or caption enrichment strategies to reduce the modality gap or object bias.

    Source: §1 · p. 2

    The reported correlations are correlation-based, and some Table 3 findings lack statistical significance because comparable models are few.

    Source: G LIMITATIONS · p. 29

Abstract

from arXiv · show

Contrastive vision-language models (VLMs), like CLIP, have gained popularity for their versatile applicability to various downstream tasks. Despite their successes in some tasks, like zero-shot object recognition, they perform surprisingly poor on other tasks, like attribute recognition. Previous work has attributed these challenges to the modality gap, a separation of image and text in the shared representation space, and to a bias towards objects over other factors, such as attributes. In this analysis paper, we investigate both phenomena thoroughly. We evaluated off-the-shelf VLMs and while the gap's influence on performance is typically overshadowed by other factors, we find indications that closing the gap indeed leads to improvements. Moreover, we find that, contrary to intuition, only few embedding dimensions drive the gap and that the embedding spaces are differently organized. To allow for a clean study of object bias, we introduce a definition and a corresponding measure of it. Equipped with this tool, we find that object bias does not lead to worse performance on other concepts, such as attributes per se. However, why do both phenomena, modality gap and object bias, emerge in the first place? To answer this fundamental question and uncover some of the inner workings of contrastive VLMs, we conducted experiments that allowed us to control the amount of shared information between the modalities. These experiments revealed that the driving factor behind both the modality gap and the object bias, is an information imbalance between images and captions, and unveiled an intriguing connection between the modality gap and entropy of the logits.

1 INTRODUCTION

The paper examines how the modality gap and object bias affect contrastive VLMs, why they emerge, and whether they can be mitigated. It identifies information imbalance between images and captions as a shared trigger.

1 INTRODUCTION

  • The study asks how the modality gap affects downstream performance, whether its dimensions and embedding structures are alike, and whether the gap is a bug or a feature.Prior work attributed the gap to the cone effect at initialization and its preservation by contrastive loss.Source: §1 · pp. 1–2
  • Because object bias was previously assessed through poorer attribute-task performance rather than formally defined, the authors introduce MOAD to measure object versus attribute bias.MOAD’s generic formulation can also be used to study other biases in learned representations.Source: §1 · p. 2
  • Information imbalance between image and caption content triggers both the modality gap and object bias in contrastive VLMs.Sparse captions leave image encoders uncertain about which image information to align, favoring salient content such as object names.Source: §1 · p. 2
  • The modality gap’s effect on performance is typically overshadowed by other factors, although controlling for confounders suggests a smaller gap may correlate with better performance.The analysis also finds that only a few embedding dimensions drive the gap and that image and text embeddings have distinct characteristics.Source: §1 · p. 2
  • The paper reports that object bias does not negatively correlate with object-task performance and links object bias to per-sample caption presence bias.It also connects the modality gap to increased logit entropy, while noting that this connection is not claimed to be causal.Source: §1 · p. 2

2 RELATED WORK

Prior research documented the modality gap and hypothesized object bias, but the impact and origins of these phenomena remained open questions. This paper builds on that work while examining both effects.

2 RELATED WORK

  • Liang et al. (2022) attributed the modality gap to the cone effect at initialization and contrastive loss preserving it; later work studied temperature and embedding geometry.The authors identify information imbalance as the main factor in the gap’s emergence.Source: §2 · p. 3
  • Bravo et al. (2023) hypothesized that VLMs may be biased toward objects; this paper validates object bias and finds that VLMs with smaller object bias may not necessarily perform better on attribute tasks.Source: §2 · p. 3
  • Related work studied VLM perception, failure modes, data, robustness, learned features, compositionality, abilities, and biases, leaving learned representations an active area of analysis.The cited works include Geirhos et al. (2021), Li et al. (2023b), and Yuksekgonul et al. (2022).Source: §2 · p. 3

3 EXPERIMENTAL SETUP

The experiments combine analysis of pretrained contrastive VLMs with controlled synthetic data and caption manipulations on real data. Evaluations cover retrieval and zero-shot recognition tasks.

3 EXPERIMENTAL SETUP

  • The analysis covers 98 pretrained contrastive VLMs, including CLIP and SigLIP, trained on varied datasets; CLIP ViT-B/16 and SigLIP ViT-B/16 receive in-depth analysis.Models are grouped by medium- versus large-scale pretraining datasets.Source: §3 · p. 3
  • Evaluations use MS COCO, ImageNet, MIT-States, and UT-Zappos, with R@1 for retrieval and top-1 accuracy for zero-shot object and attribute recognition.Prompts prepend “a photo of” to descriptions, object classes, or attributes.Source: §3 · p. 3
  • MAD varies the number of attributes in captions while keeping images unchanged, enabling controlled study of information imbalance.The dataset is based on Morpho-MNIST and uses digit classes, attributes, and captions.Source: §3 · p. 3
  • For a realistic test, the authors train CLIP RN50 models on CC12M and manipulate captions.Specific settings are described in Section 6 and Appendix F.1.Source: §3 · p. 3

4 PARTING WITH FALSE INTUITIONS ABOUT THE MODALITY GAP

This section tests whether the modality gap matters for downstream performance and how it is expressed in embeddings; it finds that few dimensions dominate the gap, while simple post-hoc closure does not improve performance.

4.2 FEW EMBEDDING DIMENSIONS DRIVE THE MODALITY GAP

  • Removing the largest gap-driving dimensions closes the modality gap but sharply decreases downstream performance before a partial recovery.Ablation and re-normalization substantially change cross-modal neighborhoods, explaining the performance behavior.Source: Figure 4 · p. 5, §4.2 · p. 6

4.1 DOES A SMALLER MODALITY GAP LEAD TO BETTER PERFORMANCE?

  • Relative Modality Gap measures the gap for matching image-text pairs and accounts for the effectively used embedding space through intra-modality distances.The measure adds matching-pair distances to its denominator to scale it to [0, 1].Source: §4.1 · pp. 4–5
  • A larger modality gap correlates with better downstream performance, but controlling for confounders gives weak indications that a smaller gap correlates with better performance on some pre-training datasets.Model or embedding size can have a stronger effect on performance than the gap.Source: §4.1 · p. 5

4.2 FEW EMBEDDING DIMENSIONS DRIVE THE MODALITY GAP

  • Two embedding dimensions suffice to perfectly separate the modalities, while most dimensions have similar means across them.The dimensions with the largest mean differences are the main contributors to the modality gap.Source: §4.2 · p. 5, Figure 4 · p. 5
  • Simple post-hoc approaches can close the gap without improving performance, in part because image and text embeddings have dissimilar neighborhood orderings.The neighborhood comparison uses normalized Kendall-τ distances across three ImageNet-100 splits.Source: §4.2 · p. 6, Table 2 · p. 6
  • The paper identifies information imbalance between modalities as the main factor behind the modality gap’s emergence.Source: §4.2 · p. 6

5 OBJECT BIAS IS A CAPTION PRESENCE BIAS

The paper defines MOAD to measure object-versus-attribute bias and finds that caption presence, rather than global word frequency, explains which concepts models favor. Object bias does not clearly predict attribute performance, while object and attribute performance tend to improve together.

5 OBJECT BIAS IS A CAPTION PRESENCE BIAS

  • MOAD measures whether matching-versus-non-matching similarity differences favor objects or attributes; positive values indicate object bias, negative values attribute bias, and zero no bias.It can be computed for either image or text embeddings.Source: §5 · p. 7
  • Most contrastive VLMs show object bias, but MOAD has no clear relationship with attribute performance.Models trained on large-scale data show smaller positive MOAD values than those trained on medium-scale data.Source: §5 · p. 7
  • Object and attribute performance correlate, so general model improvements can also improve attribute performance as a by-product.The paper reports medium-to-strong correlations between performance on object and attribute tasks.Source: §5 · p. 7
  • Attributes appear more often than objects in LAION-2B captions, arguing against global word frequency as the source of object bias.The paper instead hypothesizes that objects are more likely than their attributes to appear in a caption when present in an image.Source: §5 · pp. 7–8, Figure 6 · p. 7
  • In MAD, each model favored whichever factor was made prevalent in its captions, supporting caption presence bias as the source of concept bias.The bias was larger for image encoders than text encoders.Source: Figure 6 · p. 7, §5 · p. 8

6 INFORMATION IMBALANCE TRIGGERS MODALITY GAP AND OBJECT BIAS

The paper argues that unequal information across image and text modalities underlies both modality gap and object bias, and tests this account using controlled captions and real data. It further interprets the modality gap as flexibility for adjusting logit entropy, rather than simply a bug.

6.1 HOW DOES INFORMATION IMBALANCE CAUSE THE MODALITY GAP AND OBJECT BIAS?

  • Information imbalance limits image-text alignment, leading image encoders especially to emphasize factors most likely to appear in captions, typically objects.Less likely factors may be encoded at smaller scale or in fewer dimensions to limit their effect when absent from captions.Source: §6.1 · p. 8
  • The authors hypothesize that bounded alignment shifts optimization toward cross-modal uniformity, pushing modalities apart and producing the gap.They conjecture that a few dimensions can increase uniformity substantially while having little effect on alignment.Source: §6.1 · p. 8
  • Reducing information imbalance in MAD decreases both modality gap and object bias while improving downstream performance.The gap can close during training even when initially large under full information, and image encoders are more object-biased than text encoders.Source: §6.1 · p. 9, Figure 7 · p. 9
  • With full CC12M captions, the modality gap is smaller than with captions omitting half or three-quarters of their content.The omissions increase information imbalance.Source: §6.1 · pp. 9–10

6.2 IS THE MODALITY GAP A BUG OR A FEATURE?

  • Full and half captions yield similar final validation losses, but full captions have clearly higher retrieval R@1.The authors suggest different logit entropy may explain why the loss does not reflect the retrieval difference.Source: Figure 7 · p. 9, §6.2 · p. 10
  • Embedding changes that increase the modality gap also increase logit entropy, letting the model adjust uncertainty per sample as well as globally through temperature.Temperature affects entropy globally, whereas the modality-gap route can vary it per sample.Source: §6.2 · p. 10
  • A frozen-temperature model increases the modality gap more than a trainable-temperature model to reach similar logit entropy.The result supports interpreting the gap as added flexibility for adjusting entropy.Source: Figure 8 · p. 9, §6.2 · p. 10

7 CONCLUSION

The conclusion identifies information imbalance as the trigger for both modality gap and object bias, while noting that the gap's performance association is often obscured by other factors. It also cautions that the observed gap-entropy relationship is not claimed to be causal.

7 CONCLUSION

  • Information imbalance between modalities triggers both modality gap and object bias.The imbalance limits achievable embedding alignment.Source: §7 · p. 10
  • The modality gap's performance effect is typically overshadowed by other factors, and the gap is driven by only a few embedding dimensions.The conclusion characterizes the gap as a symptom of embedding changes associated with higher logit entropy.Source: §7 · p. 10
  • The paper does not claim that changing the modality gap causes changes in entropy; the changes may only be measurable by the gap.Source: §7 · p. 10

A FURTHER DETAILS FOR THE ANALYSIS EXPERIMENTS OF PRE-TRAINED VISION-LANGUAGE MODELS

The analysis draws on a filtered set of pretrained contrastive VLMs and evaluates them on four datasets using standard retrieval or recognition protocols. The appendix also specifies the datasets, model sources, and filtering criteria.

A FURTHER DETAILS FOR THE ANALYSIS EXPERIMENTS OF PRE-TRAINED VISION-LANGUAGE MODELS

  • The analysis uses 98 of 112 OpenCLIP-provided contrastive VLMs after filtering.The filtering removes specifically fine-tuned models, keeps the latest checkpoint among same-dataset alternatives, and excludes models below 5% MS COCO R@1.Source: A FURTHER DETAILS FOR THE ANALYSIS… · p. 15
  • The considered model pool spans CLIP variants, vision backbones, and pretraining datasets.The listed model variants include OpenAI CLIP, OpenCLIP, MetaCLIP, CoCa, and SigLIP; backbones include ResNet, ConvNeXt, and ViT.Source: A FURTHER DETAILS FOR THE ANALYSIS… · p. 15
  • Evaluations used MS COCO for image-to-text retrieval and ImageNet for object recognition, applying standard retrieval protocols or top-1 accuracy; MIT-States and UT-Zappos were treated as classification tasks.For both attribute datasets, the image’s object was assumed unknown, and the task was identifying its adjective or fine-grained state.Source: A FURTHER DETAILS FOR THE ANALYSIS… · p. 15

B FURTHER DETAILS FOR THE EXPERIMENTS ON THE MULTI-MODAL ATTRIBUTES AND DIGITS DATASET

The MAD dataset varies which digit and attribute information appears in captions while retaining all factors in the images. The section also reviews the CLIP loss and limitations of L2M, motivating the Relative Modality Gap measure.

B FURTHER DETAILS FOR THE EXPERIMENTS ON THE MULTI-MODAL ATTRIBUTES AND DIGITS DATASET

  • MAD pairs Morpho-MNIST-derived images with captions for 10 digits and 16 attributes, while captions can omit attributes that remain present in the images.The attributes include thickness, swelling, fractures, scaling, and color; the experiments progressively reduce caption information.Source: B FURTHER DETAILS FOR THE EXPERIMENTS… · pp. 16–17

C BACKGROUND ON THE CLIP LOSS

  • The CLIP loss brings matching image-text pairs together through alignment while its denominator encourages uniformity, with repulsion acting across modalities rather than within them.The section gives the symmetric multi-modal version of the contrastive objective.Source: C BACKGROUND ON THE CLIP LOSS · p. 18

D.1 FURTHER DETAILS ON RELATIVE MODALITY GAP MEASURE (RMG)

  • L2M can misrepresent modality separation because it ignores effective embedding-space use, matching-pair alignment, and outlier dimensions, while L2 distance is unintuitive on the unit hypersphere.For example, equal L2M values can correspond to different actual gaps when models use different fractions of the hypersphere.Source: §D.1 · pp. 18–19
  • When controlling for dataset choice, the gap correlates negatively with performance in most cases, but the correlations often lack statistical significance.Table 3 reports Kendall’s τ correlations for RMG and L2M; its caption cautions that small per-dataset model counts make some p-values large.Source: Table 3 · p. 19
  • RMG addresses these limitations by comparing matching image-text pairs with cosine similarity and approximating density using mean within-modality distances.This approximation is used because density estimation in high-dimensional spaces is difficult.Source: §D.1 · p. 19

D.2 FURTHER ANALYSIS ON THE RELATIONSHIP OF THE MODALITY GAP AND DOWNSTREAM PERFORMANCE

The appendix examines modality-gap and performance relationships across datasets, then adds ImageNet results and examples of dimension removal. It also finds that shifting improves ideal-word/image similarity without improving performance.

D.2 FURTHER ANALYSIS ON THE RELATIONSHIP OF THE MODALITY GAP AND DOWNSTREAM PERFORMANCE

  • Negative rank correlations between gap and performance appear in seven of ten RMG cases and six of ten L2M cases, but most are not statistically significant.The authors attribute the lack of significance most likely to the small number of models per dataset.Source: §D.2 · p. 19
  • Data filtering narrows the modality gap across filtering strategies, and ImageNet results reproduce the main-text pattern of few dimensions separating modalities.Table 4 reports RMG and L2M on MS COCO and ImageNet; Figure 11 shows the ImageNet embedding-dimension and performance patterns.Source: Table 4 · p. 20, §D.2 · p. 20, §D.3 · p. 20

D.3 IMAGENET RESULTS FOR FIGURE 4

  • On ImageNet, selecting dimensions by largest mean differences also yields dimensions that separate the modalities.This holds for OpenAI CLIP ViT-B/16 and SigLIP ViT-B/16 on MS COCO and ImageNet.Source: §D.3 · p. 20, Figure 12 · p. 21

D.4 EXAMPLE FOR THE EFFECT OF THE EMBEDDING DIMENSION REMOVAL

  • A worked cosine-similarity example shows that ablating the first embedding dimension flips image-text alignment, while ablating a second dimension flips it back.The matching pair’s similarity changes from 0.995 to 0.822, while the non-matching pair changes from 0.975 to 0.917 after the first ablation.Source: §D.4 · p. 21

D.5 CLOSING THE GAP FOR IDEAL WORDS

  • The analysis extends ideal words to ideal images by replacing text embeddings with image embeddings and using object-attribute datasets to control concepts.Ideal words marginalize other factors to represent a concept direction, with the average text embedding removed to eliminate “textness.”Source: §D.5 · p. 22
  • Ideal words and ideal images are roughly aligned; correcting for the modality gap makes them well aligned, but shifting has negligible to no effect on performance, suggesting the gap does not limit pretrained contrastive VLM performance.Source: §D.5 · p. 22

E.1 FURTHER DETAILS ON OUR OBJECT ATTRIBUTE BIAS METRIC MOAD

The appendix spells out MOAD and its generalization, BRACE, then tests whether word-frequency priors explain object bias. Object bias persists even when attributes are more frequent in captions.

E.1 FURTHER DETAILS ON OUR OBJECT ATTRIBUTE BIAS METRIC MOAD

  • MOAD compares how well embeddings distinguish matching from non-matching examples for objects versus attributes, and its positive values indicate object bias.The expanded expression uses normalized image embeddings and separates within-concept similarities from similarities to examples not containing the concept.Source: §E.1 · p. 22
  • Optimizing the shift length was necessary for shifting to improve performance.The appendix contrasts this with simple shifting, which did not improve performance.Source: §E.1 · p. 22

E.2 GENERAL FORMULATION FOR BIAS MEASURE VIA RELATIVE ANGLE CHANGE OF EMBEDDINGS (BRACE)

  • BRACE generalizes MOAD from objects and attributes to any two groups of concepts, measuring their relative angle-change effects on embeddings.MOAD is presented as an instantiation of BRACE.Source: §E.2 · p. 23

E.3 ADDITIONAL EXPERIMENT ON WHERE THE OBJECT BIAS STEMS FROM

  • Object bias toward digits persists even when attributes occur more often globally, so word-prior frequency alone cannot explain the bias.The experiments vary object and attribute word probabilities, including uniform word frequency and settings where attributes are more frequent.Source: §E.3 · p. 24, Table 7 · p. 24, Table 6 · p. 24

F.1 CC12M TRAINING AND INFORMATION IMBALANCE CAUSES THE MODALITY GAP EXPERIMENTS

Experiments manipulate caption information on synthetic and real data, test whether imperfect pairings can produce a gap at the loss optimum, and compare information imbalance with the contrastive-loss account. Additional tests examine loss-driven gap changes and fine-tuning.

F.1 CC12M TRAINING AND INFORMATION IMBALANCE CAUSES THE MODALITY GAP EXPERIMENTS

  • Full captions also produce lower validation loss and higher temperature than partial captions, while the L2I gap metric misses a late RMG decrease as positive-pair distances keep increasing.The “full” and “half” validation losses are very similar by the end of training.Source: Figure 14 · p. 25, §F.1 · p. 25

F.2 FURTHER EXPERIMENTAL DETAILS FOR SECTION 6.2

  • The smaller gap with full captions generalizes to a different vision backbone, sentence-wise information dropping, and CC3M training.The CC3M dataset is not a subset of CC12M.Source: §F.1 · p. 25, Figure 15 · p. 26

F.3 CAN THE MODALITY GAP EVEN APPEAR IN THE GLOBAL OPTIMUM?

  • In a 2D toy optimization, perfect pairs have an aligned global minimum, whereas introducing mismatches produces a global minimum with a modality gap.The mismatched-pair setup is explicitly framed as a simplified illustration of bounded alignment.Source: §F.3 · pp. 26–27, Figure 16 · p. 26

F.4 IS THE GAP PRIMARILY CAUSED BY THE CONTRASTIVE LOSS INSTEAD OF THE INFORMATION IMBALANCE?

  • In image-image experiments, information imbalance produces substantially larger gaps than the small gap emerging after training from a no-gap initialization, supporting imbalance as the main driver.The comparison replicates a small post-training gap without imbalance and reports a larger imbalance effect using RMG, with similar L2M results.Source: §F.4 · pp. 27–28, Table 8 · p. 27

F.5 CAN THE CONTRASTIVE LOSS CLOSE THE MODALITY GAP?

  • Directly optimizing embeddings shows that the contrastive loss can narrow the modality gap rather than preserve it.The appendix reports this result for an 8-dimensional embedding experiment.Source: §F.5 · p. 28, Table 9 · p. 28

F.6 CAN THE MODALITY GAP BE REDUCED IN PRE-TRAINED MODELS?

  • Fine-tuning pretrained CLIP and SigLIP on dense DCI image-text pairs reduces the modality gap but slightly worsens ImageNet zero-shot transfer accuracy.The authors note that distribution shift and benchmark suitability may matter more for performance than gap size in this setting.Source: §F.6 · p. 28, Table 10 · p. 29

G LIMITATIONS

The analysis is limited by computational constraints, a potentially imprecise method for reducing information imbalance, and correlation-based findings with limited statistical significance. It also connects CLIP-like models’ information-imbalance challenges to partial multi-label learning and suggests studying possible transfers between the fields.

G LIMITATIONS

  • Experiments validated the main hypotheses on small and medium-sized datasets, but large-scale datasets such as LAION were beyond available computational resources; large-scale CLIP analysis aligned with smaller-dataset findings.Source: G LIMITATIONS · p. 29
  • The real-data imbalance reduction method keeps contiguous word sequences, but dropping half the words may remove half the information only on average, with large variance.A more refined method could produce clearer results.Source: G LIMITATIONS · p. 29
  • Table 3 findings lack statistical significance because few models are comparable, and expanding the analysis would require prohibitively large compute resources.The results in Tables 1 and 3 are correlation-based.Source: G LIMITATIONS · p. 29

H CONNECTION TO PARTIAL MULTI-LABEL LEARNING

  • CLIP-like models face information-imbalance challenges analogous to partial multi-label learning, and future work could study similarities and potentially transfer ideas between the fields.The paper identifies missing caption information and caption mismatches as examples.Source: H CONNECTION TO PARTIAL MULTI-LABEL… · p. 29
Loading 2404.07983v3…