PaperlayerView exact revision on arXiv

Source-linked AI summary

When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP

Shota Sato, Hajime Kiyama, Tosho Hirasawa, Mamoru Komachi

arXiv:2609.01103v1cs.CLcs.CV

TL;DR

The paper asks why reducing CLIP’s average image–text gap does not consistently improve zero-shot accuracy. It analyzes class-wise decision margins and Linear correction, then tests prediction concentration across correction methods. The results associate accuracy degradation with prediction-level hubness and motivate evaluating downstream prediction structure alongside average alignment.

  • Problem

    Reducing the average modality gap does not necessarily improve CLIP zero-shot accuracy because zero-shot decisions also depend on class-wise margins.

  • Method

    The paper analyzes the mismatch through zero-shot prediction structure, using Linear correction as an analytically tractable case and comparing correction methods across datasets.

  • Results

    Accuracy degradation under Linear and learning-based corrections is associated with increased prediction concentration, while Linear correction can produce unequal class-wise score shifts.

  • Takeaways & Limitations

    Gap correction should be evaluated not only by average image–text alignment but also by its impact on downstream prediction structure.

  • Takeaways & Limitations

    AlignCLIP is supporting evidence rather than part of the controlled comparison because its public checkpoint uses a different backbone and fewer comparison points.

Abstract

from arXiv · show

Reducing the modality gap between image and text representations in CLIP is widely expected to improve cross-modal alignment and downstream performance. However, a smaller average image-text gap does not necessarily lead to consistent accuracy gains. We analyze this mismatch from the perspective of the decision structure in zero-shot classification, i.e. selecting the most similar class-text prototype for an input image. Zero-shot accuracy depends not only on average image--text alignment, but also on class-wise decision margins. Using Linear correction as an analytically tractable case, we show that modality gap correction can alter the relative decision structure among classes and cause predictions to concentrate on a small subset of classes. We refer to this output-space failure mode as prediction-level hubness. Furthermore, experiments across multiple datasets show that accuracy degradation under gap correction is consistently associated with increased prediction concentration, both for Linear correction and for learning-based correction methods. This provides a systematic explanation of why modality gap reduction does not consistently improve CLIP zero-shot accuracy from the perspective of downstream decision structure. Our results suggest that gap correction should be evaluated not only by average alignment, but also by its impact on downstream prediction structure.

1 Introduction

The paper argues that reducing CLIP’s average modality gap does not reliably improve zero-shot accuracy because correction can alter class-wise decision structure and concentrate predictions on a few classes.

  • CLIP uses a shared image–text embedding space but exhibits modality-specific clusters that motivate gap-reduction methods.
  • Reducing the modality gap does not necessarily improve downstream accuracy because zero-shot performance also depends on class-wise decision margins.
  • Linear correction monotonically reduces the modality gap across 10 zero-shot image classification datasets, yet accuracy eventually deteriorates.
  • Unequal class-wise score shifts can favor certain prototypes and cause corrected predictions to concentrate on a small subset of classes.
  • Across correction methods, increased prediction concentration is associated with accuracy degradation, while the paper recommends evaluating downstream prediction structure alongside average alignment.

2 Background and Related Work

The background reviews CLIP’s modality gap, methods for reducing it, and prior hubness work, while distinguishing the paper’s focus on final prediction concentration from embedding-space nearest-neighbor hubness.

  • CLIP enables zero-shot image classification by comparing image embeddings with class-text prototypes in a shared embedding space.
  • The modality gap refers to image and text embeddings forming modality-specific clusters despite CLIP’s joint embedding framework.
  • Gap-reduction methods include post-hoc geometric or statistical transformations and additional training, regularization, or adversarial objectives.
  • Prior work suggests that reducing the modality gap does not always improve downstream performance, motivating broader comparison across datasets and correction methods.
  • Unlike classical hubness based on nearest-neighbor relations, this paper studies concentration in the final zero-shot predicted-class distribution.

3 Problem Setup

The problem setup tests whether making image and text representations closer on average improves zero-shot classification, comparing correction profiles through the relationship between modality gap and accuracy.

  • Zero-shot classification exposes both average image–text alignment and class-level prediction behavior, making it the evaluation setting for this mismatch.
  • 3.1 Average Modality Gap: The average modality gap is the squared Euclidean distance between the mean image embedding and the mean text embedding.
  • 3.1 Average Modality Gap: A smaller average gap does not guarantee improved decision structure because changing relative scores among class-text prototypes can hurt accuracy.
  • 3.2 Gap Reduction and Accuracy: The evaluation examines Linear correction, CLIPRefine, and AlignCLIP as representative correction profiles.
  • 3.2 Gap Reduction and Accuracy: Under Linear correction, the modality gap decreases monotonically while accuracy first improves and then deteriorates; learning-based corrections reduce the gap while maintaining or improving accuracy.

4 Mechanism: Gap-Induced Class-wise Bias and Prediction-Level Hubness

The mechanism analysis shows that gap correction can change class-wise margins through unequal score shifts, producing prediction hubs and accuracy loss, with the explicit bias decomposition applying specifically to Linear correction.

  • 4 Mechanism: Gap-Induced Class-wise Bias and Prediction-Level Hubness: Average geometric gap reduction can fail because classification accuracy depends on class-wise decision margins.
  • The general analysis covers corrections that shift zero-shot scores, whereas the mechanistic analysis of explicit bias is specific to Linear correction.
  • 4.1 Accuracy Depends on Class-wise Margins: A corrected prediction can switch from the ground-truth class to a competitor when the competitor receives a larger relative score shift.
  • 4.2 Linear Correction and Class-wise Bias: For Linear correction, the gap vector induces a class-wise score bias that can change class rankings and favor particular prototypes.
  • 4.2 Linear Correction and Class-wise Bias: At α = 0.5, gap-induced bias positively correlates with prediction counts, prediction-count gains, and correct-to-wrong transition destinations across datasets.

5 Experiments: Prediction-Level Hubness and Accuracy Degradation

Across ten zero-shot image-classification datasets, the experiments test whether accuracy degradation under modality-gap correction is associated with prediction-level hubness. Results link excessive correction, concentrated predictions, harmful destination classes, and a Linear-specific bias mechanism, while noting important evaluation and mitigation boundaries.

  • 5.1 Experimental Setup and Metrics: The evaluation covers ten datasets and three correction profiles, with Linear correction controlled post hoc, CLIPRefine learning-based, and AlignCLIP supplementary because it uses a different backbone and fewer points.Predictions select the text prototype with highest similarity; accuracy and prediction concentration are reported.
  • 5.1 Experimental Setup and Metrics: Predicted-Class Gini measures concentration in predicted-class counts: larger values indicate that predictions concentrate on fewer classes.Classes with large predicted counts are treated as prediction hubs.
  • 5.2 Prediction Concentration is Associated with Accuracy Degradation: For Linear correction, moderate correction often reduces concentration and improves accuracy, whereas stronger correction increases concentration and reduces accuracy; CLIPRefine shows the same weaker trend.The over-correction regime combines a smaller average modality gap with predictions collapsing toward fewer classes.
  • 5.2 Prediction Concentration is Associated with Accuracy Degradation: All 10 datasets show the expected negative within-dataset correlation for Linear correction, while CLIPRefine shows the same direction in most datasets.The concentration–accuracy relationship is therefore not only a pooled cross-dataset artifact.
  • 5.2 Prediction Concentration is Associated with Accuracy Degradation: At fixed non-zero correction strength, datasets with greater concentration changes tend to lose more accuracy, with mean Spearman correlation −0.86 for gap-reducing α.This relationship is not redundant with correction strength itself; 10 of 20 points reach p < 0.05.
  • 5.3 Transition-Level Evidence for Prediction Hubs: In Linear-worst, correct-to-wrong transitions reach 33,804 and the top five destination classes absorb 67.2% on average, whereas CLIPRefine-worst has a top-five fraction of 0.388.These transitions indicate that previously correct samples are redirected into a small set of erroneous classes.
  • 5.4 Intervening on Gap-Induced Bias: Subtracting the gap-induced class-wise bias consistently lowers Predicted-Class Gini and usually partially recovers accuracy lost under Linear over-correction.This intervention provides mechanistic evidence for hub formation, but its scope is Linear-specific.
  • 5.5 Interpreting and Mitigating Prediction Concentration: Predicted-Class Gini must be interpreted relative to label imbalance, and no universal absolute-Gini threshold is proposed.The recommended reporting set includes modality gap, accuracy, Predicted-Class Gini, normalized prediction entropy, and label Gini relative to the uncorrected model.

6 Discussion

Gap reduction is useful only when it preserves downstream decision structure. Linear correction can distort class-wise scores and concentrate predictions, whereas CLIPRefine reduces the gap without inducing this concentration.

  • What Makes Gap Reduction Useful?: Linear correction can reduce average image–text displacement while distorting downstream decision geometry through uneven class-wise score shifts.
  • What Makes Gap Reduction Useful?: Favoring a small subset of class prototypes creates prediction hubs and is associated with deteriorating accuracy.
  • What Makes Gap Reduction Useful?: CLIPRefine reduces the modality gap without inducing prediction concentration, suggesting learned correction can preserve decision structure that Linear correction distorts.
  • A Candidate Geometric Correlate: Prototype angular structure does not separate the methods; joint image–text uniformity instead tracks accuracy and Predicted-Class Gini under Linear correction.
  • Is the Average Modality Gap the Right Quantity?: The modality-gap metric is squared Euclidean distance between mean embeddings, whereas zero-shot prediction uses cosine similarity between normalized embeddings.
  • What Makes Gap Reduction Useful?: Gap correction should be evaluated with output-space diagnostics such as predicted-class concentration, not only alignment metrics.

7 Conclusion

The paper explains inconsistent zero-shot gains from modality-gap reduction through changes in downstream decision structure. Across datasets and correction profiles, accuracy degradation is associated with prediction-level hubness, while bias subtraction reduces concentration under Linear over-correction.

  • Gap correction can unevenly change class-wise decision margins and induce prediction-level hubness, concentrating predictions on a small subset of class prototypes.
  • Across correction profiles and datasets, accuracy degradation is associated with increased prediction-level hubness.
  • A score-space intervention on the gap-induced bias reduces prediction concentration under Linear over-correction.
  • Modality-gap correction should be assessed by downstream decision structure and output distributions as well as average image–text alignment.

Limitations

The claims are scoped primarily to fixed-prototype zero-shot image classification, with controlled comparisons centered on one backbone. Mechanistic evidence is stronger for Linear correction than for training-based correction profiles.

  • The claims target zero-shot image classification with a fixed set of class-text prototypes and do not claim transfer to cross-modal retrieval.
  • Prediction-level hubness is defined over classifier outputs, distinct from conventional nearest-neighbor hubness in embedding space.
  • Mechanistic evidence is closed-form and interventional for Linear correction, but prediction concentration is diagnostic rather than causal for CLIPRefine.
  • AlignCLIP uses fewer comparison points and a different backbone, so it is treated as supplementary rather than part of the controlled comparison.
  • Controlled results use ViT-B/32; broader-backbone evidence covers the Linear-correction sweep, while training-based profiles are available at only one scale.
  • Evaluation uses standard datasets and public artifacts for aggregate zero-shot evaluation, without collecting or redistributing original images or labels.

A.3 Linear Correction Implementation

The appendix describes the correction, scoring, gap measurement, uniformity analysis, model profiles, datasets, prompts, and evaluation implementation used throughout the experiments.

  • Linear correction follows an embedding-shift formulation that moves image and text embeddings in opposite directions by strength α before L2 normalization.
  • The predicted class is selected by maximizing cosine similarity between normalized corrected image embeddings and text prototypes.
  • The correction sweep varies α from −0.5 to 0.5 in steps of 0.05 while keeping pretrained CLIP encoders fixed.
  • The modality gap is measured as squared Euclidean distance between mean image and text embeddings across correction strengths or learning-based checkpoints.
  • Uniformity is computed on a joint set of corrected normalized image embeddings and ground-truth text prototypes, with subsampling to 2,000 points when larger.
  • The evaluation includes CLIPRefine and AlignCLIP alongside Linear correction, using ViT-B/32 for the former profiles and ViT-B/16 for AlignCLIP.
  • Experiments use public implementations, standard dataset interfaces, dataset-specific prompt templates, and batched PyTorch evaluation with batch_size=256.
  • The pipeline computes Predicted-Class Gini, normalized prediction entropy, transition counts, and correlations using custom code and standard statistical routines.

B Prediction-Level Hubness Metrics and Complementary Results

The paper measures prediction-level hubness through predicted-class concentration and transition destinations. Across correction methods, greater concentration and lower prediction diversity are associated with lower zero-shot accuracy, while transition heads reveal redirection toward a few classes.

  • Metrics: Predicted-class counts identify hubs, while Predicted-Class Gini measures concentration across the class-count vector.Higher Gini means predictions are concentrated on fewer classes.
  • Complementary results: Higher normalized prediction entropy is associated with higher accuracy for Linear correction and CLIPRefine (Pearson’s r = 0.716, n = 210; r = 0.585, n = 310).The result indicates reduced diversity in predicted classes accompanies accuracy degradation.
  • Metrics: Normalized prediction entropy complements Gini: higher entropy indicates a more even distribution of predictions across classes.The expected relationship is positive between entropy and accuracy when concentration accompanies degradation.
  • Complementary results: For Linear correction, all 10 datasets show positive within-dataset entropy–accuracy correlations; CLIPRefine shows the expected direction in 9 of 10 datasets.The average correlation is weaker for CLIPRefine than for Linear correction.
  • Transition-level evidence: Correct-to-wrong transitions expose hub formation when previously correct samples are redirected toward a small set of erroneous destination classes.Destination classes are ranked independently by transition count, so curves compare concentration patterns rather than class identities.
  • Cross-method evidence: AlignCLIP follows the same qualitative pattern—higher concentration corresponds to lower accuracy and higher entropy to higher accuracy—but its limited comparison points require caution.The supporting analysis reports a negative Gini correlation and a positive entropy correlation.

C.3 Additional Results for Intervening on Gap-Induced Bias

Additional analyses test whether bias subtraction supports the proposed hubness mechanism and whether the findings depend on prompts or transition profiles. Positive-direction subtraction generally improves accuracy and reduces concentration, while richer prompts do not remove the failure mode.

  • Bias intervention: Bias subtraction’s effect depends on both Linear correction strength α and subtraction strength λ.Values are evaluated relative to the Linear-correction baseline at the same α.
  • Bias intervention: In the positive over-correction regime, positive λ generally improves accuracy and reduces Predicted-Class Gini, supporting gap-induced class-wise bias as a contributor to hubness.A mismatched direction, such as negative λ, can instead reduce accuracy or increase concentration.
  • Bias intervention: The intervention is a diagnostic test of the proposed mechanism rather than a general-purpose correction method.This follows from the contrasting effects of matched and mismatched subtraction directions.
  • Prompt robustness: Under a generic prompt configuration, the correlation structure matches the main results across datasets, including the same signs for Gini and entropy relationships.The analysis uses an independent run with re-extracted embeddings.
  • Scope: The prompt-mechanism test is limited to a single dataset, CIFAR-100, so it does not establish generality across datasets.The derivation predicts that richer descriptions may change which classes become hubs without preventing hub formation.
  • Prompt robustness: On CIFAR-100, Linear over-correction decreases accuracy and roughly doubles Predicted-Class Gini under all three richer-prompt schemes.The result includes both mean-prototype scoring and the evaluation rule averaging per-description probabilities.

D.3 Centering Ablation

Centering removes the residual gap direction and largely eliminates correction-strength dependence in accuracy and prediction concentration. CSLS reduces, but does not eliminate, over-correction penalties and preserves the concentration–accuracy relationship across scales and datasets.

  • Centering ablation: Centering removes the residual gap direction and its class-wise bias, predicting approximate independence of accuracy and concentration from α.The procedure shifts and normalizes modalities, subtracts each modality mean, and renormalizes.
  • Centering ablation: Accuracy-range variation falls from 18.5 points on average (maximum 28.6) to at most 0.7 points, while Gini-range variation falls from 0.30 on average (maximum 0.53) to at most 0.009.Table 10 reports these ranges across α ∈ [−0.5, 0.5] on every dataset.
  • Centering ablation: Centering is a diagnostic rather than a practical improvement because its effect on uncorrected accuracy is dataset-dependent, ranging from −4.2 points to +4.1 points and reaching +15.3 points under one generic-template condition.The reported range is from Flowers102 to EuroSAT under dataset-specific templates.
  • CSLS scoring: With CSLS, the mean accuracy change at α = 0.5 is −7.4 points versus −15.1 points under cosine scoring, so CSLS roughly halves the over-correction penalty.CSLS partially mitigates but does not eliminate degradation as the modality gap continues to decrease monotonically.
  • CSLS scoring: Under CSLS, higher Predicted-Class Gini remains associated with lower accuracy, and Linear-CSLS shows the expected correlation direction in all 10 datasets.Normalized entropy retains the complementary positive relationship with accuracy.
  • Model scale: On ViT-L/14, accuracy–Gini correlation is negative on all 10 datasets with mean −0.920, while α = 0.5 costs 10.5 accuracy points on average.The pooled Pearson correlation is −0.815, and ImageNet-1K loses 22.0 points.
  • Model scale: The EuroSAT exception on ViT-B/16 reflects a narrow Gini range and unstable rank correlation, while the same dataset returns to −0.930 on ViT-L/14.The differently oriented gap changes bias magnitude but not the reported failure structure.
Loading 2609.01103v1…