PaperlayerView exact revision on arXiv

Source-linked AI summary

Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models

Seungmin Oh, Seunghun Kang, Jongbin Ryu

arXiv:2609.06967v1cs.CVcs.AI

TL;DR

VLM transfer can overfit target domains and mismatch augmented images with their text prompts. This paper conditions prompts on image transformations and recalibrates contrastive loss for multiple positives, reporting consistent gains across transfer and few-shot settings.

  • Problem

    VLM transfer must address target-domain overfitting under distribution shift and the mismatch between augmented images and prompts that do not describe their transformations.

    Source: §1 · pp. 1–2

  • Method

    The method conditions each text prompt on its paired image’s transformation and recalibrates contrastive loss to account for multiple valid positives.

    Source: §1 · p. 2

  • Results

    Experiments report consistent transfer-learning gains across diverse datasets, effectiveness on 11 few-shot benchmarks, and stability under extended training.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Transformation-aware conditioning and loss re-calibration are supported as directions for robust VLM transfer learning across in-distribution, out-of-distribution, and few-shot settings.

    Source: §6 · p. 14

    Constructing the augmented contrastive space requires clean and transformed data for each training image, increasing training cost.

    Source: §5 · p. 14

Abstract

from arXiv · show

Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many existing methods overlook data characteristics and simply reuse the training strategies adopted during pre-training. Specifically, they treat same-class samples as distinct instances and transform images independently of their paired text prompts, which makes model learning more difficult. We address these limitations through transformation-aware prompt conditioning and a re-calibrated contrastive loss. Fixed text descriptors identify the transformations applied to paired images, providing transformation-level consistency without altering class semantics. This design aligns the image and text branches at the transformation level, enabling richer representations while preserving the models' ability to generalize. In addition, our loss function mitigates positive-gradient dilution in soft-target cross-entropy when each anchor has multiple valid positives. During transfer, our approach treats same-class samples as positives rather than distinct instances, enabling the model to learn domain-specific features more effectively. Experiments across distribution shift, transfer learning, and few-shot settings demonstrate consistent improvements over existing approaches. Source code for our method is available at https://github.com/SoongE/ReCalCon.

1 Introduction

VLM transfer can overfit target domains and mismatch augmented images with their text prompts. The paper pairs transformation-conditioned prompts with a re-calibrated contrastive loss, reporting stronger out-of-distribution performance and improved stability during extended training.

1 Introduction

  • VLM transfer faces target-domain overfitting under distribution shift and a mismatch between augmented images and prompts that do not describe their transformations.Several baselines show a more pronounced ID-OOD trade-off as training continues.Source: §1 · pp. 1–2
  • For state-of-the-art models fine-tuned on ImageNet with CLIP ViT-B/16, Ours achieves the best ID-OOD performance, according to the reported ID and OOD accuracy comparison.Source: Figure 1 · p. 2
  • Transformation-Aware Prompt Conditioning pairs each augmented image with a fixed descriptor of its transformation, preserving class semantics while adding augmented and clean-to-augmented contrastive matches.The additional matches provide regularization during VLM transfer.Source: §1 · p. 2, Figure 2 · p. 3
  • Re-calibrated Contrastive Loss addresses positive-gradient dilution by adding an adaptive positive-versus-negative evidence margin when anchors have multiple valid positives.The method reports improved out-of-distribution performance and less overfitting during longer training.Source: §1 · p. 2

1.1 Contributions

  • The framework jointly optimizes clean, transformed, and cross-paired matches, and combines transformation-aware conditioning with a loss requiring only a few lines of code.It is designed to integrate existing visual augmentation techniques.Source: §1.1 · p. 3
  • Where several existing transfer baselines degrade with extended training, the proposed regularization allows performance to continue improving.The discussion also reports greater stability on OOD datasets as training duration increases.Source: §1.1 · p. 3

2 Related Work

Related work covers VLM transfer strategies, methods for handling distribution shift and parameter efficiency, and extensions to cross-entropy objectives for multiple positives.

2.1 Transferring Vision-Language Models

  • VLM transfer adapts pretrained models to downstream tasks; approaches include prompt tuning, adapters, and full fine-tuning, while Figure 3 contrasts their positive-sampling objectives.Prompt tuning is used for parameter efficiency, and adapters for few-shot learning with limited samples.Source: §2.1 · p. 3, Figure 3 · p. 4

2.2 Distribution Shift

  • For distribution shift, WiseFT ensembles weights, LP-FT stages linear probing before full fine-tuning, and FLYP updates both encoders with a CLIP-style contrastive objective.Lipsum-FT adds random text with a frozen text encoder; RAda-FT learns an attention mask, and ARF uses large-language-model-generated prompt context.Source: §2.2 · p. 4

2.3 Parameter-Efficient Transfer Learning

  • Parameter-efficient transfer includes adapters, which add task-specific modules, and prompt tuning, which learns small vectors from limited samples.Prior-based methods use a cache built from few-shot training samples to separate specific knowledge from the general cache.Source: §2.3 · p. 4

2.4 Advances in Cross-Entropy Loss

  • Related cross-entropy work reweights positives and negatives for class imbalance, regularizes gradients to limit early saturation, and extends contrastive objectives to multiple same-class positives.The cited approaches are Lin et al. [30], Chen et al. [6], and Khosla et al., respectively.Source: §2.4 · pp. 4–5

3 Method

The method conditions text prompts on image transformations while preserving class semantics, then uses a re-calibrated contrastive objective and multi-sample feature fusion. These components adapt image-text matching to augmented and same-class samples.

3.1 Transformation-Aware Prompt Conditioning

  • Transformation-aware prompts combine learnable context, a fixed descriptor of the image augmentation, and the class token, with only the context optimized.A pool of image augmentations is sampled to apply combined transformations.Source: §3.1 · p. 5

3.2 Re-calibrated Contrastive Loss

  • The re-calibrated loss extends soft-target cross-entropy to multiple same-class positives and adds an adaptive margin based on positive versus negative evidence.The margin is not divided by the number of positives, so its strength is retained as positive count increases.Source: §3.2 · pp. 6–7

3.3 Multi-sample Fusion

  • Multi-sample fusion combines image and text features before projection, avoiding prompt-level merging of distinct class words.The fusion ratio is randomly chosen, and the resulting projected features are used to compute the re-calibrated contrastive loss.Source: §3.3 · pp. 7–8

4 Experiments

Experiments evaluate the method under distribution shift, extended training, standard transfer, and few-shot learning, alongside ablations of its conditioning and loss components.

4.1 Distribution Shift

  • 0.9% higher OOD performance than the previous state of the art, with the highest ID accuracy at 83.3%; harmonic-mean gains are 0.7% on ImageNet-based datasets and 1.4% on iWildCam.The comparison covers backbone-update and frozen methods on ImageNet and iWildCam.Source: §4.1 · pp. 9–10, Table 1 · p. 9
  • With extended training, our method improves ID performance and harmonic mean while several existing methods decline on both measures.Figure 4 reports training and evaluation losses in panels a–b, and ID accuracy and harmonic mean in panels c–d; our method also consistently reduces both losses.Source: §4.1 · p. 10, Figure 4 · p. 10

4.2 Transfer Learning

  • 0.8% higher average performance than the previous state of the art across six downstream transfer datasets, with the best performance on most datasets.The six datasets include Caltech101, PCam, ImageNet, Flowers102, StanfordCars, and iWildCam.Source: §4.2 · p. 10, Table 2 · p. 11

4.3 Few-shot Learning

  • 0.8% higher performance than the previous state of the art in both the 1-shot and 16-shot settings, with strong results across diverse fine-grained datasets.The experiments use a frozen backbone and compare methods across 11 benchmarks and five shot counts.Source: §4.3 · pp. 11–12, Figure 5 · p. 11

4.4 Ablation Study

  • The ablation study examines transformation-aware prompt conditioning, augmented contrastive-space construction, re-calibrated loss, and component contributions.It includes analyses of the loss and multi-sample fusion.Source: §4.4 · p. 12

4.4.1 Transformation-Aware Prompt Conditioning

  • Under RandAug, the correct transformation descriptor reaches 62.4% harmonic mean, versus 61.6% without a descriptor and 57.3% with a mismatched descriptor.The comparison tests fixed, misaligned, and aligned text prompts against the image augmentation.Source: §4.4.1 · p. 12, Table 3 · p. 12

4.4.2 Re-calibrated Contrastive Loss and Space

  • On ViT-B/32, expanding the contrastive space improves performance by approximately 1% versus a single space, while re-calibration adds approximately 1% across configurations; together they yield approximately 2%.The baseline uses only image-text matches without adaptive re-calibration and achieves 77.2% accuracy on the ID dataset and 60.4% harmonic mean.Source: §4.4.2 · p. 13, Table 4 · p. 13

4.4.3 Objective and Component Isolation

  • Adding re-calibration and multi-sample fusion to the SoftCE baseline improves ID accuracy by 1.0% and harmonic mean by 0.6%.All configurations already include transformation-aware prompt conditioning and the augmented contrastive space.Source: §4.4.3 · pp. 13–14, Table 5 · p. 13

5 Discussion

The discussion presents augmentation as regularization that reduces overfitting and helps maintain OOD stability during extended training. It also weighs increased training cost against unchanged inference cost and notes compatibility with PETL and few-shot settings.

5 Discussion

  • The method uses image augmentation as regularization to improve transfer learning and reduce overfitting, particularly during extended training.Previous methods show degraded OOD performance as training duration increases, while this method remains more stable.Source: §5 · p. 14
  • Training requires clean and transformed data from each image, increasing cost, while inference cost remains identical to the backbone.Relative to FLYP, per-step runtime and peak memory are reported as 1.0× at batch size 8 and 1.6× at batch size 64, respectively.Source: §5 · p. 14
  • Compatibility with parameter-efficient transfer learning and few-shot scenarios supports adaptation settings where labeled data are limited.The authors propose predefined augmentation combinations from the wider space of possible image and text transformations as a future direction.Source: §5 · p. 14

6 Conclusion

The paper concludes that transformation-aware conditioning and re-calibrated contrastive loss form a practical VLM transfer-learning framework. Experiments report improved transfer performance across downstream datasets, few-shot benchmarks, and extended training.

6 Conclusion

  • Transformation-aware conditioning preserves transformation-level consistency, while the augmented contrastive space supports clean, transformed, and cross-paired image-text matches.The re-calibrated loss mitigates positive-gradient dilution as the number of positives increases; multi-sample fusion adds regularization without prompt-level ambiguity.Source: §6 · p. 14
  • The method mitigates overfitting and consistently improves transfer-learning performance across diverse downstream datasets.Experiments also report effective performance on 11 few-shot benchmarks and stability under extended training.Source: §6 · p. 14
  • Experiments on ImageNet and iWildCam demonstrate transferability in both in-distribution and out-of-distribution settings.Source: §6 · p. 14

A Experiment Details

The appendix identifies hyperparameter tables for transfer learning, distribution shift, and PETL experiments. It also reports task-specific settings and compute requirements.

A Experiment Details

  • Tables 6–9 cover default, transfer-learning, and distribution-shift hyperparameters, including separate ImageNet and iWildCam settings.Table 6 lists defaults, Table 7 transfer-learning settings, and Tables 8–9 distribution-shift settings for ImageNet and iWildCam.Source: A Experiment Details · p. 19, Table 6 · p. 19, Table 7 · p. 19
  • Few-shot classification follows APE, with learning rate and weight decay set differently for each dataset; PETL updates a few parameters and uses a large batch size.Source: A Experiment Details · p. 19
  • Transfer learning takes approximately 6 hours on four RTX 3090 GPUs, while the other tasks use a single GPU.The reported software versions are PyTorch 2.2.1+cu121 and Python 3.10.Source: A Experiment Details · p. 19

B Additional Diagnostics and Ablations

The diagnostics report stronger gradients than SoftCE as positive counts increase, finite loss and gradients across random batches, and bounded training overhead. Performance is stable for two or three sampled transformations.

B Additional Diagnostics and Ablations

C Additional Results

Additional results show overfitting under extended training with ViT-B/16 on ImageNet and its distribution-shifted variants.

C Additional Results

Related papers

Loading 2609.06967v1…