PaperlayerView exact revision on arXiv

Source-linked AI summary

TeMo: Temperature Modulation for Multimodal Contrastive Learning

Dhimitrios Duka, Bernt Schiele, Hilde Kuehne, Anna Kukleva

arXiv:2609.07540v1cs.CV

TL;DR

Contrastive learning commonly uses a global temperature, although finer control may be useful for local semantic structure. TeMo assigns temperatures according to pair similarity and progressively combines modulated multimodal and unimodal losses with the standard objective. The paper reports improved multimodal alignment and unimodal embedding structure, while noting its experiments use constrained batch sizes and pretraining data.

  • Problem

    Most existing methods use one global learnable temperature across pairs, limiting fine-grained control over local structures and underrepresented classes.

    Source: §1 · p. 1

  • Method

    TeMo assigns similarity-dependent temperatures to individual pairs and progressively integrates modulated multimodal and unimodal losses into standard contrastive training.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    TeMo improves multimodal alignment by reducing the modality gap and improves the structure of unimodal embeddings.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The reported benefits span both multimodal alignment and unimodal embedding structure under the paper’s evaluated settings.

    Source: §1 · p. 1

    Experiments use a maximum batch size of 4096 and pretraining only on CC3M and CC12M, so results are not directly comparable to large-batch, large-data methods.

    Source: §14 · pp. 13–14

Abstract

from arXiv · show

Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart. A crucial component of contrastive learning is the temperature hyperparameter $\tau$, which controls the penalty strength applied to negative samples. However, most existing methods either fix this hyperparameter or learn a global value during training. In this paper, we introduce TeMo, Temperature Modulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity, enabling more fine-grained multimodal contrastive learning. Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them. This design allows the model to capture both coarse- and fine-grained semantics at different training stages. Extensive experiments demonstrate that each component of TeMo consistently enhances performance across diverse zero-shot retrieval and classification tasks, establishing new state-of-the-art results.

1. Introduction

A shared temperature limits local control over contrastive relationships, even though temperature shapes semantic granularity and the modality gap. TeMo assigns temperatures by pair similarity and progressively combines modulated multimodal and unimodal objectives with the standard loss.

1. Introduction

  • Global temperatures apply similar repulsion across negative samples, while finer local control may better preserve structure for underrepresented classes such as those in long-tail datasets.Temperature also shapes semantic granularity and, in multimodal learning, the modality gap.Source: §1 · p. 1

3.2. Temperature Modulation Framework

  • TeMo applies similarity-based temperatures to individual multimodal and unimodal pairs, then progressively combines those losses with standard multimodal contrastive training.This schedule moves from early instance-level detail toward coarser semantic groupings as training progresses.Source: §1 · p. 1, §3.2 · pp. 3, 5

2. Related Work

  • Prior work explores temperature schedules and pairwise adaptation; TeMo extends similarity-driven per-pair modulation to multimodal learning and combines it with self-supervision to address the modality gap.The related work describes schedules, pairwise temperature functions, alignment-aware temperatures, and temperature-free alternatives.Source: §2 · p. 2

3.1. Preliminaries

  • The method uses InfoNCE with batch-wise negatives and symmetric image-to-text and text-to-image objectives, alongside unimodal objectives for augmented views.The paired images and texts, their augmentations, and paraphrases are encoded as normalized representations for cosine-similarity comparisons.Source: §3.1 · p. 3, §3 · p. 3

3.2. Temperature Modulation Framework

  • TeMo maps rescaled pair similarity to a bounded temperature: less similar pairs receive lower temperatures, while more similar pairs receive higher temperatures.The resulting temperatures modify contrastive-logit sharpness pair by pair across image–text, image–image, and text–text relationships.Source: §3.2 · pp. 3–5
  • A quadratic schedule starts with the standard multimodal loss and shifts training toward the modulated objectives as training progresses.At t = 0 the objective is standard multimodal loss; at t = 1 it is entirely modulated loss.Source: §3.2 · p. 5

4. Experiments

Experiments compare TeMo with contrastive and temperature-adaptation baselines on zero-shot retrieval and classification, then examine its components and representation dynamics. TeMo improves retrieval and classification results across the reported settings, while ablations and analyses examine how progressive scheduling and unimodal losses contribute.

4. Experiments

  • The experiments pretrain on CC3M or CC12M and evaluate zero-shot retrieval on MSCOCO and Flickr30k and classification on CIFAR-10, CIFAR-100, ImageNet-1k, and 17 CLIP Benchmark datasets.The study compares fixed, scheduled, and adaptive temperature methods and uses ResNet-50 or ViT-B/16 vision encoders.Source: §4 · p. 5

4.1. Comparison with State of the Art

  • For zero-shot retrieval with ResNet-50, TeMo improves over InfoNCE by +1.64/+1.76% on MSCOCO and +2.04/+2.40% on Flickr30k in IR@1/TR@1.Source: §4.1 · p. 5
  • With ResNet-50, TeMo improves Top-1 accuracy over InfoNCE by +10.26/+12.44% on CIFAR-10/CIFAR-100; with ViT-B/16, the gains are +10.99/+9.09%.CWCL has higher ViT-B/16 classification accuracy but 7.38% lower Flickr30k IR@1 than TeMo, which the authors describe as a more balanced result across tasks.Source: §4.1 · p. 6, Table 1 · p. 6
  • On ImageNet-1k, TeMo reaches 34.79% Top-1 and 48.32% Top-3 on CC3M, and 41.76% Top-1 and 63.05% Top-3 on CC12M.For scratch-trained CC3M models, TeMo gains +2.23 points over its InfoNCE baseline versus SoftCLIP’s +2.00 points over its baseline.Source: §4.1 · p. 6, Table 2 · p. 6, Table 3 · p. 6

4.2. Ablations

  • Progressive scheduling outperforms fixed-weight mixing of standard and modulated multimodal losses; adding modulated unimodal losses further improves retrieval across all reported metrics.The authors attribute the schedule's benefit to emphasizing standard multimodal contrastive loss early, then increasing modulated objectives as representations become structured.Source: §4.2 · p. 7, Table 4 · p. 7
  • Across the 17 CLIP Benchmark datasets, row g achieves macro-average Top-1 accuracy of 26.98%, compared with 22.54% for the InfoNCE baseline in row a, an improvement of +4.44 percentage points.Source: §4.2 · p. 7

4.3. Representation and Training Dynamics

  • Positive- and negative-pair temperatures initially decrease and then stabilize, with positive pairs receiving higher temperatures by the end; the pattern also occurs under a fixed learning rate.The authors suggest stabilization arises from learned representation structure rather than the learning-rate schedule.Source: §4.3 · pp. 7–8, Figure 2 · p. 8
  • Unimodal objectives improve visual k-NN accuracy, while the complete TeMo objective yields the smallest modality gap; TeMo also separates positive and negative similarity distributions more clearly than InfoNCE.On CC3M validation data, InfoNCE shifts both distributions toward higher similarity, whereas TeMo shifts positives higher and negatives lower.Source: §4.3 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

5. Conclusions

TeMo combines per-pair temperature modulation across multimodal and unimodal contrastive losses with progressive blending into InfoNCE. It reports consistent improvements across retrieval, classification, and evaluation metrics.

5. Conclusions

  • TeMo adapts each training sample’s temperature to its local embedding-space neighborhood, adjusting each contrastive pair’s pushing force.The method adapts the force to the respective neighborhood.Source: §5 · p. 8
  • TeMo extends temperature modulation to unimodal contrastive losses and combines them with multimodal modulation in a unified loss formulation.The combined objective is intended to reinforce representation learning within each modality.Source: §5 · p. 8
  • Progressive blending with InfoNCE supports instance-level details at lower temperature while adaptive modulation gradually enables coarser semantic grouping.Source: §5 · p. 8
  • TeMo reports consistent improvements across MSCOCO and Flickr30K retrieval and zero-shot classification on CIFAR10, CIFAR100, and ImageNet-1k.The reported improvements span evaluation metrics as well.Source: §5 · p. 8

Supplementary Material

The supplementary material provides the full TeMo training algorithm, extended task comparisons and ablations, and additional analyses.

Supplementary Material

  • The supplement presents the full TeMo training algorithm.Source: Supplementary Material · p. 11
  • Extended main-paper tables cover retrieval and classification comparisons, loss-component ablations on both tasks, and k-NN evaluation of visual representations.Source: Supplementary Material · p. 11
  • Additional analyses examine alignment and uniformity, objective smoothness, similarity-source choice, false-positive robustness, and limitations.Source: Supplementary Material · p. 11

6. TeMo Algorithm

The TeMo algorithm combines standard multimodal contrastive losses with modulated multimodal and unimodal losses, using a quadratic schedule to form its final loss.

6. TeMo Algorithm

  • The algorithm takes image and text batches, augmented image and text batches, a baseline temperature τ0, normalized step t, minimum temperature τmin, and scaling factor τα.Source: §6 · p. 11
  • The standard multimodal loss averages image-to-text and text-to-image InfoNCE losses, each using the fixed baseline temperature τ0.Source: §6 · p. 11
  • The final loss weights the standard multimodal loss by α and the modulated multimodal and unimodal losses by β, with α=(1−t)^2 and β=t^2.The algorithm identifies modulated image-to-image and text-to-text losses, alongside modulated multimodal losses.Source: §6 · p. 11
  • It computes pairwise normalized similarity matrices between sets, with operations applied elementwise.Source: §6 · p. 11

7. Comparison to State-Of-The-Art

The extended comparisons cover zero-shot retrieval and classification metrics. TeMo is reported to deliver the strongest performance across every metric.

7. Comparison to State-Of-The-Art

  • TeMo delivers the strongest performance across every reported metric in the zero-shot retrieval and classification comparisons.Source: §7 · p. 11
  • The retrieval comparisons add R@5, R@10, and R-Mean for MSCOCO and Flickr30k.R-Mean averages image-to-text and text-to-image results.Source: §7 · p. 11
  • The classification comparisons add Top-3 and Top-5 accuracy on CIFAR-10, CIFAR-100, and ImageNet-1k.Source: §7 · p. 11

8. Ablations on Loss Components

The component ablation compares TeMo configurations across retrieval and zero-shot classification. Adding components improves performance, and the full model performs best on most reported metrics and datasets.

8. Ablations on Loss Components

  • The full model performs best on the large majority of retrieval and zero-shot classification metrics and datasets.It combines modulated multimodal and unimodal losses with the progressive scheduler.Source: §8 · p. 11
  • Adding each component improves performance across both evaluation tasks.The ablation covers retrieval and zero-shot classification.Source: §8 · p. 11
  • The ablation compares configurations combining modulated multimodal and unimodal losses with a progressive scheduler.Source: §8 · p. 11

9. k-NN Evaluation on Visual Representations

The k-NN evaluation tests whether TeMo's gains extend to visual embedding neighborhoods. Unimodal losses improve this structure, and TeMo performs best on three of four reported metrics.

9. k-NN Evaluation on Visual Representations

  • TeMo attains the best accuracy on three of the four visual k-NN metrics and remains competitive on the fourth.The evaluation extends k-NN accuracy with k-NN@10 alongside k-NN@1.Source: §9 · p. 11
  • Including unimodal losses raises CIFAR10 k-NN@1 from around 73% to roughly 80%.Every configuration with unimodal losses improves visual neighborhood structure markedly over configurations without them.Source: §9 · p. 11
  • The reported gains extend to visual-space geometry, not only cross-modal alignment.Source: §9 · p. 11

10. Effect of the Similarity Source

The similarity-source study compares external image and text expert encoders with similarities computed from TeMo's own embeddings. Expert-based variants improve over InfoNCE on most metrics, but TeMo performs better without external encoders or added computation.

10. Effect of the Similarity Source

  • Image similarities come from averaged DINOv2-Small patch features, while text similarities use all-roberta-large-v1 sentence embeddings.Source: §10 · pp. 11–12
  • TeMo outperforms all expert-similarity variants while using its own embeddings, without an external encoder or added computation.The comparison includes image-only, text-only, and combined expert similarities.Source: §10 · p. 12
  • Expert similarities improve over the InfoNCE baseline on most metrics, regardless of whether the source is image, text, or both.Source: §10 · p. 12

11. Robustness to False Positives

TeMo assigns lower scores to mismatched positive pairs and higher scores to strongly aligned pairs. Its temperature modulation balances alignment and uniformity pair by pair, with a scheduler that activates modulation after an initial low-temperature phase.

11. Robustness to False Positives

  • TeMo assigns low scores to positive image-caption pairs that do not actually match and higher scores to strongly aligned pairs.Its positive-pair scores have greater standard deviation than InfoNCE, consistent with reinforcing meaningful associations while suppressing noisy labels.Source: §11 · p. 12

12. Connection to Alignment and Uniformity

  • Dissimilar pairs receive low temperatures to promote uniformity, while similar pairs receive high temperatures to preserve semantic proximity.This is TeMo's pair-level alignment–uniformity trade-off.Source: §12 · p. 13
  • The scheduler starts with fixed low temperature, then activates per-pair modulation to refine neighborhoods and consolidate semantic clusters.Applying modulation before uniform structure is established degrades performance, supporting the coarse-to-fine ordering.Source: §12 · p. 13

13. Effect on the Smoothness of the Objective

TeMo raises temperature for high-similarity pairs, locally softening the softmax and dampening gradients. The experiments are constrained by batch size and pretraining scale.

13. Effect on the Smoothness of the Objective

  • Raising temperature for high-similarity pairs locally softens the softmax, dampens gradient magnitude, and prevents explosive updates from hard negatives.With fixed low temperature, high-similarity negatives can produce sharp gradients and steep loss-landscape regions that destabilize training.Source: §13 · p. 13

14. Limitations

  • Experiments use a maximum batch size of 4,096 and pretraining datasets CC3M and CC12M; results are competitive under these constrained settings, not directly comparable to large-batch, large-data methods.The authors state that performance is highly sensitive to both batch size and data scale.Source: §14 · pp. 13–14

15. LLM Usage

  • The authors used a large language model to improve clarity and readability, and reviewed and approved all edits.Source: §15 · p. 14
  • TeMo separates semantically aligned positive pairs from mismatched pairs in the similarity distribution.Additional examples are provided in Figure 6.Source: Figure 5 · p. 15
  • For false-positive image–caption pairs, TeMo assigns consistently lower similarity scores than InfoNCE, indicating improved discrimination and robustness to label noise.Source: Figure 6 · p. 15

Related papers

Loading 2609.07540v1…