PaperlayerView exact revision on arXiv

Source-linked AI summary

TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis, Kaifeng Chen, Arjun Karpur, Ye Xia, Sahil Dua, Tanmaya Dabral, Guangxing Han, Bohyung Han, Joshua Ainslie, Alex Bewley, Mithun Jacob, René Wagner, Washington Ramos, Krzysztof Choromanski, Mojtaba Seyedhosseini, Howard Zhou, André Araujo

arXiv:2604.12012v1cs.CV

TL;DR

Vision-language encoders remain limited in aligning dense patch representations with corresponding text concepts, despite strong global and downstream capabilities. TIPSv2 addresses this gap with patch-level distillation, iBOT++, multi-granularity captions, and head-only EMA, achieving strong results across 9 tasks and 20 datasets that generally match or exceed prior vision-language encoders.

  • Problem

    Existing vision-language models struggle to maintain precise patch-level alignment between visual representations and text concepts, a capability needed for tasks such as open-vocabulary segmentation.

  • Method

    TIPSv2 combines patch-level distillation findings with iBOT++ visible-token supervision, multi-granularity synthetic captions, and a resource-efficient head-only EMA recipe.

  • Results

    TIPSv2 generally matches or exceeds previous vision-language encoders across 9 tasks and 20 datasets, including a +14.1 mIoU gain from replacing iBOT with iBOT++ in zero-shot segmentation.

  • Takeaways & Limitations

    The resulting image-text encoders provide strong global, dense, and image-only performance while improving dense patch-text alignment through the revised pretraining recipe.

Abstract

from arXiv · show

Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment -- surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/ .

1. Introduction

TIPSv2 addresses the persistent difficulty of combining global image-text understanding with precise patch-level alignment. It introduces iBOT++, multi-granularity captions, and head-only EMA to improve alignment, efficiency, and downstream performance.

  • Unified vision-language models still struggle to align individual patches with corresponding text concepts, limiting capabilities such as open-vocabulary segmentation.
  • Patch-level distillation improves grounding alignment by supervising all patch tokens, with smaller distilled students sometimes outperforming larger teacher models.
  • TIPSv2 introduces iBOT++, which enforces representation consistency on visible tokens during self-supervised pretraining.
  • Multi-granularity captions from PaliGemma and Gemini, together with a head-only EMA that updates only projection layers, improve training effectiveness while reducing memory requirements.
  • The resulting recipe is evaluated across 9 tasks and 20 datasets, including strong downstream performance and new state-of-the-art zero-shot semantic segmentation results.

2. Related Work

The paper builds on self-supervised and image-text contrastive pretraining to produce representations that are both text-aligned and spatially aware. Its main advances target visible-token supervision, efficient EMA, and spatially enhanced distillation.

  • Self-supervised learning provides label-free visual pretraining, while image-text methods support text-aligned representations; TIPSv2 combines these directions.
  • iBOT++ extends masked image modeling by applying patch-level supervision to both visible and masked tokens, improving patch-text alignment and downstream tasks.
  • A head-only EMA updates projection heads instead of the full model, reducing trainable parameters by nearly half in the combined self-supervised and contrastive setup.
  • Spatially targeted distillation can enhance patch-text alignment even when the teacher is a large pretrained vision-language encoder.

3. TIPSv2

TIPSv2 combines contrastive and self-supervised pretraining with distillation-informed changes to improve patch-text alignment while reducing training costs. Its central recipe adds visible-token supervision through iBOT++, uses head-only EMA, and samples captions at multiple granularities.

  • 3.2. Bridging Pretraining and Distillation: Distillation removes patch masking and uses a frozen larger teacher, exposing the student to teacher representations across all patch tokens.The ablations further identify random visual initialization as critical for strong patch-text alignment.
  • 3.1. Preliminaries: Patch-level distillation significantly enhances patch-text alignment, with the distilled TIPS ViT-L student surpassing its larger ViT-g teacher in zero-shot segmentation.The study evaluates alignment using zero-shot segmentation benchmarks including ADE150, Pascal Context, and Pascal VOC.
  • 3.3. iBOT++: iBOT++ extends iBOT by applying patch-level loss to both masked and visible tokens, anchoring visible-token representations while retaining masked-image modeling.Unlike standard iBOT, iBOT++ directly supervises unmasked tokens, which improves their patch-level loss and preserves local semantics.
  • 3.4. Head-only EMA: TIPSv2 combines iBOT++ with head-only EMA and multi-granularity captions in a joint contrastive and self-supervised pretraining recipe.Head-only EMA reduces training parameters by 42% on ViT-B, while captions from PaliGemma and Gemini provide varied textual descriptions.
  • 3.4. Head-only EMA: The resulting recipe improves zero-shot segmentation in pretraining, with iBOT++ showing significant gains over iBOT.The method is designed to combine global context learning with stronger local patch-text alignment.

4. Results

TIPSv2 is evaluated frozen, off-the-shelf across 9 tasks and 20 datasets, spanning dense/global image-text alignment and image-only understanding. It generally matches or exceeds competing encoders, with especially strong dense and broad image-only performance but weaker ImageNet classification.

  • Evaluation protocol: TIPSv2 evaluates frozen pretrained representations on 9 tasks spanning 20 datasets, covering dense/global image-text alignment and image-only understanding.Image-text evaluations include retrieval, zero-shot classification, and dense patch-text alignment; image-only evaluations include segmentation, depth, and surface normals.
  • Dense image-text tasks: TIPSv2 outperforms other methods in all dense image-text evaluations, despite some competitors using the more expensive TCL protocol.The dense evaluations measure patch-text alignment through segmentation-related tasks.
  • Ablations: The cumulative ablation adds iBOT++, multi-granularity captions, and head-only EMA, with increased or comparable performance across evaluations.iBOT++ produces a dramatic +14.1 mIoU improvement in zero-shot segmentation; caption sampling improves global and dense image-text tasks, while head-only EMA primarily reduces training resources.
  • Global and image-only tasks: TIPSv2 achieves the best or second-best results in 5 of 7 global image-text evaluations and 7 of 9 image-only evaluations.Its ViT-g models outperform larger PE ViT-G models in 3 of 5 reported global image-text evaluations.
  • Additional comparisons: TIPSv2 surpasses DINOv3 on 4 of 6 metrics, improves PASCAL segmentation by +1.5, and reduces NYUv2 depth error by −0.019 relative to prior best results.The models are less effective on ImageNet classification, consistent with the stated focus on broad general-purpose capability.

5. Conclusion

The paper introduces TIPSv2 to address weak dense image-text alignment in vision-language encoders. Its iBOT++ objective, efficient EMA setup, and multi-granularity captions yield strong results across 9 tasks and 20 datasets.

  • Core finding: Patch-level distillation substantially enhances dense patch-text alignment and motivates the iBOT++ pretraining objective.iBOT++ upgrades masked image modeling to improve dense image-text alignment.
  • Method: TIPSv2 combines iBOT++ with simplified EMA updates and captions at different granularities to obtain more efficient, robust representations.The conclusion identifies these components as the main recipe changes beyond the original TIPS framework.
  • Overall outcome: TIPSv2 achieves strong performance on 9 tasks spanning 20 datasets, generally matching or exceeding previous vision-language encoders.The model is presented as suitable for a variety of multimodal applications.
  • Generality: Applying iBOT++ to CLIP significantly enhances performance across several tasks beyond standard iBOT.This supports the usefulness of the objective beyond the TIPSv2 architecture.

A.1. Applying iBOT++ to CLIP

The appendix tests whether iBOT++ transfers to vanilla CLIP and examines masking in iBOT++ pretraining. The objective improves CLIP across several tasks, while 75% masking gives the strongest overall results.

  • Applying iBOT++ to CLIP: Integrating iBOT++ into vanilla CLIP significantly enhances performance across several tasks beyond what standard iBOT achieves.The experiment uses a ViT-L backbone and the same dataset used throughout the paper.
  • Masking ratio: A 75% masking ratio is critical for strong performance across evaluations, particularly for patch-text alignment.The best overall model is identified using the masking-ratio ablation on TIPS ViT-L models.
  • Patch-text alignment: The improvement in zero-shot segmentation highlights iBOT++’s importance for aligning image patches with language.The appendix presents this result as corroborating the usefulness of the proposed recipe for vision-language pretraining.

A.2. iBOT++ Ablation Study on Masking Ratios

The masking ablation distinguishes the pretraining design choices underlying iBOT++ and motivates applying visible-token supervision during pretraining.

  • Objective design: iBOT++ transfers visible-token supervision from distillation into pretraining, in addition to the masked-token objective.The ablation contrasts applying loss to visible tokens and lowering the masking ratio from 75% to 0%.

A.3. Ablations on Multi-Granularity Captions

The caption ablations identify alternating real and synthetic captions with two [CLS] tokens as the optimal multi-granularity recipe for TIPSv2.

  • A.3. Ablations on Multi-Granularity Captions: Alternating real and synthetic captions with the dual CLS setup is the optimal recipe in TIPSv2.The ablations vary one or two [CLS] tokens and their assignment across web, PaliGemma, and Gemini captions.
  • A.3. Ablations on Multi-Granularity Captions: The ablations compare assignments of web, PaliGemma, and Gemini captions to one or two [CLS] tokens.Texts for two CLS tokens are separated by '/', with sources uniformly sampled for each CLS.
  • A.3. Ablations on Multi-Granularity Captions: The table reports segmentation, depth, ImageNet, image-to-text retrieval, and zero-shot segmentation evaluation columns.The listed columns include ADE20k, NYUv2, KNN, COCO, and ADE150 among the evaluation labels.

A.4. Qualitative Comparisons to DINOv2 and v3

Compared with DINOv2 and DINOv3, TIPSv2 produces more semantically focused and spatially coherent PCA features, although DINOv3 maps appear smoother.

  • A.4. Qualitative Comparisons to DINOv2 and v3: TIPSv2 PCA maps are more granular than DINOv3 maps, while retaining comparable improvements in spatial coherence over TIPS.The comparison uses common ViT-L models and larger ViT-g or ViT-7B models, with different input resolutions and model scales.
  • A.4. Qualitative Comparisons to DINOv2 and v3: TIPSv2 separates backpacks from their associated people, whereas DINOv3 PCA maps make them appear semantically similar.This comparison is shown in the third row of Figures 7 and 8.
  • A.4. Qualitative Comparisons to DINOv2 and v3: TIPSv2 distinguishes ceiling lamps and other features that DINOv3 tends to undercluster.The contrast is described for the second row of Figure 8.
  • A.4. Qualitative Comparisons to DINOv2 and v3: Compared with DINOv2, TIPSv2 maps are smoother and more spatially coherent rather than noisy.This is an overall qualitative comparison of the PCA maps.

A.5. Qualitative Analysis: iBOT++ vs iBOT

The iBOT++ objective yields substantially better patch-text alignment than iBOT, producing cleaner zero-shot segmentation maps and improved qualitative structure.

  • A.5. Qualitative Analysis: iBOT++ vs iBOT: iBOT++ significantly improves patch-text alignment over the iBOT baseline during pretraining.The improvement is quantified through zero-shot segmentation metrics in Table 4.
  • A.5. Qualitative Analysis: iBOT++ vs iBOT: The qualitative comparison uses the first three PCA components of ViT-L models from DINOv2, DINOv3, and TIPSv2.Images are processed at 1372 resolution for patch-size-14 models and 1568 resolution for the patch-size-16 DINOv3 model.
  • A.5. Qualitative Analysis: iBOT++ vs iBOT: iBOT++ produces significantly cleaner zero-shot segmentation maps than iBOT.Figure 9 visualizes the corresponding first and second rows of Table 4.

A.6. Zero-shot Segmentation with SigLIP2

Within the SigLIP2 family, smaller models can outperform larger ones on zero-shot segmentation; more broadly, TIPSv2 performs strongly across benchmarks and generally wins head-to-head comparisons.

  • A.6. Zero-shot Segmentation with SigLIP2: The smallest SigLIP2 model outperforms larger versions in two zero-shot segmentation evaluations, while the SO model wins the third.The comparison covers three model sizes, with the largest ViT-g not winning these evaluations.
  • A.6. Zero-shot Segmentation with SigLIP2: TIPSv2 variants include ViT-B, ViT-L, ViT-g, and SO-400m, with only ViT-g pretrained directly.The smaller variants are obtained by patch-level distillation from the ViT-g teacher.
  • A.6. Zero-shot Segmentation with SigLIP2: TIPSv2 generally ranks ahead of competing vision encoders on shared evaluations in head-to-head comparisons.Figure 10 counts the metrics where each model achieves the best result.
  • A.6. Zero-shot Segmentation with SigLIP2: The training loss combines contrastive, DINO, and iBOT components with weights α = 1.0 and β = 2.0.The contrastive loss averages results from two captions.
Loading 2604.12012v1…