PaperlayerView exact revision on arXiv

Source-linked AI summary

DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment

Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, Oriane Siméoni, Huy V. Vo, Patrick Labatut, Piotr Bojanowski

arXiv:2412.16334v1cs.CV

TL;DR

Self-supervised visual features are powerful but are not readily aligned with language for open-vocabulary tasks. dino.txt adapts LiT to DINOv2 with changes to alignment and data curation. The resulting model reports strong zero-shot classification and open-vocabulary segmentation with a fraction of CLIP’s computational cost.

  • Problem

    Self-supervised visual features perform well on downstream tasks but lack the language alignment needed for open-vocabulary use, while LiT applied to DINOv2 performs unsatisfactorily on dense tasks.

    Source: §1 · p. 1

  • Method

    dino.txt aligns a text encoder to frozen DINOv2 features, combining global and patch-level visual information and curating data across image and text modalities.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    dino.txt reports state-of-the-art zero-shot classification and high-quality open-vocabulary segmentation with fast training at a fraction of CLIP’s computational cost.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that classic semantic-segmentation benchmarks require rethinking for open-vocabulary evaluation because they do not accommodate overlapping concepts or finer-grained predictions.

    Source: §4.5 · p. 8, §5 · p. 9

    Existing segmentation datasets may not suit open-vocabulary evaluation because classes overlap and class names do not align with common natural-language use.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Self-supervised visual foundation models produce powerful embeddings that achieve remarkable performance on a wide range of downstream tasks. However, unlike vision-language models such as CLIP, self-supervised visual features are not readily aligned with language, hindering their adoption in open-vocabulary tasks. Our method, named dino.txt, unlocks this new ability for DINOv2, a widely used self-supervised visual encoder. We build upon the LiT training strategy, which trains a text encoder to align with a frozen vision model but leads to unsatisfactory results on dense tasks. We propose several key ingredients to improve performance on both global and dense tasks, such as concatenating the [CLS] token with the patch average to train the alignment and curating data using both text and image modalities. With these, we successfully train a CLIP-like model with only a fraction of the computational cost compared to CLIP while achieving state-of-the-art results in zero-shot classification and open-vocabulary semantic segmentation.

1. Introduction

Self-supervised visual features are strong but lack a language interface for open-vocabulary tasks. The paper introduces dino.txt, adapting LiT with changes intended to improve image- and pixel-level alignment while keeping training efficient.

1. Introduction

  • Self-supervised vision features support broad downstream tasks, but their lack of a language interface limits their use in open-vocabulary scenarios.The paper positions DINOv2 as a versatile model with global and local features and aims to add language alignment for open-vocabulary recognition.Source: §1 · p. 1
  • Applying LiT to DINOv2 is not straightforward: global-only alignment yields unsatisfactory performance on fine-grained tasks such as semantic segmentation and image-text retrieval.LiT freezes the vision encoder and trains the text encoder to align with its embedding space.Source: §1 · p. 1
  • dino.txt achieves state-of-the-art zero-shot classification with fast, efficient training and produces high-quality open-vocabulary segmentation results.Figure 1 presents results for the method trained with weak image/caption annotation.Source: Figure 1 · p. 2, §1 · p. 2
  • The method aligns DINOv2 with language by combining the [CLS] token and average patch features, then adding two learnable vision blocks above the frozen encoder.The combined representation is intended to capture global context and local image information, while the blocks adapt vision features to the training data.Source: §1 · p. 2
  • Balancing image and text data during curation eases training, supports good performance at a fraction of the compute cost, and permits experiments with a wider text encoder.The paper also connects data quality with both model performance and training efficiency.Source: §1 · p. 2
  • The paper also contributes an error analysis that identifies limitations of existing segmentation benchmarks and distinguishes error types in these models.It presents these findings as pointing toward future directions for more effective and efficient language-aligned vision foundation models.Source: §1 · p. 2

2. Related work

Related work spans self-supervised visual representations, text-image pretraining and automatic data curation, as well as methods for dense, open-vocabulary alignment. The paper positions dino.txt among approaches that use weak image/caption supervision to produce both image- and pixel-level alignment.

Self-supervised feature learning

  • Self-supervised visual learning includes contrastive and reconstruction-based methods, with DINOv2 identified as state of the art for scaling SSL models.Examples include MoCo, BYOL, SwAV, DINO, MAE, BeiT, and I-JEPA.Source: §2 · p. 2

Contrastive text-image pre-training

  • CLIP established contrastive image-caption training, while later work scales or refines it through open datasets, rebalancing, data filtering, and larger models.The passage names CLIP [64], LAION [70], OpenCLIP [16], MetaCLIP [88], DFN [27], EVA-CLIP [73] [74], and InternVL [15].Source: §2 · pp. 2–3
  • LiT [92] freezes the pretrained image encoder and trains the text encoder, whereas dino.txt retains CLIP’s original contrastive loss and a learnable text encoder.Other modifications include SigLIP’s binary log loss and LLiP’s [48] use of image register tokens.Source: §2 · p. 3

Automatic data curation at scale

  • Prior data-curation methods balance or filter training data in different ways, while dino.txt balances both image and caption distributions for better performance and training efficiency.Examples include CLIP’s [64] frequent-word querying, DFN’s [27] data evaluation, SemDeDup’s [2] duplicate pruning, and Vo et al.’s [79] support-uniform sampling.Source: §2 · p. 3

Open-vocabulary segmentation

  • Open-vocabulary segmentation methods derive dense text-aligned features through multiple image views, concept codebooks, or adaptations such as MaskCLIP [97].Feature refinement can use improved attention [7] [80] or SSL guidance [43] [47].Source: §2 · p. 3
  • Dense-alignment approaches use pixel annotations or weak image/caption data; dino.txt focuses on the latter and targets image- and pixel-level alignment with a simple loss.Related methods include ViewCO [68], CLIPSelf [84], GroupViT [89], PACL [58], TCL [12], and CoDe [83]; CLIPpy reports worse classification results.Source: §2 · p. 3

3. Proposed approach: dino.txt

dino.txt aligns a text encoder with a frozen self-supervised vision model, using changes to image representation and data curation to support classification and dense prediction.

3. Proposed approach: dino.txt

  • After 50k iterations, dino.txt achieves state-of-the-art results in zero-shot classification and open-vocabulary segmentation.The method overview reports both outcomes.Source: Figure 2 · p. 4

3.1. Locked-image text alignment

  • The text encoder is trained from scratch to align its output [EOS] token with the image embedding space, while the vision backbone remains frozen.The trainable parts also include the added vision blocks.Source: §3.1 · p. 4
  • For alignment, dino.txt concatenates the [CLS] embedding with average-pooled patch embeddings to support global and local alignment without pixel-level supervision.Two trainable transformer blocks sit atop the frozen vision backbone, and the resulting image descriptor is aligned with the text encoder using contrastive learning.Source: §3.1 · p. 4

3.2. Text- and image-based data curation

  • The curation pipeline intersects text-based curation with image-based hierarchical-k-means clustering and subsampling to balance caption and image distributions.The image-based curation reduces the impact of head clusters; the motivation is that noisy captions may not describe the visual concepts in their images.Source: §3.2 · p. 5

3.3. Inference

  • For dense prediction, the model compares patch representations with the text-embedding part aligned to average patches, then upsamples logits without task-specific model adaptation.For classification and retrieval, it compares a global image descriptor with text queries using cosine similarity.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate dino.txt on zero-shot classification, image-text retrieval, and open-vocabulary segmentation, alongside ablations and analysis of efficiency and failure modes. Results show strong classification and segmentation performance, with retrieval and dataset evaluation presenting important qualifications.

4.1. Tasks and metrics

  • The evaluation covers zero-shot classification, cross-modal retrieval, and open-vocabulary segmentation, using cosine similarity, Recall@1, and mIoU, respectively.Classification uses six benchmarks, retrieval uses COCO2017 and Flickr30K, and segmentation uses five datasets; segmentation follows a sliding-window protocol.Source: §4.1 · p. 5

4.2. Implementation details

  • Training uses a frozen DINOv2 encoder and curated CommonCrawl-derived image-text data; high-resolution inference aggregates varied crops and classifies k-means feature centroids.The inference procedure processes about 800 crops in 10 seconds on an A100, with each pixel visited about 40 times. Failure analysis reports a 38.9 mIoU ADE20K topline using ground-truth masks, and notes that class-name choices can add 2.1 mIoU.Source: §4.2 · p. 6, §4.5 · p. 8

4.3. Ablation study of our method dino.txt

  • The ablations report gains from larger batch size, two learnable vision blocks, and increasing text embedding size from 768 to 1280; combining image- and text-based curation performs best across three tasks.The curation strategies each help rebalance long-tailed data and boost performance; the combined strategy leads on all three tasks.Source: Table 3 · p. 7, Table 4 · p. 7

4.4. Comparisons to state of the art

  • dino.txt is on par with or better than CLIP-like alternatives on classification benchmarks, but trails retrieval competitors such as SigLIP.The authors report state-of-the-art classification performance on IN-v2, IN-A, and iNaturalist; the retrieval shortfall is attributed likely to the trained text encoder's quality.Source: §4.4 · p. 7, Table 5 · p. 8
  • dino.txt greatly outperforms alternative methods on open-vocabulary segmentation and performs on par with or better than specialized models, without engineering refinement.Performance trends lower on VOC20 and higher on other datasets, which the authors attribute to a domain gap; the table also reports a high-resolution inference variant.Source: §4.4 · p. 7, Table 6 · p. 8
  • 19 hours of training on 128 A100 GPUs yields 81.4% IN1K accuracy, versus CLIP's 79.0% after 110 hours; at the matched GFLOP budget, CLIP reaches 73%.The matched-budget result is 8.4% below dino.txt.Source: §4.4 · p. 7

4.5. Further Analysis

  • The trained text encoder scores 4.2% lower on average than CLIP's on MTEB tasks, and removing two learnable vision blocks lowers performance by a further 3.2% on average.The authors identify balancing the frozen encoder's quality with its ability to adapt to new data domains as future work.Source: §4.5 · pp. 8–9

5. Conclusion

dino.txt aligns a text encoder with frozen DINOv2 features to add open-vocabulary capabilities. It reports strong zero-shot classification and dense segmentation, while arguing that classic segmentation benchmarks need to accommodate overlapping concepts and finer-grained predictions.

5. Conclusion

  • dino.txt aligns a text encoder from scratch with frozen DINOv2 features, using self-supervised data curation without human annotation.The recipe is described as enabling fast training.Source: §5 · p. 9
  • The resulting text encoder aligns with patch-level features, supporting precise dense open-vocabulary segmentation.The conclusion attributes this capability to the quality of the frozen vision encoder.Source: §5 · p. 9
  • The paper reports zero-shot classification on par with the state of the art and argues that classic segmentation benchmarks need to allow overlapping concepts and finer-grained predictions.Source: §5 · p. 9

A. Additional details

The additional details discuss the training-cost comparison and the amortization of DINOv2 pretraining, as well as manually optimized ADE20K class names for segmentation.

A. Additional details

  • The paper presents dino.txt as a more efficient paradigm than training CLIP from scratch, with DINOv2 pretraining costs considered amortized over downstream adaptations.Table 7 reports computational costs in GPU hours, but the supplied passage does not give the values.Source: A. Additional details · p. 13, Table 7 · p. 13
  • Optimizing ADE20K class names instead of using dataset defaults improves open-vocabulary segmentation by 2.1 mIoU points; main-text experiments use original names for comparison with prior work.Source: A. Additional details · p. 13
  • The main-text experiments retain the original ADE20K class names to facilitate comparison with previous work.Source: A. Additional details · p. 13

B. Additional ablation studies

The additional ablations examine patch embeddings for dense segmentation and the effect of image-embedding dimensionality during training.

B. Additional ablation studies

  • Using only the model output patch descriptor with its corresponding text-embedding part gives the best ADE20K and Cityscapes segmentation results, while concatenating [CLS] at inference hurts performance.The authors particularly attribute the Cityscapes drop to the salient visual concept dominating [CLS].Source: B. Additional ablation studies · p. 13, Table 8 · p. 13
  • Retrieval performance increases from 30.9 to 34.7 with the trained [CLS avg] representation, and the authors attribute the gain largely to concatenating [CLS] with [avg].Source: B. Additional ablation studies · p. 13
  • Projecting the [CLS] embedding to dimension 2048 yields minimal benchmark gain, so higher image-embedding dimensionality does not explain the benefit of [CLS avg].Table 9 analyzes image-embedding size at training time.Source: Table 9 · p. 13, B. Additional ablation studies · p. 13

C. Additional qualitative results

The qualitative results show dino.txt segmenting complex scenes using descriptive text prompts. The appendix also presents manually refined ADE20K class names for segmentation.

C. Additional qualitative results

  • dino.txt uses a zero-shot semantic segmentation pipeline with a small number of descriptive text prompts per image and can segment complex scenes containing multiple semantic objects using specific text inputs.Source: C. Additional qualitative results · p. 13
  • The ADE20K class-name examples replace broad or ambiguous labels with more specific alternatives, including “building” with “facade” and “throne” with “toilet.”The modified names were selected through manual analysis.Source: Table 10 · p. 14
  • The qualitative segmentation figures use input images at 896×896 pixels.Source: Figure 5 · p. 15, Figure 6 · p. 16

Related papers

Loading 2412.16334v1…