PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Language-Free Visual Representation Learning

David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, Saining Xie

arXiv:2504.01017v1cs.CV

TL;DR

The paper asks whether visual SSL underperforms CLIP because it lacks language supervision or because it is trained on different data. It compares both approaches on matched MetaCLIP data using diverse VQA and vision evaluations, finding that visual SSL scales better and reaches CLIP-level performance across tasks. The results support visual SSL as a vision-centric alternative at scale.

  • Problem

    Visual SSL has underperformed CLIP in multimodal VQA despite stronger performance on classic vision tasks, leaving the roles of language supervision and training-data differences unresolved.

  • Method

    The study trains visual SSL and CLIP models on the same MetaCLIP data and evaluates them with diverse Cambrian-1 VQA tasks alongside classic vision benchmarks.

  • Results

    Visual SSL matches and can surpass language-supervised methods across VQA, scales with model capacity and data, and maintains competitive classification and segmentation performance.

  • Takeaways & Limitations

    Pure visual SSL can match language-supervised visual pretraining at scale, including on language-related OCR & Chart tasks.

  • Takeaways & Limitations

    The study does not explore zero-shot image classification through language-based adaptation, other LLM backbones, or larger and/or uncurated datasets.

Abstract

from arXiv · show

Visual Self-Supervised Learning (SSL) currently underperforms Contrastive Language-Image Pretraining (CLIP) in multimodal settings such as Visual Question Answering (VQA). This multimodal gap is often attributed to the semantics introduced by language supervision, even though visual SSL and CLIP models are often trained on different data. In this work, we ask the question: "Do visual self-supervised approaches lag behind CLIP due to the lack of language supervision, or differences in the training data?" We study this question by training both visual SSL and CLIP models on the same MetaCLIP data, and leveraging VQA as a diverse testbed for vision encoders. In this controlled setup, visual SSL models scale better than CLIP models in terms of data and model capacity, and visual SSL performance does not saturate even after scaling up to 7B parameters. Consequently, we observe visual SSL methods achieve CLIP-level performance on a wide range of VQA and classic vision benchmarks. These findings demonstrate that pure visual SSL can match language-supervised visual pretraining at scale, opening new opportunities for vision-centric representation learning.

1 Introduction

The paper examines whether visual SSL trails CLIP because it lacks language supervision or because the methods use different training data. Using matched data and broader evaluation, it finds that scaled visual SSL can match or surpass language-supervised models across VQA and classic vision tasks.

  • Visual SSL previously performed strongly on classic vision tasks but lagged as an encoder for multimodal VQA, especially OCR & Chart interpretation.
  • The study controls for data differences by training visual SSL and CLIP models on the same billion-scale MetaCLIP web data.
  • The evaluation combines 16 Cambrian-1 VQA tasks with vision-centric, reasoning, and OCR & Chart capabilities.
  • Visual SSL can match and even surpass language-supervised methods across VQA tasks, including OCR & Chart understanding.
  • Visual SSL scales with model capacity and data while maintaining competitive classification and segmentation performance.
  • A higher proportion of images containing text is especially effective for improving OCR & Chart performance.

2 From Visual SSL 1.0 to 2.0

Visual SSL 2.0 extends prior experiments by scaling data and model size while adding open-ended VQA evaluation alongside classic benchmarks. The controlled setup uses matched MetaCLIP data, frozen vision encoders, and a common multimodal instruction-tuning recipe.

  • From Visual SSL 1.0 to 2.0: The experimental setup scales dataset size to billion-scale images, model size beyond 1B parameters, and evaluation to open-ended VQA.
  • Beyond ImageNet Pretraining: MC-2B contains 2 billion MetaCLIP samples; SSL uses only images, whereas CLIP uses image-text pairs.
  • Beyond ImageNet Pretraining: Using the same MetaCLIP data controls data distribution and size as confounding variables for a fairer comparison of pretraining methods.
  • Scaling Up Vision Models to Billion Scale: Vision Transformers with 1B, 2B, 3B, 5B, and 7B parameters are trained on images from MC-2B.
  • Multimodal LLMs as an Evaluation Protocol: All experiments use the same two-stage instruction-tuning procedure, with the vision encoder frozen and Llama-3 8B Instruct as the LLM backbone.
  • Multimodal LLMs as an Evaluation Protocol: The Cambrian-1 evaluation suite contains 16 VQA benchmarks across General, Knowledge, OCR & Chart, and Vision-Centric domains.

3 Scaling Visual SSL

Using the same MC-2B image data, the study scales Web-DINO and CLIP across model capacity and training examples, finding stronger continued gains for visual SSL. Web-DINO reaches or exceeds CLIP performance across VQA categories, with especially notable progress on OCR & Chart and Vision-Centric tasks.

  • Experimental setup: Web-DINO and same-size CLIP models are trained on 2 billion MC-2B images, using images alone for SSL and image-text pairs for CLIP.The controlled setup keeps data and model sizes aligned while preserving each method’s supervision format.
  • Scaling model capacity: As model size increases from 1B to 7B parameters, Web-DINO’s Average, OCR & Chart, and Vision-Centric VQA performance improves nearly log-linearly.General and Knowledge VQA improve less strongly, but still show gains with increased capacity.
  • Scaling model capacity: CLIP VQA performance largely saturates after 3B parameters, whereas Web-DINO continues improving and can exceed CLIP’s Average VQA performance at 5B parameters and above.DINO increasingly outperforms CLIP on Vision-Centric VQA and largely closes the gap on OCR & Chart and Average VQA.
  • Scaling examples seen: With Web-DINO ViT-7B, General and Knowledge performance saturates at 4B and 2B examples respectively, while Vision-Centric performance saturates beyond 2B examples.Training examples range from 1B to 8B images from MC-2B.
  • Scaling examples seen: OCR & Chart is the only VQA category that consistently improves as Web-DINO sees more examples, producing steady average-performance gains.After 8B samples, Web-DINO closes its OCR & Chart gap with same-size CLIP and consistently outperforms CLIP on average VQA.
  • Overall finding: Together, model- and data-scaling results indicate that visual SSL learns increasingly effective VQA features and does not have an absolute CLIP advantage.The reported gains are strongest for OCR & Chart while remaining broad across VQA capabilities.

4 Scaling Analysis and Findings

Scaling visual SSL with web-scale data and model capacity improves diverse VQA performance, including OCR and Chart understanding, while these trends generalize across SSL methods. Data composition also matters: text-rich filtering can outperform language supervision on text-centric tasks.

  • Scaling across SSL methods: Web-MAE performs especially well on OCR & Chart tasks but underperforms Web-DINO in other domains, showing that SSL methods learn distinct representations under the same training conditions.The comparison suggests scaling behavior generalizes across joint-embedding and masked-modeling approaches without making their representations identical.
  • Data scale and diversity: MC-2B pretraining improves with increasing data diversity and scale, whereas ImageNet-1k pretraining remains inferior and shows no clear improvement with larger models.The ImageNet-1k comparison evaluates Web-DINO models from 1B to 3B parameters against models trained on MC-2B.
  • Classic vision performance: Web-DINO models from 1B to 7B parameters achieve strong classic-vision performance, with moderate gains as model size increases, although scaling is less pronounced than on VQA.The benchmarks include ImageNet-1k classification, ADE20K segmentation, and NYU Depth estimation.
  • Data composition and OCR: Aggressive filtering to 1.3% of MC-2B improves OCR & Chart performance and yields +4.3% over full-data language-supervised CLIP ViT2B on OCR & Chart.The same filtered model also improves Average VQA by +2.6% over full-data Web-DINO ViT-2B and +0.7% over full-data CLIP ViT2B.
  • Data composition and OCR: Text-rich filtering produces particularly large gains on ChartQA (+24.2%), OCRBench (+13.8%), and DocVQA (+13.0%), indicating data composition is crucial for OCR & Chart understanding.Performance remains competitive in the other categories despite the filtering.
  • Alignment: Increasing training samples, model size, and data diversity progressively improves vision–LLM alignment without explicit language supervision.The reported trends compare DINOv2 with Web-DINO, Web-DINO models of different sizes, and Web-DINO trained on 2B versus 8B samples.

5 The Web-SSL Model Family

The Web-SSL model family performs competitively with language-supervised encoders across VQA and classic vision benchmarks. Web-DINO improves with higher resolution and can match strong CLIP baselines despite using no language supervision.

  • Benchmark comparison: Web-DINO can outperform off-the-shelf MetaCLIP on both VQA and classic vision tasks at 224px.It matches SigLIP and SigLIP2 on VQA despite seeing 5× less data and receiving no language supervision.
  • Benchmark comparison: Web-DINO outperforms off-the-shelf visual SSL methods in all VQA categories while remaining competitive on traditional vision benchmarks.The comparison includes DINOv2 and language-supervised CLIP models.
  • Resolution scaling: Higher-resolution Web-DINO improves steadily in average VQA from 224 to 378 to 518 resolution, with notable gains in OCR & Chart performance.Classic vision performance improves modestly with higher resolution, while Web-DINO trails SigLIP at 384 resolution.

6 Related Work

Prior work separates language-supervised and visual SSL representation learning, with SSL traditionally evaluated on classic vision tasks and often trained on smaller or different datasets. This work extends SSL to web-scale data and evaluates it across broader VQA challenges.

  • Visual representation learning: CLIP learns from paired image-text data, whereas visual SSL learns representations from images alone without language.These approaches represent two distinct training paradigms for visual representation learning.
  • Training data: Earlier SSL methods commonly relied on datasets such as ImageNet, while language-supervised models increasingly use billion-scale image-text data.Prior SSL work also scaled to larger unlabeled datasets including YFCC, LVD-142M, and IG-3B.
  • This work’s positioning: The paper bridges these paradigms by training visual SSL models on web-scale data and showing that data diversity, quantity, and text concentration affect VQA performance.The controlled experiments specifically emphasize OCR & Chart understanding and training-distribution sensitivity.
  • Evaluation scope: The evaluation literature has expanded beyond classification to segmentation, depth estimation, and broader VQA tasks that capture real-world perceptual challenges.The paper follows recent proposals to use VQA with MLLMs as a complementary evaluation framework for vision encoders.

7 Limitations

The study’s scope is limited to vision-only pretraining without language supervision and to a fixed Llama-3 8B Instruct integration setting. Larger or uncurated datasets and alternative LLM backbones remain outside the study.

  • Scope boundaries: Vision-only models do not support zero-shot image classification out of the box, unlike language-supervised models.The paper instead integrates visual SSL models into MLLM frameworks through instruction tuning.
  • Scope boundaries: The experiments fix the base LLM to Llama-3 8B Instruct, leaving the behavior of other LLM backbones outside the study’s scope.The authors hypothesize that findings with other backbones would be similar but do not evaluate them.
  • Data scope: The demonstrated scaling results use MetaCLIP data, while exploration of larger or uncurated datasets is left for future work.This constrains the tested data regime rather than establishing behavior for all web-scale datasets.

8 Discussion

Large-scale visual encoders trained with language-free self-supervised objectives can produce high-quality features for multimodal models. The findings support reducing supervision, including language, as a promising direction for computer vision.

  • Language-free self-supervised training can produce high-quality visual features for multimodal models.
  • The results echo the “bitter lesson” by supporting less supervised approaches, including those without language.
  • The authors hope vision-only approaches will advance models spanning traditional vision and modern multimodal capabilities.

A Implementation Details

The study follows established open-source training and evaluation procedures for Web-DINO, Web-MAE, and CLIP, with controlled VQA and classic-vision comparisons. It also specifies scaling architectures, training adjustments, and text-filtering variants.

  • Training: Web-DINO, Web-MAE, and CLIP training follows the official DINOv2, MAE, and MetaCLIP codebases with distributed FSDP training.
  • Training: For the largest models, Web-DINO and CLIP retain original recipes, while MAE uses a lower learning rate and longer warmup to reduce divergence.The MAE learning rate changes from 2.4e-3 to 1.6e-3, with warmup extended to 80K iterations.
  • VQA evaluation: VQA evaluation freezes the vision encoder and uses Cambrian-Alignment for projector training, Cambrian-7M for fine-tuning, and Llama-3 8B Instruct.
  • Classic vision evaluation: Classic vision evaluation uses DINOv2 procedures with linear probes on ImageNet-1k, ADE20K, and NYU Depth v2.ADE20K reports linear and +ms settings, while NYU Depth v2 reports lin. 1 and lin. 4.

B.1 Web-DINO

The appendix provides complete quantitative results for model and data scaling, reference models, high-resolution adaptation, and text-filtering experiments. These results show clearer scaling for visual SSL than for CLIP and identify training-distribution effects on downstream capabilities.

  • Scaling experiments: Quantitative results cover Web-DINO model scaling on VQA and classic vision evaluation.
  • Scaling experiments: Web-DINO ViT-7B results evaluate scaling the number of training images on both VQA and classic vision benchmarks.
  • Additional experiments: The appendix also reports results for reduced-data Web-DINO training, scaled MAE and MetaCLIP training, reference models, and high-resolution Web-DINO adaptation.
  • Scaling experiments: CLIP models do not exhibit clear scaling behavior in contrast to the visual SSL results.
  • Training distribution: Text-rich training distributions make SSL features particularly well-suited for OCR & Chart tasks and can steer models toward that application.

D Evaluation

The evaluation benchmarks are catalogued together with their purposes.

  • Table 16 lists the evaluation benchmarks used and their purposes.

E Pretraining Dataset Cards

The dataset cards document comparisons between curated and web-scale pretraining data, alongside evaluation tables for VQA and classic vision benchmarks. They cover Web-DINO, Web-MAE, MetaCLIP, CLIP, and SSL configurations, including scaling and resolution variants.

  • Dataset comparison: LVD-142M is a carefully curated data mix aligned with downstream classic vision evaluation tasks.It was used to train the off-the-shelf DINOv2 model.
  • VQA evaluation: The evaluation tables cover VQA results for Web-DINO, Web-MAE, MetaCLIP, CLIP, and SSL models across multiple training settings.Listed variants include MC-2B training, text-filtered MC-2B, ImageNet-1k training, and off-the-shelf models.
  • Scaling evaluation: Additional tables evaluate Web-DINO ViT-7B under increased image counts and different resolutions.These configurations are reported for both VQA and classic vision evaluation.
  • Benchmark coverage: The benchmark list includes both VQA and classic vision evaluation categories.The cards separately identify VQA evaluation and classic vision evaluation tables.
  • Dataset comparison: MetaCLIP data was collected from 15 CommonCrawl snapshots and is less curated than LVD-142M.The snapshots span January 2021 through January 2023.
Loading 2504.01017v1…