Source-linked AI summary
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, Radu Soricut
TL;DR
PaLI-3 addresses how to build practical, smaller VLMs without sacrificing multimodal performance. It combines contrastive SigLIP image encoding with staged training and higher resolutions, achieving strong results across tasks, especially localization and visually situated text understanding. The paper also scales SigLIP to 2B parameters and reports state-of-the-art multilingual retrieval.
Problem
Smaller VLMs remain important for practical training, serving, environmental impact, and faster research cycles, while the relative value of classification versus contrastive image pretraining was unclear.
Method
PaLI-3 combines a contrastively pretrained SigLIP image encoder with an improved multimodal data mixture and higher-resolution staged training in a 5B-parameter VLM.
Results
PaLI-3 achieves strong or state-of-the-art performance across diverse VLM benchmarks, with particularly strong results on localization and visually situated text understanding.
Takeaways & Limitations
Contrastive image-text pretraining can yield better and more efficient VLMs than classification pretraining, especially for localization and text understanding tasks.
Takeaways & Limitations
The paper states that its limitations are similar to those already identified in prior literature.
Abstract
from arXiv · showhide
This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger. As part of arriving at this strong performance, we compare Vision Transformer (ViT) models pretrained using classification objectives to contrastively (SigLIP) pretrained ones. We find that, while slightly underperforming on standard image classification benchmarks, SigLIP-based PaLI shows superior performance across various multimodal benchmarks, especially on localization and visually-situated text understanding. We scale the SigLIP image encoder up to 2 billion parameters, and achieves a new state-of-the-art on multilingual cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles research on fundamental pieces of complex VLMs, and could fuel a new generation of scaled-up models.
1 INTRODUCTION
PaLI-3 targets practical small-scale VLMs by combining a 5B backbone with a refined training recipe. Its contrastive image encoder and broader training design achieve strong results, especially for visually situated text and localization.
- The paper positions smaller VLMs as more practical to train and serve, more environmentally friendly, and better suited to faster research cycles.
- PaLI-3 uses only 5B parameters while achieving competitive and new state-of-the-art results across diverse VLM benchmarks.Its recipe combines contrastive image-encoder pretraining, an improved multimodal dataset mixture, and higher-resolution training.
- PaLI-3 achieves new state-of-the-art results on eight visually situated text-understanding tasks and RefCOCO referring-expression segmentation.It also performs strongly across classical vision tasks.
- Contrastively pretrained SigLIP models outperform classification-pretrained ViT models on visually situated text understanding and localization tasks.The comparison supports contrastive pretraining on noisy web-scale image-text data as an alternative to classification-style pretraining.
2 RELATED WORK
Related work distinguishes image-encoder pretraining choices and multimodal task categories. Earlier VLMs often specialized in either natural-scene understanding or visually situated text, motivating comparisons across both.
- Recent VLMs use supervised classification, CLIP-style contrastive, or custom multimodal pretraining for their image encoders.
- Scaling classification-pretrained image encoders can improve vision-language tasks more noticeably than ImageNet performance.Prior PaLI results found larger gains when scaling from ViT-G to ViT-e on VL tasks than on ImageNet.
- Multimodal understanding spans natural scene understanding and visually situated text understanding, which require different levels of visual granularity.Previous models often showed strong performance mainly in one category, leading their training recipes to specialize accordingly.
3 MODEL
PaLI-3 combines a contrastively pretrained 2B SigLIP vision encoder with a 3B UL2 encoder-decoder and trains them through staged multimodal and resolution-increase procedures. The model is designed to improve detailed visual understanding while preserving a compact architecture.
- 3.1 ARCHITECTURE: PaLI-3 encodes images into visual tokens with a contrastively pretrained 2B SigLIP ViT and passes them with text to a 3B UL2 encoder-decoder.The UL2 model generates the text output from the projected visual and input-text tokens.
- 3.1 ARCHITECTURE: The image encoder is initialized from a contrastively pretrained ViT-G/142 model using the SigLIP training recipe.
- 3.2 STAGES OF TRAINING: Stage 0 contrastively pretrains the image encoder on filtered web image-text pairs, unlike the classification-pretrained encoders used by PaLI and PaLI-X.About 40% of image-text pairs are retained after model-based filtering.
- 3.2 STAGES OF TRAINING: Stage 1 combines the image encoder with the text encoder-decoder for multimodal training while keeping the image encoder frozen.
- 3.3 ENCODER COMPARISON: Across tasks, SigLIP encoders provide moderate gains on captioning and question answering and large gains on detailed scene-text and spatial-understanding tasks.The comparison uses the same PaLI setup for contrastive and classification-pretrained encoders.
- 3.2 STAGES OF TRAINING: Stage 2 fine-tunes the whole model through increasing resolutions, retaining checkpoints at 812×812 and 1064×1064.The data mixture emphasizes visually situated text and object detection.
- 3.2 STAGES OF TRAINING: Task specialization fine-tunes the model separately for each benchmark with the ViT image encoder frozen.Most tasks use the 812×812 checkpoint, while two document-understanding tasks use 1064×1064.
4 EXPERIMENTS
PaLI-3 is evaluated across multimodal tasks, with controlled comparisons showing that SigLIP-based vision encoders improve multimodal performance despite weaker standard classification results. The model is especially strong for visually situated text, localization, natural-image understanding, and video question answering.
- Classification or contrastively pretrained ViT?: SigLIP models provide moderate gains on captioning and question-answering and large gains on scene-text and spatial-understanding tasks such as TextVQA and RefCOCO.Few-shot linear classification performance is weaker than classification-pretrained models, motivating the switch to sigmoid-contrastive pretraining.
- Visually-situated text understanding: Across eight visually situated text benchmarks, PaLI-3 is 4.4 points ahead of combined SOTA methods without external OCR and only 0.7 points behind them with external OCR.For TextCaps, TextVQA, InfographicVQA, and DocVQA, the no-external-OCR advantage is at least 8 points.
- Referring expression segmentation: Contrastive pretraining is much more effective for the evaluated localization task, and full PaLI-3 slightly outperforms the state of the art on referring expression segmentation.PaLI-3 predicts segmentation masks from referring expressions using bounding-box coordinates followed by 16 mask tokens.
- Natural image understanding: On natural-image benchmarks, PaLI-3 outperforms all listed models but BEiT-3 and the 17B and 55B PaLI models on COCO, while remaining within 1 point of prior results on VQAv2.On OKVQA, PaLI-3 trails only PaLM-E and PaLI-X and outperforms the 32-shot Flamingo model.
- Video captioning and question answering: Without video pretraining, PaLI-3 achieves state-of-the-art video QA on MSR-VTT-QA and ActivityNet-QA, competitive NextQA results, and video-captioning results within 3 CIDEr points of SOTA on average.The authors attribute consistent image- and video-QA improvements to the contrastive ViTs.
- Direct image encoder evaluation: The isolated SigLIP encoder slightly lags classification-pretrained ViTs on standard image classification but significantly outperforms them on vision-language tasks.The comparison includes multilingual image-text retrieval and classification evaluations of the visual component without the surrounding language model.
5 MODEL FAIRNESS, BIASES, AND OTHER POTENTIAL ISSUES
PaLI-3 is evaluated for toxicity, demographic parity, and subgroup detection accuracy. It maintains low detection error across subgroups, while demographic-parity differences remain for some occupations.
- Demographic parity: A mean demographic-parity difference of µ = 0.37 indicates higher average log-perplexity for women than men across most occupations.Demographic parity is computed from differences in average generation log-perplexity between demographic groups.
- Comparison with PaLI-X: Compared with PaLI-X, fewer occupations in PaLI-3 fall outside the interval µ ± 2σ.The fairness evaluation follows the prior PaLI-X protocol and also includes toxicity and profanity analysis using FairFace and MIAP.
- Subgroup detection: PaLI-3 maintains a low detection error rate for identifying a person across all MIAP demographic subgroups.The evaluation uses images containing exactly one individual.
- Limitations: The paper states that its limitations are similar to those already reported for prior work and refers readers to Chen et al. (2023a).The supplied limitation statement does not specify additional PaLI-3-specific boundaries.
6 CONCLUSION
The conclusion isolates image-encoder pretraining as a key component of VLMs and reports that contrastive pretraining can produce better and more efficient models, especially for localization and text understanding. The authors frame this as one aspect of VLM training that merits further investigation.
- Conclusion: Controlled experiments compare classification and image-text contrastive pretraining for PaLI image encoders.The comparison focuses specifically on the image-encoder pretraining component of large PaLI-type VLMs.
- Conclusion: Contrastive pretraining can lead to better and more efficient VLMs, especially for localization and text understanding tasks.This is the conclusion’s stated finding from the controlled comparison.
- Conclusion: The study covers one small aspect of VLMs and is intended to spur further detailed investigations of other training components.The conclusion explicitly limits the scope of the study to image-encoder pretraining.
A.1 DATASETS & BENCHMARKS
The appendix documents that some benchmark videos were unavailable during data collection, reducing the effective dataset sizes. The resulting training and evaluation subsets therefore differ from the public benchmark splits.
- Dataset availability: Some benchmark videos were unavailable online when the datasets were freshly collected, reducing the effective number of videos.The paper reports exact subset details in Table 10.
- Experimental settings: The experiments retain PaLI-X’s dataset splits, evaluation metrics, and experimental settings.The appendix refers to PaLI-X for additional data-related details.
- Dataset availability: Except for NextQA and SMIT, more than 10% of videos are missing from both training and evaluation.The table characterizes the reduced datasets as making the tasks more challenging.
B ADDITIONAL RESULTS: CROSSMODAL-3600 RETRIEVAL
Additional Crossmodal-3600 results compare zero-shot image-text retrieval for SigLIP ViT-G and classification-pretrained ViT-e across 36 languages. SigLIP ViT-G is consistently stronger, including a large average text-to-image retrieval gain.
- Crossmodal-3600 retrieval: Figures 3 and 4 report image-to-text and text-to-image zero-shot retrieval recall@1, respectively, for SigLIP ViT-G and Classif ViT-e.Table 11 provides detailed Crossmodal-3600 results alongside the figures.
- Crossmodal-3600 retrieval: SigLIP ViT-G significantly outperforms Classif ViT-e across all languages in Crossmodal-3600 retrieval.The comparison uses zero-shot recall@1 and covers 36 languages.
- Crossmodal-3600 retrieval: 28.5% to 44.0% average text-to-image retrieval recall@1 is reported for SigLIP ViT-G versus Classif ViT-e.These values are the table’s average text-to-image retrieval results.