Source-linked AI summary
PaliGemma: A versatile 3B VLM for transfer
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, Xiaohua Zhai
TL;DR
PaliGemma addresses the need for a versatile, transferable open base VLM that works across diverse tasks. It combines SigLIP and Gemma, trains on broad multimodal objectives, and achieves strong performance across benchmarks while remaining small. The paper also identifies boundaries involving limited-example transfer and the modest speed benefit of windowing.
Problem
Existing evidence motivates a versatile base VLM that can transfer across standard and specialized tasks, rather than serving only a narrow benchmark or out-of-the-box use case.
Method
PaliGemma combines a SigLIP encoder with Gemma-2B and trains the whole model through broad multimodal pretraining before fine-tuning it for individual tasks.
Results
PaliGemma delivers state-of-the-art performance across a wide variety of benchmarks, including standard and specialized tasks, despite having fewer than 3 B total parameters.
Takeaways & Limitations
The small open base model provides a starting point for instruction tuning, specific applications, and clearer separation between base models and fine-tunes.
Takeaways & Limitations
Few-example fine-tuning remains unstable across random seeds, and windowing yields at most a 5% training speedup in the reported setups.
Abstract
from arXiv · showhide
PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model. It is trained to be a versatile and broadly knowledgeable base model that is effective to transfer. It achieves strong performance on a wide variety of open-world tasks. We evaluate PaliGemma on almost 40 diverse tasks including standard VLM benchmarks, but also more specialized tasks such as remote-sensing and segmentation.
1. Introduction
PaliGemma is an open, sub-3B vision-language model combining SigLIP and Gemma, designed as a versatile base model for transfer. It targets strong performance across both standard and specialized vision-language tasks.
- PaliGemma combines the 400 M SigLIP encoder with the 2 B Gemma model in a sub-3 B VLM.
- The model continues PaLI’s scaling trend while maintaining performance comparable to PaLI-X, PaLM-E, and PaLI-3.
- PaliGemma uses the 2 B pretrained version of Gemma, an autoregressive decoder-only language model.
- Its main goal is a versatile base VLM that reaches state-of-the-art results on standard benchmarks and specialized tasks such as remote-sensing VQA and referring expression segmentation.
2. Related work
Vision-language models evolved from web-scale contrastive pretraining toward generative multimodal models and instruction-tuned systems. PaliGemma is positioned as an open base VLM without instruction tuning within this progression.
- Early vision-language models extended large-scale classification pretraining by using caption embeddings instead of fixed class sets.
- A second generation unified captioning and question answering through generative encoder-decoder modeling backed by advances in language models.
- PaLI and related models scaled multimodal systems, while later works added instruction tuning to make raw models more user-friendly.
- PaliGemma is an open base VLM without instruction tuning that studies what matters in vision-language models.
3. Model
PaliGemma uses an image-plus-text-to-text interface built from SigLIP, Gemma-2B, and a linear projection, then trains through staged multimodal pretraining and transfer. Its task mixture and representation are designed for broad fine-tuning versatility.
- Architecture: PaliGemma maps one or more images and a task prompt to an autoregressively generated text answer.
- Architecture: The image-plus-text-to-text interface can represent classification, captioning, question answering, detection, and segmentation as text outputs.
- Architecture: The model combines a SigLIP ViT-So400m encoder, Gemma-2B decoder-only language model, and linear projection into shared token dimensions.
- Architecture: Prefix-LM masking provides full attention over image and prefix tokens while applying autoregressive attention to the suffix.
- Training: Training proceeds through unimodal initialization, multimodal pretraining, resolution increase, and task-specific transfer.
- Training: The resulting model has less than 3 B total parameters while targeting performance comparable to much larger vision-language models.
- Training: The pretraining mixture builds transferable skills across tasks, including captioning in over 100 languages, while avoiding commercial VLM-generated task outputs.
4. Results
PaliGemma is fine-tuned on more than 30 benchmarks using task-specific validation-based selection, with results reported across tasks and resolutions. The experiments also examine transferability, resolution, and augmentation choices.
- PaliGemma is transferred to more than 30 academic benchmarks whose tasks and datasets were excluded from pretraining.
- Hyper-parameters are selected using validation or held-out minival data, followed by retraining on combined training and validation data for final testing.
- A single recommended hyper-parameter value performs almost as well as explored settings on most tasks.
- Results are reported at multiple resolutions for non-video tasks to show which tasks benefit from increased resolution.
- Simple square resizing performs as well as aspect-ratio-preserving zoom and crop augmentations for RefCOCO segmentation.
5. Ablations
The ablations identify training choices that preserve transfer performance: longer multimodal pretraining, prefix-LM supervision, standard new-token initialization, and tuning the full model are generally beneficial. Resolution-specific continued pretraining and native-resolution checkpoints further improve resolution-sensitive transfer, while windowing offers only limited speed gains.
- Multimodal pretraining duration: 100 M examples provide a 10x-shorter ablation setting without significantly hurting any task, whereas shorter training generally hurts and skipping Stage1 is worst.Task sensitivity varies substantially, motivating broad evaluation.
- Causal masking and learning objective: Prefix-LM masking with loss only on suffix tokens outperforms masking the prefix or image tokens and applying loss to prefixes.Task-prefixes do not noticeably affect transfer performance, although they change some pretraining perplexities.
- New token initialization: Standard σ= 0.02 initialization eventually surpasses AvgEmb in Stage1 perplexity and transfer on RefCOCO segmentation, despite AvgEmb's better initial loss.AvgEmb's initial advantage vanishes after roughly a thousand steps.
- To freeze or not to freeze?: Freezing the language model or resetting model parts significantly worsens performance, while tuning all weights improves spatial-task predictability and avoids the main freezing penalties.Keeping the image encoder frozen produces no transfer difference in one comparison, but spatial-task validation perplexity improves when it is tuned.
- Connector choice: 77.2 and 77.1 average transfer points show nearly identical performance for linear and MLP connectors when all weights are tuned, with a small deterioration for MLP.In the all-frozen setting, linear and MLP achieve 70.7 and 69.7 points respectively.
- Resolution and windowing: For resolution-sensitive tasks, increased image information and increased sequence-length capacity contribute roughly equally to the performance gain.This balance holds for each individual task examined, not only in aggregate.
- Resolution and windowing: Native 448 px checkpoints outperform transferring 224 px checkpoints upward or 448 px checkpoints downward, supporting separate checkpoints after higher-resolution Stage2 pretraining.Windowing can help when higher-resolution checkpoints or continued pretraining are unavailable, but remains below native 448 px performance.
- Resolution and windowing: Windowing yields at most a 5% training speedup because Gemma is much larger than ViT-So400m and is unaffected by windowing.Thus its speed advantage is limited relative to native higher-resolution pretraining.
6. Transferability
PaliGemma transfers reliably across many tasks, with a simple shared setup performing well for most tasks. Limited transfer data often suffices, although few-example results can vary substantially across seeds and batch orders.
- Transfer results are generally highly repeatable across five reruns, and Stage1 pretraining is similarly repeatable across three reruns.The reported standard deviations are generally very small across most tasks, with Stage1 reruns falling in the same range.
- A single simple hyper-parameter setup works well for the majority of tasks.The exceptions include RefCOCO and SciCap, which benefit from more epochs together with label smoothing and dropout.
- Few-example fine-tuning has high variance for some tasks, with RefCOCO mIOU varying from 10% to 30%.Variance also occurs when repeating training with the same examples but changing batch order.
- Most tasks reach within 10% of the full-data score with 4k examples and within 20% with 256 examples.The comparison uses the best runs for each hyper-parameter and seed against full-dataset results.
- Results with 64 transfer examples are often good enough to prototype a new application.Figure 12 summarizes relative regret when using limited numbers of transfer examples.
7. Noteworthy tidbits
The paper reports several practical findings spanning segmentation, counting, evaluation, prompting, positional encoding, and zero-shot transfer. It also identifies problems in prior WidgetCaps evaluations and reports a strong MMVP result.
- Segmentation: Simple square resizing to 224×224 works as well as aspect-ratio-preserving zoom-and-crop augmentations for segmentation.
- Counting: The authors introduce CountBenchQA because TallyQA’s skewed number distribution and varying image quality limit its usefulness for evaluating counting.
- Evaluation: Published WidgetCaps evaluations contain issues in at least three prior works, making their numerical comparisons invalid.
- WidgetCaps: Red-box image annotations perform as well as specifying the target widget with <loc> tokens in the prompt.
- Upscaling: RoPE interpolation provides no benefit when upscaling Stage2 image tokens.
- Zero-shot generalization: PaliGemma generalizes well to Objaverse 3D renders without explicit training or fine-tuning.
- MMVP: At 224px, PaliGemma achieves 47.3% paired accuracy on MMVP, versus 38.7% for GPT4-V and 40.7% for Gemini.
8. Conclusion
PaliGemma is presented as a small, open base VLM that performs strongly when transferred across diverse tasks. The authors position it as a starting point for instruction tuning and application-specific research.
- PaliGemma is a small, open base VLM that performs strongly across a broad range of transferred tasks.
- The base model is intended to support further instruction-tuning research, specific applications, and clearer separation between base models and fine-tunes.
Model development contributors
The paper lists its development contributors and situates PaliGemma within established vision-language model architectures and training approaches.
- Contributors: The listed contributors include Lucas Beyer, Andreas Steiner, André Susano Pinto, and other named collaborators.
- Contributors: The contributor list continues with Elisa Bandy, Joe Fernandez, Antonia Paterson, Jenny Brennan, and additional collaborators.
- Contributors: Further listed contributors include Divyashree Sreepathihalli, Alek Andreev, Arthur Zucker, Maryam Moosaei, Joelle Barral, and others.
- Background: VLMs can connect a vision encoder to a language model or use a decoder-only architecture for both modalities.
- Background: PaliGemma belongs to the class of models that train both vision and language components through multi-stage procedures.
B.7. COCOcap: COCO image captioning task
COCOcap evaluates image captioning with CIDEr, alongside zero-shot transfer variants and multilingual extensions. The section specifies prompts, training sizes, splits, and evaluation metrics for these tasks.
- COCOcap trains on 113287 images with five captions and evaluates using CIDEr on the validation split.
- A zero-shot COCOcap evaluation reuses the COCO-trained model and reports CIDEr on the validation split.
- COCO-35L trains on five captions per image across 35 languages and reports mean CIDEr across languages on the development split.
- Additional transfer tasks use task-specific prompts and report CIDEr, ANLS, exact-match accuracy, or accuracy on their stated splits.The supplied examples include VQA, document, video, visual reasoning, and counting-style tasks.
- Referring-expression segmentation uses location and segmentation token outputs, with mean intersection over union reported across RefCOCO-family test splits.
C. Image augmentations for RefCOCO
For RefCOCO-style structured prediction, extensive experiments find that simple square resizing can outperform more elaborate aspect-ratio-preserving augmentations. Label-smoothing and dropout prevent long-training overfitting while retaining the simple strategy.
- Simple resizing to a square 224 × 224 achieves the best and state-of-the-art performance in pretrained VLMs.This contradicts the common expectation that structured output tasks require image augmentation and aspect-ratio-preserving zooms.
- Training and inference preprocessing must be explored together because mismatched resizing strategies can reduce performance.Square training pairs well with square inference, whereas aspect-preserving inference benefits from corresponding training augmentations.
- Plain resizing overfits beyond 10 epochs, while cropping and zooming avoid this through their augmentation effects.The comparison concerns multiple runs at 5, 10, 20, and 30 epochs rather than training curves.
- Label-smoothing and dropout eliminate overfitting and let simple resizing match or outperform more complicated augmentation strategies.The final setting uses simple resizing with a longer training schedule, label-smoothing, and dropout.
E. Issues with published WidgetCaps numbers
The section identifies invalid or inconsistent published WidgetCaps comparisons caused by evaluation-set and CIDEr-computation errors. Corrected comparisons place PaliGemma’s results near the revised competing score.
- PaLI-X’s 153.0 and PaLI-3’s 159.8 were validation scores presented as test results, making them invalid comparisons with PaliGemma’s test-set 148.4.
- Re-running transfers with comparable train-on-train and evaluate-on-dev conditions gives PaliGemma CIDEr scores of 140.2 at 224 px and 155.2 at 448 px.
- Correcting ScreenAI’s CIDEr computation from one caption to all five changes its reported score from 167 to 156.4, close to PaliGemma’s 148.4.
- WidgetCap can mark the target widget by drawing a red box in the image rather than encoding box coordinates in the prompt, with comparable validation results.The reported scores are 135.99 CIDEr for the red box and 135.28 for coordinate tokens.
G. More details and results with Objaverse
Objaverse evaluates open-vocabulary recognition from multiple views of specialized 3D renders, using ScoreAgg to aggregate predictions. PaliGemma shows out-of-the-box 3D object understanding, while multitask transfer changes most task scores only modestly.
- Objaverse: Objaverse contains 800 k diverse, noisy or unannotated 3D models, with 44 k human-verified examples in the Objaverse-LVIS subset.The verified subset covers only 5% of examples and a restricted category set.
- Objaverse: ScoreAgg aggregates view predictions using logprobabilities and is substantially simpler and stronger than the ad-hoc CAP3D combination.
- Objaverse: PaliGemma demonstrates clear 3D object understanding out of the box, and its VQAv2 fine-tune performs better in the reported evaluation.The evaluation uses four VQA prompts, five sampled responses per view, and ScoreAgg aggregation.
- Multitask transfer: Multitask transfer evaluates a single model across 27 datasets using a uniformly mixed training setup.The mixture excludes some tasks, including multi-image tasks, for simplicity.
- Multitask transfer: Most multitask results change only slightly, although a few tasks change dramatically; the largest average loss comes from unified hyperparameters, followed by multitasking.The multitasking setup was not extensively tuned.