Source-linked AI summary
PaliGemma 2: A Family of Versatile VLMs for Transfer
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, Xiaohua Zhai
TL;DR
PaliGemma 2 addresses the need for transferable vision-language models across model sizes, resolutions, and diverse tasks. It combines Gemma 2 language models with a shared vision encoder, enabling broad transfer performance and state-of-the-art results on many newly explored applications.
Problem
Existing evaluation covered common captioning, VQA, video, and referring-expression tasks, leaving broader applications and controlled analysis of model size and resolution to investigate.
Method
The authors build open-weight VLMs by combining the PaliGemma vision encoder with Gemma 2 language models at three sizes and resolutions, using a shared three-stage training recipe for fine-tuning.
Results
PaliGemma 2 delivers strong transfer performance across captioning, VQA, and video tasks, with state-of-the-art results on many added tasks and slight gains over PaliGemma on 30+ transfer tasks.
Takeaways & Limitations
The model family supports transfer across domains including music, molecules, medical imaging, OCR, and other vision-language applications.
Takeaways & Limitations
The underlying Gemma 2 27B model is trained from scratch, whereas the 2B and 9B models are distilled.
Abstract
from arXiv · showhide
PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM) based on the Gemma 2 family of language models. We combine the SigLIP-So400m vision encoder that was also used by PaliGemma with the whole range of Gemma 2 models, from the 2B one all the way up to the 27B model. We train these models at three resolutions (224px, 448px, and 896px) in multiple stages to equip them with broad knowledge for transfer via fine-tuning. The resulting family of base models covering different model sizes and resolutions allows us to investigate factors impacting transfer performance (such as learning rate) and to analyze the interplay between the type of task, model size, and resolution. We further increase the number and breadth of transfer tasks beyond the scope of PaliGemma including different OCR-related tasks such as table structure recognition, molecular structure recognition, music score recognition, as well as long fine-grained captioning and radiography report generation, on which PaliGemma 2 obtains state-of-the-art results.
1. Introduction
PaliGemma 2 upgrades PaliGemma with Gemma 2 language models across three sizes and resolutions, producing open-weight VLMs designed for fine-tuning. The family improves transfer performance, broadens evaluated tasks, and enables controlled analysis of model size, resolution, and learning rate.
- Model family: PaliGemma 2 replaces PaliGemma’s language model with Gemma 2 models, yielding 3B, 10B, and 28B variants at 224px2, 448px2, and 896px2.The models retain the SigLIP vision encoder and use the same three-stage training recipe.
- Transfer performance: PaliGemma 2 slightly outperforms PaliGemma at matched size and resolution and improves substantially at larger model sizes across 30+ transfer tasks.These tasks include captioning, VQA, video, and referring-expression tasks.
- Scaling analysis: Comparable building blocks and a shared training recipe support controlled analysis of how model size and image resolution affect downstream transfer performance.The analysis also examines which tasks benefit more from resolution or from a larger language model.
- Scaling analysis: Larger models tend to have a lower optimal transfer learning rate.This relationship is identified through the family’s controlled transfer experiments.
- Expanded applications: PaliGemma 2 extends evaluation to OCR, table structure, molecular structure, music score, long caption, spatial reasoning, and radiography report tasks.It obtains state-of-the-art results on many of these tasks and is also benchmarked in low precision for CPU deployment.
- Model interface: Figure 1 shows image tokens produced at 224px2, 448px2, and 896px2 being projected, concatenated with text tokens, and completed autoregressively by Gemma 2.The corresponding token counts are 256, 1024, and 4096.
2. Related work
Prior work has studied several axes of VLM scaling and uses zero-shot, few-shot, and fine-tuning transfer paradigms. The paper identifies a lack of controlled joint analysis of image resolution and language-model size during fine-tuning.
- VLM development: VLMs have evolved from separately trained encoders or encoder-decoder systems toward architectures combining pretrained vision encoders with pretrained language models.Transfer is commonly performed through zero-shot, few-shot, or fine-tuning approaches, alongside instruction tuning for usability.
- Scaling studies: Previous studies have examined scaling through training data, compute, image resolution, model size, and component quality, especially the vision encoder.These studies cover multiple axes but do not establish the paper’s specific controlled comparison.
- Research gap: Prior work had not jointly studied image resolution and language-model size for transfer via fine-tuning in a controlled family of comparable models.Comparisons across language-model sizes often confound architecture and training recipes from different labs.
3. Model
PaliGemma 2 combines a pretrained SigLIP-So400m vision encoder with Gemma 2 language models and trains the resulting VLMs in multiple stages for transfer. Its interface concatenates projected visual embeddings with text prompts, while the training mixture covers captioning, OCR, VQA, detection, and segmentation.
- Architecture: PaliGemma 2 maps SigLIP-So400m visual embeddings into Gemma 2’s input space, combines them with a text prompt, and autoregressively samples predictions.A linear projection connects the vision encoder’s embeddings to the language model.
- Training recipe: The model is pretrained in three stages: multimodal joint training at 224px2, higher-resolution training at 448px2 and 896px2, then task-specific fine-tuning.Stage 2 upweights tasks benefiting from high resolution and increases output sequence length for long visual text.
- Optimization: Logits soft-capping is used in Stages 1 and 2 but omitted in Stage 3 because it worsened results for some transfer tasks.Learning rates are reduced relative to PaliGemma according to model size.
- Training data: The pretraining mixture includes captioning, grounded captioning, OCR, machine-generated VQA, detection, and instance segmentation.Many corresponding labels are machine generated and rely mostly on publicly available specialist models.
- Compute: Increasing resolution incurs a similar additional cost to increasing language-model size in the reported training setup.The 28B model at 896px2 uses TPUv5p, with an assumed 2.3× per-chip speed-up.
4. Experiments
Experiments examine how PaliGemma 2’s model size, image resolution, and learning rate affect transfer across academic and newly introduced tasks. Results show task-dependent scaling, strong performance on several specialized applications, and practical benefits from choosing resolution and model size appropriately.
- Experiment scope: PaliGemma 2 evaluates more than 30 academic benchmarks alongside new tasks spanning OCR, tables, molecules, music scores, long captions, spatial reasoning, and radiography.The experiments also include CPU deployment and low-precision variants.
- Model size and resolution: Resolution- and model-size increases both improve many tasks, but text, document, screen, and chart understanding mainly benefit from higher resolution.These benchmarks often contain images whose native resolution is substantially larger than 224px2.
- Model size and resolution: Tasks involving multilingual data or advanced visual reasoning mostly benefit from increasing language-model size rather than resolution.Examples include XM3600, AI2D, CountBenchQA, and NLVR2.
- Model size and resolution: Increasing model size from 3B to 10B often helps, whereas scaling further to 28B frequently yields only moderate or no additional improvement.The 28B model may be less transferable because its underlying Gemma 2 27B model was trained from scratch rather than distilled.
- Learning rate: Larger models tend to require lower optimal transfer learning rates, motivating smaller learning-rate sweeps as model size increases.PaliGemma 2 3B also generally has a smaller optimal transfer learning rate than PaliGemma.
- Comparison with PaliGemma: PaliGemma 2 slightly outperforms corresponding 3B PaliGemma models by 0.65 points at 224px2 and 0.85 points at 448px2 on average across 30+ benchmarks.The comparison uses the same resolution and model size.
- Resolution-sensitive tasks: OCR quality drops substantially at lower resolution, while increasing model size does not improve it; table recognition likewise regresses slightly at lower resolution and gains no benefit from larger models.For optical music score recognition, error rates decrease with resolution and are best at 896px2, while increasing size from 3B to 10B yields no further reduction.
5. Conclusion
PaliGemma 2 introduces an open-weight family spanning model sizes and input resolutions, with strong transfer performance across diverse tasks and domains.
- PaliGemma 2 presents open-weight models spanning a broad range of model sizes and input resolutions.
- The family achieves strong transfer performance across captioning, VQA, and video tasks.
- Larger variants significantly improve over PaliGemma for users with larger compute budgets.
- PaliGemma 2 extends transfer applications to music, molecules, and medical imaging.
Contributions and Acknowledgments
The listed contributors include Andreas Steiner, André Susano Pinto, Michael Tschannen, and additional collaborators.
- Andreas Steiner, André Susano Pinto, and Michael Tschannen are listed among the contributors.
- Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, and Matthias Minderer are also listed.
- The contributor list further includes Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, and Emanuele Bugliarello.
A. Tasks
This section presents examples and training details for added transfer tasks spanning document understanding, molecular and music recognition, radiography, and related applications.
- The added-task examples include Total-Text text recognition, FinTabNet table structure recognition, molecular rendering with SMILES output, and pianoform transcription.
- Radiography examples pair MIMIC-CXR indications and reports with PaliGemma 2 predictions.
- For one task, training uses 15k steps, batch size 256, learning rate 10^-5, label smoothing 0.1, and 896px2 resolution.
- Other task setups vary output lengths, label smoothing, learning rates, preprocessing, and training duration.
B.5. Generating long, fine-grained captions (DOCCI)
The DOCCI procedure fine-tunes PaliGemma 2 for long captions and evaluates factual grounding sentence by sentence through human ratings.
- DOCCI caption models are selected using perplexity and human evaluations of sentence-level factual alignment with images.
- Raters classify each highlighted sentence as Entailment, Neutral, Contradiction, or Nothing to assess.
- Each sentence receives ratings from five individuals, with majority agreement used as the result.
- The overall binary agreement is 0.8407, and Contradiction and Neutral are grouped as Non-entailment.
C. Object detection
PaliGemma is evaluated on classical object detection using sequence-based transfer on COCO and DocLayNet. Performance improves with resolution and, generally, language-model size, but specialized detectors remain stronger.
- PaliGemma is transferred to MS COCO and DocLayNet to evaluate classical object detection.
- The transfer strategy lists annotated boxes and class names, then adds noise boxes to train the model to output more detections.Noise-box coordinate tokens receive no loss, while <noise> class tokens are trained normally.
- COCO performance strongly depends on resolution, with small but consistent improvements from better language models.At 896px2, performance is roughly on par with prior sequence-based approaches but below ViTDet.
- DocLayNet performance likewise increases with resolution and Gemma 2 size, although Gemma 1 performs on par with Gemma 2.YOLOv11 reaches 79.5 mAP and outperforms the transferred model on this task.
- Classical detection remains challenging for general-purpose VLMs, possibly because Average Precision rewards prediction quantity and confidence accuracy beyond the language-modeling objective.Task-specific reward fine-tuning is proposed as a possible remedy but is outside the presented transfer approach.
D. Ethics and Safety
PaliGemma 2 is evaluated for child safety, content safety, representational harms, and toxicity-related outcomes in captioning and VQA settings. The reported toxicity and profanity levels are low across slices, with comparable performance across models.
- Safety evaluations cover child safety, content safety, and representational harms using image captioning and VQA prompts.
- The Perspective API uses a threshold greater than 0.8 to detect toxicity, profanity, and related issues in captions generated from FairFace images.Results report maximum and median subgroup values for perceived gender, ethnicity, and age.
- Observed toxicity and profanity levels are low across all evaluated slices and models.
- All PaliGemma 2 models perform comparably in the reported safety evaluations.
E. Detailed results
The detailed results examine scaling, learning-rate selection, and comparisons between PaliGemma variants across resolutions. Figures and tables summarize relative improvement, error reduction, repeated fine-tuning results, learning-rate sweeps, and model comparisons.
- Figure 13 presents relative improvement and error reduction when changing model size or resolution.Error reduction is identified as a better improvement metric for saturated benchmarks; CIDEr is omitted because it lacks clear percentage normalization.
- Table 13 reports means and standard deviations over five fine-tuning runs across more than 30 academic tasks at 224px2 and 448px2.Learning rate is selected per model size using validation splits.
- Tables 14’s learning-rate sweeps cover various tasks and model sizes at 224px2, with selection based on validation rather than zero-shot results.
- Table 15 compares PaliGemma 3B with PaliGemma 2 3B at 224px2 and 448px2 resolutions.