PaperlayerView exact revision on arXiv

Source-linked AI summary

PaliGemma: A versatile 3B VLM for transfer

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, Xiaohua Zhai

arXiv:2407.07726v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Whether a compact VLM can serve as a versatile base across diverse tasks remains an important research question. PaliGemma combines SigLIP and Gemma into a sub-3B VLM, achieving state-of-the-art performance across a wide variety of benchmarks.

  • Problem

    The research question is whether a compact VLM can serve as a versatile base across standard and specialized tasks, given prior saturation on standard image classification and retrieval.

    Source: §1 · p. 1

  • Method

    PaliGemma combines 400 M SigLIP and the 2 B Gemma models into a sub-3 B VLM.

    Source: §1 · p. 1

  • Results

    PaliGemma achieves state-of-the-art performance across a wide variety of benchmarks.

    Source: §8 · p. 14

  • Takeaways & Limitations

    The authors position the open base model as a useful starting point for research in instruction tuning and specific applications.

    Source: §8 · p. 14

    Few-example fine-tuning remains an open stability problem, with variance even across different batch orders for the same examples.

    Source: §6.3 · p. 13

Abstract

from arXiv · show

PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model. It is trained to be a versatile and broadly knowledgeable base model that is effective to transfer. It achieves strong performance on a wide variety of open-world tasks. We evaluate PaliGemma on almost 40 diverse tasks including standard VLM benchmarks, but also more specialized tasks such as remote-sensing and segmentation.

1. Introduction

PaliGemma is an open, sub-3 B vision-language model combining SigLIP and Gemma, designed as a versatile base that performs across standard and specialized tasks.

1. Introduction

  • PaLI-3 [25], using SigLIP pretraining and curated multimodal data, matches the 10x larger PaLI-X [24] and 100x larger PaLM-E [36] across most benchmarks.Earlier PaLI models scaled to 17 B [23], while PaLI-X and PaLM-E increased model sizes but standard image classification and retrieval performance saturated.Source: §1 · p. 1
  • PaliGemma combines 400 M SigLIP and 2 B Gemma into a sub-3 B VLM with performance comparable to PaLI-X, PaLM-E, and PaLI-3.The model uses the pretrained 2 B Gemma version.Source: §1 · p. 1
  • Gemma is an open decoder-only language-model family, and PaliGemma uses its pretrained 2 B version.Gemma models are available in 2 B and 7 B sizes, both pretrained and instruction fine-tuned.Source: §1 · p. 1
  • The work aims for a versatile base VLM, with state-of-the-art results spanning standard benchmarks and specialized remote-sensing, video, and segmentation tasks.Examples include COCO captions, VQAv2, InfographicVQA, Remote-Sensing VQA, TallyVQA, video captioning and QA, and referring expression segmentation.Source: §1 · p. 1

2. Related work

The section traces vision-language models from web-scale caption-embedding classification to generative captioning and question answering, then situates PaliGemma among scaled and instruction-tuned systems. It presents PaliGemma as an open base model without instruction tuning that addresses questions about what matters in VLMs.

2. Related work

  • CLIP [94] and ALIGN [49] extended large-scale classification pretraining by using web captions and language encoders to open classification and retrieval vocabularies.This first generation built on ConVIRT [135] and VirTex [32].Source: §2 · p. 1
  • A second generation unified captioning and question answering through generative encoder-decoder modeling [111] [138].The passage compares this shift to T5 in language.Source: §2 · p. 1
  • Flamingo, BLIP-2 [62], and PaLI [23] further scaled VLMs, while recent works [70] added instruction tuning to make raw models more user-friendly.Source: §2 · p. 2
  • PaliGemma is an open base VLM without instruction tuning, and its report addresses questions about what matters in VLMs raised by systematic studies [81] [107].The passage points to Appendix A for more discussion.Source: §2 · p. 2

3. Model

PaliGemma takes images and a task prompt and autoregressively produces text, enabling varied vision-language tasks; its training builds a transferable base model through multimodal pretraining, resolution increases, and task-specific fine-tuning.

3.1. Architecture

  • PaliGemma combines a SigLIP ViT-So400m encoder, Gemma-2B decoder, and linear projection, with full attention over image and prefix tokens and autoregressive attention on the suffix.Images are resized to 224, 448, or 896 pixels, yielding 256, 1024, or 4096 image tokens per variant.Source: §3.1 · pp. 2–3, Figure 2 · p. 3

3.2.1. Stage0: Unimodal pretraining

  • Stage0 reuses public SigLIP and Gemma checkpoints without custom unimodal pretraining, continuing the push toward comparable vision-language performance with fewer than 3B total parameters.The image encoder is the smaller, strong ViT-So400m, and Gemma-2B is selected as a size-performance balance.Source: §3.2.1 · pp. 3–4

3.2.2. Stage1: Multimodal pretraining

  • Stage1 trains the whole model, including the image encoder, on a broad multimodal task mixture to build transferable skills rather than optimize only out-of-the-box usability.It uses a slow linear learning-rate warm-up for the image encoder and runs for 1 billion examples at 224px resolution with 128 text tokens.Source: §3.2.2 · p. 4

3.2.3. Stage2: Resolution increase

  • Stage2 extends image understanding from 224px to 448px and 896px using 50M and 10M additional examples, respectively, with more high-resolution tasks and 512 text tokens.The higher resolutions target tasks such as small-object detection and segmentation, and reading small text in charts, infographics, and documents.Source: §3.2.3 · pp. 4–5

3.2.4. Stage3: Transfer

  • Stage3 fine-tunes base checkpoints for specialized tasks, new input formats, instruction or chat use, and a multi-task mix, using a unified transfer recipe with few hyperparameters.The authors report that good results can be achieved starting with the suggested initial hyperparameter values.Source: §3.2.4 · p. 5

3.2.5. Pretraining task mixture

  • The pretraining mixture includes captioning, OCR, generated VQA and question generation, and open-world detection and instance segmentation objectives.Captioning uses datasets including WebLI in over 100 languages and CC3M-35L; generated VQA and question generation use CC3M-35L in 35 languages.Source: §3.2.5 · p. 6
  • Pretraining uses grounded captioning and avoids tasks generated by larger commercial VLMs; the paper also describes removing near-duplicates of transfer-task images to better measure transfer to new tasks.Each task receives a unique prefix to reduce conflicting learning signals across skills.Source: §3.2.5 · pp. 5–6

3.2.6. Other pretraining details

  • Training achieved 55% model FLOPS utilization and 5189 tokens/second/device; parameters and optimizer state were kept in float32, while inference also worked with bfloat16 parameters.The model was trained in the open-source big_vision codebase on Cloud TPUv5e.Source: §3.2.6 · pp. 6–7

4. Results

PaliGemma is fine-tuned on more than 30 academic benchmarks whose tasks and images were excluded from pretraining. The section reports final test results and describes the tuning and evaluation choices used to obtain them.

4. Results

  • PaliGemma is evaluated after fine-tuning on more than 30 academic benchmarks, with their tasks and images excluded from pretraining.Table 1 presents results for the transferred model.Source: §4 · p. 7, Table 1 · p. 7
  • Hyper-parameters are selected using validation data or a held-out minival set, then the model is retrained on full training and validation data for final test results.The search begins from recommended values and may extend or supersample promising ranges; it does not necessarily use a full cross-product.Source: §4 · p. 7
  • A single recommended value for each hyper-parameter, without exploration, works almost as well on most tasks.Source: §4 · pp. 7–8
  • For tasks other than video, results are reported at at least two resolutions to show which tasks benefit from increased resolution.Source: §4 · p. 8
  • Data augmentation showed no significant benefit; inputs are simply resized to a fixed square resolution, including for RefCOCO segmentation.Source: §4 · p. 8

5. Ablations

The ablations show that pretraining duration, the learning objective, token initialization, and which model components are tuned all affect transfer or pretraining behavior. They also examine connector and image-encoder choices, resolution-specific training, windowing, and Stage2 mixture weights.

5.2. Causal masking and learning objective

  • Prefix-LM outperformed alternative masking, prefix-token loss reduced average transfer performance, and task-prefixes had no noticeable transfer effect.Task-prefixes reduced uncertainty during pretraining for tasks not perfectly disambiguated by the prefix; overall, prefix-LM with task-prefixes and suffix-only supervision was an effective objective.Source: Figure 5 · p. 8, §5.2 · p. 9

5.1. Multimodal pretraining duration

  • Skipping Stage1 was the worst setting, while shorter pretraining generally hurt; 100 M examples was 10x shorter without significantly hurting any task.Ablations generally used the main-model setup with Stage1 shortened to 100 M examples and reported validation rather than withheld test results.Source: §5.1 · p. 8, §5 · p. 8

5.3. New token initialization

  • Gaussian initialization yielded better end-of-pretraining perplexities and RefCOCO segmentation MIoU than AvgEmb, whose initial perplexity advantage vanished after a thousand steps.The Gaussian initialization used σ=0.02.Source: Figure 6 · p. 9, §5.3 · p. 9

5.4. To freeze or not to freeze?

  • Tuning rather than freezing the image encoder improved predictability on spatial-understanding tasks, while freezing the language model or resetting model parts significantly worsened results.After transfer, tuning the image encoder made no difference relative to keeping it frozen.Source: Figure 7 · p. 10, §5.4 · p. 10

5.5. Connector design

  • With all weights tuned, linear versus MLP connectors scored 77.2 versus 77.1; with everything frozen but the connector, they scored 70.7 versus 69.7.Removing SigLIP and passing raw image patches to Gemma still worked but lagged behind; its scaling with pretraining duration appeared potentially promising, though it was less training-efficient because it could not reuse vision components.Source: §5.5 · p. 10, Figure 8 · p. 10, §5.6 · pp. 10–11

5.7.1. Resolution or sequence length?

  • For tasks affected by resolution, increased image information and longer sequence length contributed roughly equally to performance gains.Among the evaluated approaches, Stage2 at increased resolution without tricks performed best; windowing worked better than increasing resolution only during transfers.Source: Figure 9 · p. 11, §5.7.1 · p. 11, Figure 10 · p. 11

5.7.2. Need resolution-specific checkpoints?

  • A checkpoint pretrained at the target resolution performed better than transferring a checkpoint pretrained at another resolution.Transferring a 448 px checkpoint at 224 px performed significantly worse than using a 224 px checkpoint; transferring a 224 px checkpoint at 448 px improved results but still lagged behind a native 448 px checkpoint. Without flexible-resolution methods, the authors recommend extended pretraining at each resolution and separate checkpoints.Source: Figure 10 · p. 11, §5.7.2 · pp. 11–12

5.7.3. To resize or to window?

  • Windowing lagged behind a native 448 px model and yielded at most a 5% training speedup; Stage2 mixture reweighting worsened only three tasks, with other results within per-task variance.The authors describe windowing as a possible higher-resolution transfer option when no suitable checkpoint exists and continued pretraining is infeasible. Changing mixture weights helped a little, and precise ratios may matter less for a base model intended for fine-tuning than for zero-shot sampling.Source: §5.7.3 · p. 12, §5.7.4 · p. 12

6. Transferability

PaliGemma transfers repeatably across reruns, works well with a simple shared hyper-parameter setup on most tasks, and can reach near full-data performance with relatively few examples. Few-example transfer is variable for some tasks, but 4,000 examples bring most tasks within 10% of full-data scores.

6.1. Repeatability (variance)

  • Standard deviation across five transfer reruns is generally very small, and three Stage1 reruns show similarly low variance, indicating repeatable transfer and pretraining.Source: §6.1 · pp. 12–13

6.2. Transfer hyper-parameter sensitivity

  • A single simple hyper-parameter setup works well for most tasks, while RefCOCO and SciCap benefit from more epochs, label smoothing, and dropout.The shared setup used 224px resolution, learning rate 1e-5, batch size 256, no weight decay, and no frozen parameters; epochs were task-dependent and capped at 10.Source: §6.2 · p. 13, Table 2 · p. 13

6.3. Transfer with limited examples

  • Most tasks reach within 10% of full-data scores with 4,000 examples and within 20% with 256; 64 examples can suffice for prototyping.Each setting used five seeds, and limited-example performance showed high variance on some tasks; RefCOCO mIOU varied by 10%–30%.Source: Figure 12 · p. 13, §6.3 · p. 13

7. Noteworthy tidbits

The section reports findings on segmentation preprocessing, counting evaluation, WidgetCaps comparisons, widget annotation, upscaling, 3D generalization, and MMVP performance.

Plain resize to square for segmentation

  • Simple 224×224 square resizing worked just as well as aspect-ratio-preserving zoom-and-crop augmentations for segmentation.The paper reports this finding in Appendix C.Source: §7 · p. 13

Counting: introducing CountBenchQA

  • CountBenchQA, a VLM-ready version of CountBench [88], was introduced because skewed counts and varying image quality made TallyQA inadequate for assessing current VLM counting ability.The paper provides further details in Appendix D.Source: §7 · p. 14

Issues in published WidgetCaps numbers

  • At least three prior works’ WidgetCaps evaluations had issues that rendered their numerical comparisons invalid.The paper discusses these issues in Appendix E.Source: §7 · p. 14

Image annotations work as well as prompts

  • Marking a widget with a red box in the image gave the same captioning results as indicating it with <loc> tokens in the prompt.The paper provides further details in Appendix F.Source: §7 · p. 14

RoPE interpolation unnecessary for upscaling

  • Interpolating image-token RoPE position indices in Stage2 to preserve Stage1 semantics provided no benefit.The paper reports this result for its upscaling experiments.Source: §7 · p. 14

Zero-shot generalization to 3D renders

  • Without explicit training or fine-tuning on 3D renders, PaliGemma generalized to renders from Objaverse.The paper describes this as zero-shot generalization and provides further details in Appendix G.Source: §7 · p. 14

Our MMVP result is SOTA by a large margin

  • 47.3% paired accuracy at 224px exceeded GPT4-V (38.7%) and Gemini (40.7%), while all other models, including LLaVa, scored below chance.The paper describes this MMVP result as state of the art by a large margin.Source: §7 · p. 14

8. Conclusion

PaliGemma is a small, open base VLM that transfers well across a broad range of tasks, showing that smaller VLMs can achieve state-of-the-art performance across diverse benchmarks. Its release without instruction tuning is intended to support further research and clearer separation between base models and fine-tunes.

  • PaliGemma shows that smaller VLMs can provide state-of-the-art performance across a wide variety of benchmarks.Source: §8 · p. 14
  • As a small, open base VLM, PaliGemma shines when transferred to a broad range of tasks.Source: §8 · p. 14
  • Released without instruction tuning, PaliGemma is intended as a starting point for research on instruction tuning and specific applications, and to encourage clearer separation of base models and fine-tunes.Source: §8 · p. 14

Model development contributors … B.6. ChartQA: VQA about charts with logical reasoning

The appendix names contributors across model development and describes related VLM approaches alongside example transfer-task formats, training data, and evaluation metrics. It distinguishes jointly trained vision-language models from connector-based and decoder-only approaches.

Model development contributors — Core Contributors; Go-to-Market; Kaggle; Documentation; Ethics and Safety; Vertex AI

  • Core contributors include Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, and Xiaohua Zhai.Go-to-Market: Kat Black, Luiz Gustavo Martins, Glenn Cameron, Raj Gundluru, and Manvinder Singh; Kaggle: Meg Risdal, Nilay Chauhan, Nate Keating, and Nesh Devanathan; Documentation: Elisa Bandy and Joe Fernandez; Ethics and Safety: Antonia Paterson, Jenny Brennan, Tom Eccles, Pankil Botadra, and Ben Bariach; Vertex AI: Lav Rai, Minwoo Park, Dustin Luong, Daniel Vlasic, Bo Wu, and Wenming Ye.Source: Core Contributors · p. 27, Go-to-Market · p. 27, Kaggle · p. 27

Model development contributors — Keras; Gemma Model; Hugging Face Partners; Nvidia Partners; Executive Sponsors

  • Additional contributors are listed for Keras, Gemma Model, Hugging Face and Nvidia Partners, and Executive Sponsors.Keras: Divyashree Sreepathihalli, Kiranbir Sodhia, Gabriel Rasskin, Matthew Watson, and Varun Singh; Gemma Model: Alek Andreev, Armand Joulin, Surya Bhupatiraju, and Minh Giang; Hugging Face Partners: Arthur Zucker, Lucain Pouget, Merve Noyan, Omar Sanseviero, Pablo Montalvo, and Pedro Cuenca; Nvidia Partners: Maryam Moosaei, Fangzhou Mu, Santosh Bhavani, Anjali Shah, Vladislav Kozlov, Dong Meng, Niaz Syed, Chintan Patel, Ankit Patel, Marta Stepniewska-Dziubinska, Anna Warno, Xinqi Wang, David Tamok, Steven Baughman, Sandip Bhaskar, and Jason Dudash; Executive Sponsors: Joelle Barral and Zoubin Ghahramani.Source: Keras · p. 27, Gemma Model · p. 28, Hugging Face Partners · p. 28

A. More related work

  • Related VLMs either connect vision and language components or use decoder-only architectures for both modalities.Connector approaches include lightweight adapters, resamplers, and Q-Former; examples include Flamingo, BLIP2 [62], LLaVA [70], and CogVLM [118]. PaliGemma's connector ablation is discussed in Section 5.5.Source: A. More related work · p. 29
  • PaliGemma jointly trains vision and language components, while the paper also reports early Fuyu-style decoder-only ablations.Prior work includes multi-stage training [10] [23] [24] [25] [26], PaLI's three-stage resolution scaling [23] [24] [25], and unified models such as Unified-IO 2 [72]; Fuyu processes image patches as projected tokens, while CM3 [2] [3] [129] and Chameleon use discrete image tokens. The decoder-only ablation appears in Section 5.6.Source: A. More related work · p. 29

B. Tasks

  • Each transfer task includes an example as presented to the model, with its image resized to 224 × 224 pixels.The appendix frames the examples as illustrating what can be recognized at that resolution.Source: B. Tasks · p. 30

B.1. ActivityNet-CAP; B.2. ActivityNet-QA; B.3. AI2D

  • ActivityNet-CAP uses 44,775 examples and CIDEr; ActivityNet-QA uses 43,130 examples and exact-match accuracy; AI2D uses 12,413 examples and exact-match accuracy.ActivityNet-CAP combines train and validation splits and evaluates on the test split (Krishna et al. [57]); ActivityNet-QA also combines train and validation splits and evaluates on test (Yu et al. [130]); AI2D trains on its training split and evaluates on test (Aniruddha Kembhavi).Source: §B.1 · p. 30, §B.2 · p. 30, §B.3 · p. 30

B.4. AOKVQA-DA; B.5. AOKVQA-MC; B.6. ChartQA

  • AOKVQA-DA and AOKVQA-MC each use 18,201 examples, evaluated by exact-match accuracy and accuracy, respectively; ChartQA uses relaxed-match accuracy.AOKVQA-DA uses train and validation splits and test-server evaluation (Schwenk et al. [98]); AOKVQA-MC uses the same split sizes and test-server evaluation (Schwenk et al. [98]); ChartQA trains on 30,219 human and augmented train and validation examples, reporting ChartQA-human and ChartQA-aug test splits (Masry et al. [78]).Source: §B.4 · p. 31, §B.5 · p. 31, §B.6 · p. 31

B.7. COCOcap: COCO image captioning task … B.39. WidgetCap: Descriptions for Mobile User Interface Elements

These sections catalog training and evaluation setups across captioning, question answering, reasoning, segmentation, and specialized domains. They specify dataset sizes, evaluation splits, metrics, and several zero-shot evaluations, but provide no benchmark scores here.

B.7. COCOcap: COCO image captioning task; B.8. NoCaps: Novel object captioning; B.9. COCO-35L: COCO captions translated in 35 languages; B.10. XM3600: caption geographically-diverse images in 36 languages

  • Captioning setups cover COCOcap (CIDEr), zero-shot NoCaps (CIDEr), 35-language COCO-35L (mean CIDEr), and XM3600 (mean CIDEr across 36 languages).COCOcap uses 113287 images with five captions each from train+restval; COCO-35L uses 113287 images with five captions in each of 35 languages. NoCaps evaluates the COCOcap-trained model zero-shot, and XM3600 evaluates the COCO-35L-trained model zero-shot; the reported splits are val for COCOcap and NoCaps, and dev for COCO-35L and XM3600.Source: §B.7 · p. 31, §B.8 · p. 32, §B.9 · p. 32

B.11. DocVQA: VQA on document images; B.12. GQA: VQA on image scene graphs; B.13. xGQA: Cross-lingual VQA; B.14. InfoVQA: VQA on infographics

  • Document, scene-graph, cross-lingual, and infographic VQA use ANLS, exact-match accuracy, exact-match accuracy, and ANLS, respectively.DocVQA trains on 44812 examples and reports ANLS on the test server; GQA uses 1075062 examples and exact-match accuracy on testdev balanced. xGQA evaluates the GQA-trained model zero-shot on test zero-shot splits, while InfoVQA uses 26747 examples and ANLS on the test server.Source: §B.11 · p. 32, §B.12 · p. 33, §B.13 · p. 33

B.15. MSRVTT-CAP: Open-domain short video captioning; B.16. MSRVTT-QA: Open-domain sort video question answering; B.17. MSVD-QA: Answering questions about short video segments of events; B.18. NLVR2: Reasoning about natural language grounded in photographs; B.19. MaRVL: Reasoning about multilingual language grounded in multicultural photographs

  • Video captioning and QA use CIDEr or exact-match accuracy, while NLVR2 and zero-shot MaRVL assess reasoning with accuracy.MSRVTT-CAP uses 4965 examples and CIDEr on test; MSRVTT-QA uses 121646 examples and exact-match accuracy on test; MSVD-QA uses 29749 examples and exact-match accuracy on test. NLVR2 trains on 93355 examples and reports test accuracy, while MaRVL evaluates the NLVR2-trained model zero-shot on test splits using accuracy.Source: §B.15 · p. 33, §B.16 · p. 34, §B.17 · p. 34

B.20. OCR-VQA: VQA by reading text in images; B.21. OKVQA: Outside knowledge VQA; B.22. RefCOCO_seg: Referring expression segmentation; B.23. RSVQA-hr: VQA for remote sensing (high res); B.24. RSVQA-lr: VQA for remote sensing (low res)

  • Text-reading and knowledge VQA use exact-match accuracy or ANLS; referring segmentation uses mIoU, and remote-sensing VQA reports accuracy.OCR-VQA trains on 901717 examples and uses exact-match accuracy on test; OKVQA trains on 9009 and uses exact-match accuracy on val. RefCOCO_seg trains on 24407 examples and reports mean intersection over union (mIoU) on all test splits. RSVQA-hr uses 432239 examples and mean accuracy across question types on test and test2; RSVQA-lr uses 47173 and average accuracy across question types on the non-numeric test split.Source: §B.20 · p. 35, §B.21 · p. 35, §B.22 · p. 35

B.25. SciCap: Captions for scientific figures; B.26. ScienceQA: Science question answering; B.27. Screen2Words: Mobile UI summarization; B.28. ST-VQA: Scene text VQA; B.29. TallyQA: Complex counting questions

  • Scientific-figure captioning uses CIDEr; science QA uses exact-match accuracy; mobile UI summarization uses CIDEr; scene-text VQA uses ANLS; counting uses accuracy.SciCap uses 120188 examples and evaluates first-sentence, no-subfigure captions on test; ScienceQA uses 8315 examples and test exact-match accuracy. Screen2Words uses 18107 examples and test CIDEr; ST-VQA uses 26074 examples and test-server ANLS; TallyQA uses 249318 examples and reports test accuracy on simple and complex splits.Source: §B.25 · p. 36, §B.26 · p. 36, §B.27 · p. 37

B.30. CountBenchQA: Evaluate counting in a structured, controlled way; B.31. TextCaps: Image captioning with reading comprehension; B.32. TextVQA: Visual reasoning based on text in images; B.33. VATEX-CAP: Broad video captioning; B.34. VizWizVQA: VQA from people who are blind

  • CountBenchQA evaluates TallyQA-trained models zero-shot with exact match; TextCaps and VATEX-CAP use CIDEr, while TextVQA and VizWizVQA use test-server evaluation.CountBenchQA uses the full test split; TextCaps trains on 21953 examples and evaluates CIDEr on test, and VATEX-CAP uses 24850 examples with test CIDEr. TextVQA uses 39602 examples and test-std on the test server; VizWizVQA uses 24842 examples and test-std on the test server.Source: §B.30 · p. 37, §B.31 · p. 38, §B.32 · p. 38

B.35. VQAV2: Visual Question Answering; B.36. MMVP: Hard questions about CLIP-blind image pairs; B.37. POPE: Object presence as yes/no VQA (object hallucination)

  • VQAV2 uses test-server and local-minival evaluation; zero-shot MMVP uses paired accuracy, and zero-shot POPE averages yes/no accuracy across three splits.VQAV2 trains on 658111 examples; its local minival metric is computed on a local split of 10k validation examples. MMVP and POPE both evaluate models trained for VQAV2 zero-shot; POPE averages accuracies on equally sized random, popular, and adversarial splits.Source: §B.35 · p. 39, §B.36 · p. 39, §B.37 · p. 39

B.38. Objaverse Multiview: view-consistent 3D object annotation; B.39. WidgetCap: Descriptions for Mobile User Interface Elements

  • Objaverse Multiview evaluates zero-shot predictions by cosine similarity, and WidgetCap evaluates UI-element descriptions with CIDEr.Objaverse Multiview scores image-level predictions with Universal Sentence Encoder v4 cosine similarity, then uses a score-weighted average across views; it covers 44k objects, 1100 categories, eight rendered views per object, and four prompts per object view. WidgetCap uses 44704 examples and overlays a red bounding box on the widget to be captioned.Source: §B.38 · p. 40, §B.39 · p. 40

C. Image augmentations for RefCOCO · D. Introducing CountBenchQA

For RefCOCO, simple square resizing performs strongly when training duration, regularization, and inference resizing are considered together. CountBenchQA provides a manually verified counting dataset in question-answer format, addressing issues in TallyQA while retaining some availability limitations.

C. Image augmentations for RefCOCO

  • 224×224 square resizing achieved best, state-of-the-art RefCOCO performance; with longer training, label-smoothing and dropout removed overfitting and matched or beat complex augmentations.Figure 14 reports that label-smoothing and dropout hurt the 224 px, 10-epoch setting but help the 448 px, 30-epoch setting.Source: Figure 14 · p. 41, C. Image augmentations for RefCOCO · p. 41
  • Training and inference resizing strategies need to be explored together: square resizing works well at both stages, whereas square training resizing pairs poorly with aspect-preserving inference.At inference, square-resized images can have their output logits resized back to the original resolution for evaluation.Source: C. Image augmentations for RefCOCO · p. 41
  • Plain resizing overfits beyond 10 epochs, unlike cropping and zooming; label-smoothing and dropout eliminate this overfitting, enabling simple resizing with an increased schedule.The paper’s final RefCOCO setting uses simple image resizing, a longer training schedule, label-smoothing, and dropout.Source: C. Image augmentations for RefCOCO · p. 41, Figure 14 · p. 41

D. Introducing CountBenchQA

  • TallyQA has a ground-truth distribution skewed toward small counts and noisy labels, with validation and test splits lacking further verification or cleanup.Source: D. Introducing CountBenchQA · p. 42
  • CountBench [88] contains 540 manually verified image-caption-count examples from LAION-400M, addressing TallyQA’s label-distribution and verification shortcomings.Source: D. Introducing CountBenchQA · p. 42
  • CountBenchQA converts the examples into questions using manually extracted object classes; it has 60 examples per count from 2 through 10, with no 0 or 1 counts.Overly specific or difficult object-class descriptions were replaced with simpler terms, and three questions deviated from the standard template to avoid ambiguity.Source: D. Introducing CountBenchQA · p. 42
  • Only 490 of CountBench’s 540 image files remain available; the paper reports metrics on the full split and has not recomputed them since.Source: D. Introducing CountBenchQA · p. 42

E. Issues with published WidgetCaps numbers · F. Image annotations work as well as prompts

The paper identifies errors in prior WidgetCaps comparisons and finds that image annotations perform as well as prompt-based box coordinates. It also reports comparable-setting validation results for PaliGemma and corrects a published ScreenAI score.

E. Issues with published WidgetCaps numbers

  • PaLI-X’s 153.0 and PaLI-3’s 159.8 WidgetCaps scores are validation-set results, not test-set results, making comparison with PaliGemma’s 148.4 test score invalid.The paper says the prior results were presented while implying they were test-set results.Source: E. Issues with published WidgetCaps… · p. 43
  • 140.2 CIDEr at 224 px and 155.2 at 448 px are PaliGemma’s validation results when trained on train and evaluated on dev, between PaLI-X and PaLI-3.The paper notes these results use much smaller model size and resolution.Source: E. Issues with published WidgetCaps… · p. 43
  • 156.4 CIDEr is ScreenAI’s recomputed WidgetCap score using all five ground-truth captions, versus 167 originally reported, close to PaliGemma’s 148.4.The paper attributes the original score to a computation mistake that used only one of the five captions.Source: E. Issues with published WidgetCaps… · p. 43

F. Image annotations work as well as prompts

G. More details and results with Objaverse … K.9. Transfer with limited examples

The appendix reports PaliGemma’s out-of-the-box 3D object understanding and VQAv2 fine-tune gains, alongside multitask transfer results and per-task analyses of pretraining and transfer choices. It also documents inference, hyperparameter selection, and limited-example evaluation.

G. More details and results with Objaverse

H. Multitask transfer

  • Multitasking causes dramatic changes on a few tasks, while most tasks win or lose little; average regret is largest from unified hyperparameters, then multitasking, and smallest from removing task prefixes.The setup transfers to a uniformly mixed set of 27 datasets, with each task run for 10 epochs; it was not tuned much.Source: Table 4 · p. 46, H. Multitask transfer · p. 46

I. Inference

  • Megatron-style sharding splits parameters and activations to process one example across accelerators and reduce per-device memory reads, unlike FSDP sharding.The inference measurements use a TPUv3 with 8 devices, prefilling 512 tokens and a 640-token cache.Source: Table 5 · p. 47, §I · p. 47

J. Hyper-parameters

  • Hyperparameters were tuned on validation data or a training-set minival, usually at 224px, while zero-shot settings could underperform on shifted test distributions.Higher-resolution settings were carried over or tuned slightly; zero-shot evaluations reused hyperparameters chosen for the fine-tune task.Source: J. Hyper-parameters · p. 48

K.1. Pretraining duration

  • Longer pretraining benefits most tasks, with remote-sensing tasks a notable exception because their image distribution differs substantially.Figure 16 presents per-task results for pretraining duration.Source: Figure 16 · p. 50

K.2–K.4. Masking and learning objective; To freeze or not to freeze?; Image encoder: with or without?

  • Per-task analyses compare learning objectives and task prefixes, freezing and resetting patterns, and use of an image encoder versus a Fuyu-style approach.The supplied figures identify these comparisons but do not provide specific outcome values.Source: Figure 17 · p. 51, Figure 18 · p. 52, Figure 19 · p. 53

K.5–K.6. Resolution or sequence length?; Resolution-specific checkpoints and windowing

  • Figures 21 and 22 show per-task results when the 448px model’s input is first resized to 224px, distinguishing resolution-insensitive from resolution-sensitive tasks; Figure 23 shows that, with Stage 2 using Stage 1’s exact pretraining task mixture, most result changes are below 2 standard-deviations of re-runs.Source: Figure 21 · p. 55, Figure 22 · p. 56, Figure 23 · p. 57

K.8. Transfer with simple hyper parameters

  • With simple hyperparameters, transfer works out of the box for most tasks; limited-example results report the maximum after sweeping learning rates, epochs, and batch size.The figures show per-task results for both simple hyperparameters and limited-example transfer.Source: Figure 24 · p. 58, Figure 25 · p. 59

Related papers

Loading 2407.07726v2…