PaperlayerView exact revision on arXiv

Source-linked AI summary

PaLI: A Jointly-Scaled Multilingual Language-Image Model

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, Radu Soricut

arXiv:2209.06794v4cs.CVcs.CL

TL;DR

Existing vision-language datasets and models provide strong capabilities but have limited multilingual coverage and uneven modality scaling. PaLI addresses this with a unified text-generation interface, reusable pretrained backbones, jointly scaled vision and language components, and multilingual multitask pretraining. It achieves state-of-the-art results across multiple vision-language benchmarks while retaining a simple, modular design.

  • Problem

    Large image-text datasets have fueled vision-language models but are English-only, while prior vision-language systems have limited multilingual coverage and mismatched modality scaling.

  • Method

    PaLI combines reusable mT5 and ViT backbones with joint vision-language scaling, a unified image-and-text-to-text interface, and eight multilingual pretraining objectives over WebLI.

  • Results

    PaLI-17B achieves state-of-the-art results on multiple benchmarks, including 149.1 CIDEr on COCO Captioning and 84.3% on VQAv2.

  • Takeaways & Limitations

    Jointly scaling both modalities, especially expanding vision capacity, provides a scaling roadmap for multilingual vision-and-language models.

  • Takeaways & Limitations

    Automatically harvested and automatically filtered pretraining data may leave undesirable images or text annotations in the model.

Abstract

from arXiv · show

Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pre-trained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To train PaLI, we create a large multilingual mix of pretraining tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. PaLI achieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text understanding), while retaining a simple, modular, and scalable design.

1 INTRODUCTION

PaLI jointly scales language and vision in a multilingual, multimodal model built around a unified text-generation interface. It combines reusable unimodal backbones, large multilingual data, and a scalable architecture to achieve strong performance across tasks.

  • PaLI performs image-only, language-only, and image+language tasks across many languages through one “image-and-text to text” interface.
  • PaLI reuses large unimodal checkpoints and introduces ViT-e, a 4B-parameter vision model, to transfer capabilities while increasing visual capacity.ViT-e reaches 90.9% ImageNet fine-tuning accuracy and 84.9% on ObjectNet.
  • Jointly scaling vision and language benefits multimodal performance, with vision providing better accuracy improvement per parameter/FLOP.PaLI-17B allocates about 25% of its parameters to the ViT-e vision component.
  • WebLI contains 10 billion images and tens of billions of image-text pairs in over 100 languages, expanding multilingual pretraining data and task diversity.
  • PaLI-17B achieves state-of-the-art results on multiple benchmarks, including 149.1 CIDEr on COCO Captioning and 84.3% on VQAv2.The reported VQAv2 result uses an open-vocabulary text-generative setting.
  • The model uses a simple, modularized, scalable sequence-to-sequence architecture that can be trained by reusing existing Transformer-based unimodal checkpoints.
  • Performance does not saturate for either modality at PaLI-17B, and multimodal performance benefits from scaling vision beyond the previous-largest ViT.
  • A mixture of pretraining objectives benefits large vision-and-language models.

2 RELATED WORK

Prior vision-language research combines pretrained unimodal models, contrastive or generative objectives, and increasingly large automatically mined datasets. However, major image-text datasets and much prior work remain predominantly English-only, motivating multilingual scaling.

  • Pretrained vision and language models provide effective foundations for vision-language tasks, while some systems depend on external object-detection components.
  • Vision-language models use contrastive learning, generative text modeling, or combinations of both to support downstream tasks.
  • Recent systems scale jointly modeled vision-language architectures to billions of parameters, with differing allocations between vision and language components.
  • Automatically mined image-text datasets grew from millions to billions of examples, supporting larger vision-language models.Examples include CC3M, CC12M, LEMON, GIT, and ALIGN.
  • These large image-text datasets are English-only, while attempts to create multilingual image-language data have been limited.

3 THE PALI MODEL

PaLI is a modular encoder-decoder system that converts visual and textual inputs into text, reusing pretrained language and vision backbones. Its training combines multilingual web data with diverse objectives and scalable model variants.

  • 3.1 ARCHITECTURE: The architecture supports unimodal and multimodal tasks by expressing inputs and outputs through a shared text-generation interface.
  • 3.1 ARCHITECTURE: PaLI uses an encoder-decoder Transformer whose text encoder receives unpooled visual tokens produced by a Vision Transformer.Previously trained unimodal checkpoints are reused.
  • 3.1 ARCHITECTURE: ViT-e scales a vanilla Vision Transformer to 4B parameters while retaining the ViT-G architecture and training recipe.
  • 3.1 ARCHITECTURE: PaLI reuses mT5-Large or mT5-XXL as its language backbone and trains on pure language tasks to help preserve language understanding and generation abilities.
  • 3.1 ARCHITECTURE: PaLI-3B, PaLI-15B, and PaLI-17B combine different mT5 and ViT configurations to study model scaling.
  • 3.2 DATA: WebLI expands image-language data collection from English-only datasets to 109 languages using publicly available web images and text.
  • 3.2 DATA: PaLI is pretrained with a mixture of eight tasks spanning text-only, captioning, OCR, and English and cross-lingual VQA objectives.
  • 3.2 DATA: Training uses template-based prompts and teacher forcing with softmax cross-entropy over a 1.6B-example mixture.

4 EXPERIMENTS

Experiments evaluate PaLI across English, multilingual, VQA, scaling, language-understanding, and ablation settings. PaLI achieves strong captioning and VQA results, improves with larger language and vision components, and retains language-understanding capabilities.

  • Image captioning: 149.1 CIDEr on COCO Captions establishes a new high for models without CIDEr optimization.
  • Multilingual evaluation: Crossmodal-3600 evaluates multilingual captioning across seven listed languages and reports an average over 35 languages.
  • Visual question answering: PaLI-17B achieves 84.3 accuracy on VQAv2, exceeding Flamingo by +2.2 points in open-vocabulary generation and BEiT-3 by +0.3 points in closed-vocabulary classification.
  • Scaling: Scaling language from 1B to 13B parameters and vision from 2B to 4B improves CIDEr and VQA accuracy across tasks.The experiments use CIDEr for captioning and accuracy for VQA; standard results use 224×224 pre-training, while orange bars denote high-resolution PaLI-17B checkpoints.
  • Language understanding: PaLI-17B maintains high-level English language understanding and is on par with mT5-XXL on XTREME benchmarks despite multimodal pre-training.
  • Ablation studies: The split-caption objective is most critical across ablation benchmarks, while object-related objectives improve performance across all benchmarks.

ETHICS STATEMENT AND BROADER IMPACTS

The paper highlights risks from large multimodal models and automatically harvested data, including bias, distorted information, undesirable content, and uneven multilingual output quality.

  • Automatically harvested and automatically filtered data may introduce undesirable images, text annotations, descriptions, or concepts into the model.
  • Language coverage varies across the more than 100 supported languages, so generated text quality may differ and include inaccurate or undesirable outputs.
  • The authors caution that large models may contain unknown biases or stereotypes and should be carefully reassessed before specific uses.

REPRODUCIBILITY STATEMENTS

The paper documents PaLI’s open-source components, model variants, ViT-e backbone, training frameworks, compute, and resolution-specific training procedures. It also provides examples and supporting data and model documentation.

  • Reproducibility resources: PaLI uses open-source ViT and mT5 components, with architecture, training, and data-card details provided in the paper and appendix.
  • Model specifications: The largest model is PaLI-17B, while ViT-e scales to 4B parameters through increased width, depth, and MLP dimensions.
  • Implementation: The models are implemented in JAX/Flax using T5X and Flaxformer, with documented learning-rate schedules and optimization settings.
  • Compute: PaLI-17B is pretrained for seven days on 1,024 GCP-TPUv4 chips with a batch size of 4,096.
  • Training procedure: The standard training pass uses 224×224 images with the vision component frozen, while PaLI-17B receives a second 588×588 high-resolution pretraining phase.
  • Examples: Figure 3 provides multilingual examples of captioning, visual question answering, OCR-oriented captioning, and question answering.

A.2 THE PRETRAINING TASK MIXTURE

PaLI uses a mixed pretraining curriculum spanning text-only, image-text, multilingual, VQA, object-aware, and detection objectives. These tasks are organized through a shared image-and-text-to-text interface and specified dataset mixing and fine-tuning settings.

  • Task mixture: The mixture includes span corruption, split-captioning, captioning, OCR, VQA, visual question generation, object-aware VQA, and generative object detection.These objectives use text-only, WebLI, translated captioning and VQA, and public detection datasets.
  • Task mixture: Generative object detection predicts bounding-box coordinates and object labels from prompts containing positive and negative class labels.Training uses 16M deduplicated images from Open Images, Visual Genome, and Object365 because WebLI lacks bounding-box annotations.
  • Training setup: The pretraining setup uses a documented task mixing ratio, while fine-tuning selects batch size and learning rate from limited candidate sets.The reported fine-tuning search does not cover every possible hyperparameter combination.
  • Training setup: Zero-shot image classification scores each class by prompting PaLI to generate an English alt-text target containing the class name.The interface scores all 1,000 ImageNet class labels directly rather than training a classifier head.

B WEBLI DATASET DETAILS

WebLI is a large multilingual image-language dataset built from public-web images and texts, with OCR annotations and similarity filtering. The construction also removes near-duplicates against evaluation datasets to reduce leakage risk.

  • Dataset scale: WebLI contains about 10 billion images and 12 billion alt-texts in 109 languages, plus 29 billion image-OCR pairs.OCR annotations are produced using a publicly available automatic service.
  • Dataset construction: WebLI images are near-deduplicated against the train, validation, and test splits of 68 common vision and vision-language datasets.The removal shrinks the dataset by only 0.36% while addressing potential pretraining-to-evaluation leakage.
  • Dataset construction: Image-alt-text pairs are scored using cross-modal cosine similarity between learned image and text embeddings.The image embeddings use graph-based semi-supervised learning, while text embeddings are learned contrastively from frozen image embeddings.
  • Dataset construction: Only the top 10% of original WebLI image-text pairs, about 1 billion examples, are retained for PaLI training.A tuned similarity threshold selects the retained pairs.
  • Dataset overview: Figure 4 presents sampled images, recognized-language statistics, and image-text pair counts relative to other large-scale vision-language datasets.The figure distinguishes multilingual alt-text from computed OCR annotations.

C.2 ADDITIONAL SCALING RESULTS

Additional scaling experiments show that larger PaLI models improve multilingual performance, while vision and language capacity jointly support gains across captioning and VQA. The 5B results provide additional, differently configured evidence for this scaling pattern.

  • Multilingual scaling: Captioning quality across French, Hindi, Hebrew, Romanian, Thai, and Chinese is fairly consistent after back-translation to English.CIDEr is computed against English references for the six-language subset.
  • Joint scaling: The 5B PaLI model combines mT5-Large with ViT-e and is evaluated on COCO-Cap and OKVQA as additional evidence for joint scaling.Its training mixture and hyperparameters differ slightly from those of other PaLI sizes.
  • Joint scaling: +2.1 CIDEr points separates PaLI-3B from 5B on COCO, compared with +2.8 for PaLI-15B versus 17B.The authors report that scaling ViT provides better return on investment, measured as improvement per 1B parameter.
  • Joint scaling: The scaling table also reports results for 700M, 3B, and 17B PaLI models on TallyQA.

C.3 ADDITIONAL ABLATIONS

Ablations examine initialization, multilingual WebLI data, vision scaling, OCR input, and zero-shot transfer. They show that pretrained initialization, multilingual data, larger vision backbones, and task-specific OCR each materially affect performance, with some tasks remaining strong without OCR.

  • Initialization and training: Initializing PaLI from existing unimodal checkpoints plays a critical role in quality, while freezing ViT during pretraining improves downstream COCO fine-tuning in the reported comparison.
  • Multilingual data: English-only WebLI leaves multilingual captioning capability very low and causes a clear drop in cross-lingual VQA performance on xGQA.
  • Multilingual data: Removing non-English WebLI data significantly harms XM-3600 performance even when other translated multilingual datasets remain in the mixture.On xGQA, the impact is reduced but remains apparent; including non-English WebLI improves performance by +0.4 overall.
  • Multilingual data: Training with all WebLI languages improves English COCO captioning by about +2 CIDEr points relative to English-only WebLI.The authors suggest richer image descriptions or cross-lingual transfer as possible explanations.
  • Vision scaling: ViT-e benefits vision-language tasks more than smaller ViT variants in the reported comparisons, including COCO and VQAv2.The corresponding table reports 10-shot and 25-shot ImageNet accuracy for vision-only tasks and captioning and VQA results for paired PaLI models.
  • Vision scaling: LiT transfer with ViT-e consistently improves over ViT-g, including 84.9% zero-shot accuracy in the reported evaluation.The standalone ViT-e evaluation includes supervised fine-tuning and frozen-representation transfer.
  • OCR and transfer: Without detected OCR input, PaLI slightly loses performance on TextCaps, TextVQA, and VizWiz-QA, but establishes state of the art on VizWiz-QA.The comparison also notes that PaLI remains close to the first version of GIT without OCR.
  • OCR and transfer: PaLI sets new state-of-the-art zero-shot performance on the natural subset of VTAB, alongside additional zero-shot ImageNet and Crossmodal-3600 evaluations.

D MODEL FAIRNESS, BIASES, AND OTHER POTENTIAL ISSUES

The paper treats web-data bias as an ongoing fairness risk and begins addressing it through transparency and demographic measurement. Its reported materials include sampled-data analyses and multilingual retrieval and task evaluations.

  • Web-derived training data may contain bias or unfairness, making transparency and measurement necessary first steps.
  • Crossmodal-3600 reports zero-shot image-to-text and text-to-image retrieval accuracy across all 36 languages.
  • A random sample of 112,782 WebLI examples was analyzed with the Know Your Data tool for demographic properties.The analysis covered both images and associated texts.
  • 54% of sampled image subjects appeared feminine presenting and 46% masculine presenting; sampled text used female, male, and they/them pronouns 30%, 38%, and 31% of the time.
  • The paper identifies demographic-bias analysis as a first step and calls for continued measurement and mitigation across new tasks.Future analysis is intended to examine recognition capabilities and biases related to attributes such as gender and age.

E LIMITATIONS

The paper reports limitations involving complex-scene description, multilingual retention after English-only fine-tuning, benchmark evaluation, and model availability. It also documents the model’s architecture, inputs, outputs, and implementation context.

  • Limitations: Complex scenes with many objects may not be described thoroughly because most source data lacks complex annotations.Object-aware and localization-aware queries were added as mitigation.
  • Limitations: Some multilingual capabilities are lost when the model is fine-tuned on English-only data.The paper states that multilingual mixtures would be preferable for fine-tuning.
  • Limitations: Open-vocabulary evaluation can mark correct synonyms or paraphrases as incorrect when they do not exactly match the target response.Fixed-vocabulary methods avoid this issue but generalize less beyond dataset answers.
  • Model status: PaLI is a static model trained on an offline dataset and is presented as a research prototype not currently available publicly.
  • Model characteristics: The largest PaLI model has 17B parameters, combining a 13B mT5-XXL model with a 4B ViT-e model.

G WEBLI DATASHEET

WebLI is a large web-collected image-language dataset documented through a datasheet. Its instances combine images, associated texts, metadata, and filtering signals, while preprocessing removes several excluded categories.

  • WebLI is documented through a datasheet following the dataset documentation framework of Gebru et al.
  • Each instance contains 20+ features, including image pixels, alt-text, page titles, OCR, metadata, and alt-text filter signals.
  • The dataset likely contains offensive, insulting, or threatening material because it was collected from the web.Algorithmic methods and classifiers were used to remove sensitive or personally identifiable information and pornographic images.
  • WebLI is not annotated, and filtering excludes adult-content images, empty texts, PII-identified texts, improper shapes, and images with too many paired texts.
Loading 2209.06794v4…