PaperlayerView exact revision on arXiv

Source-linked AI summary

PaLI-X: On Scaling up a Multilingual Vision and Language Model

Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, Radu Soricut

arXiv:2305.18565v1cs.CVcs.CLcs.LG

TL;DR

PaLI-X addresses whether scaling vision-language models and broadening their training mixture improves varied multilingual multimodal capabilities. It uses scaled visual and language components with mixed objectives, and reports broad benchmark gains plus emergent counting and multilingual detection. The paper also notes that fairness evaluation cannot capture the full societal impact of deployed models.

  • Problem

    The paper investigates how scaling both components of a multilingual vision-language model affects performance, capabilities, and emergent properties.

  • Method

    PaLI-X combines a large pretrained visual encoder and language encoder-decoder with at-scale vision-language training using self-supervision and full supervision.

  • Results

    PaLI-X substantially outperforms previous state-of-the-art models across diverse tasks and benchmarks, with emergent complex counting and multilingual object detection.

  • Takeaways & Limitations

    Scaling and the improved training recipe advance document, chart, infographic, captioning, question-answering, counting, video, few-shot, and object-detection capabilities.

  • Takeaways & Limitations

    Fairness analysis is limited because fairness is a societal concept that cannot be reduced to statistical metrics, which do not substitute for studying deployed models’ broad and delayed impact.

Abstract

from arXiv · show

We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.

1 Introduction

PaLI-X scales both vision and language components and broadens training objectives, improving performance across diverse benchmarks while exhibiting emergent counting and multilingual capabilities.

  • Scaling both vision and language components increases performance across a wide range of vision-language tasks.The gains extend to existing PaLI benchmarks and newer chart, document, infographic, and video understanding tasks.
  • A mixture of prefix-completion and masked-token completion improves the Pareto frontier between fine-tuning and few-shot performance.
  • PaLI-X exhibits emergent complex and plain counting improvements and multilingual transfer, including object detection with non-English labels.The counting capabilities are reported as not attributable to training design, and the model can switch between image-text and caption languages.
  • A high-capacity ViT-22B vision encoder co-trained for image classification and OCR label classification improves tasks requiring text-within-image understanding.
  • PaLI-X improves state-of-the-art results through fine-tuning on 15+ benchmarks without significant performance degradation under simultaneous multitask adaptation.

2 Related Work

Prior vision-language work explored scaling individual or joint components, multimodal in-context learning, and multitask fine-tuning; PaLI-X builds on these directions.

  • Complex counting requires fine-grained alignment between language-based specifications and visually based occurrences, unlike plain counting.
  • Flamingo scaled a frozen language component to 70B parameters for few-shot multimodal capabilities while keeping its vision encoder fixed at 435M parameters.
  • GIT scaled the vision component to 4.8B parameters with a 300M-parameter language decoder, while PaLI jointly scaled vision and language components.
  • Multimodal in-context learning extends earlier language-focused in-context learning by using pairs of images and captions as examples.
  • Unified-IO and Spotlight showed that multitask fine-tuning can produce solid cross-benchmark or near-task-specific performance, motivating PaLI-X’s broader evaluation.

3 Model

PaLI-X uses a large ViT encoder and encoder-decoder backbone to process multimodal inputs, with few-shot exemplars and a broad multilingual pretraining mixture.

  • Architecture: PaLI-X processes images with a ViT encoder and feeds projected visual embeddings with text embeddings into an encoder-decoder backbone.
  • Visual component: The visual backbone has 22B parameters and receives OCR-based pretraining alongside JFT-based image classification.OCR targets indicate whether detected tokens occurred in the image.
  • Overall model: The encoder-decoder backbone is initialized from a 32B-parameter UL2 variant with 50 layers in both encoder and decoder.
  • Few-shot formulation: Few-shot inputs provide labeled image-text pairs as exemplars, whose visual patch embeddings are concatenated with text embeddings for target prediction.Additional optimizations distribute exemplars between encoder and decoder and reweight attention.
  • Pretraining data: WebLI supplies roughly one billion multilingual image-alt-text pairs with OCR annotations across over 100 languages, supplemented by episodic related-image sets.
  • Training objectives: The pretraining mixture combines text span corruption, captioning, OCR objectives, and visual question answering over image-text data.

4 Experiments

PaLI-X is evaluated across image captioning, VQA, text understanding, multitask and few-shot settings, video tasks, classification, and detection. Across these evaluations, it generally matches or improves state-of-the-art performance while exposing trade-offs in multitask and few-shot configurations.

  • Per-task fine-tuning results: PaLI-X outperforms state of the art across diverse vision-language tasks, with particularly strong gains on challenging scene-text and document-understanding benchmarks.The evaluation covers 6 image-captioning and 13 VQA benchmarks, alongside additional tasks.
  • Per-task fine-tuning results: +11.1 and +18.8 points are reported over MoVie on TallyQA simple and complex counting questions, respectively, while VQAv2 reaches 86.1 accuracy and OKVQA reaches 66.1 accuracy.These results use end-to-end modeling without external OCR pipeline input.
  • Per-task fine-tuning results: +42.9 points are reported on AI2D, and PaLI-X reaches 46.3 in the question-only configuration, more than 4 points above Pix2Struct.External OCR generally improves performance, although intrinsic OCR is close on several well-oriented, large-font benchmarks at 756×756 resolution.
  • Multitask Fine-tuning: Multitask fine-tuning produces a single checkpoint across tasks, with average captioning performance decreasing by 1.4 points overall and 0.2 points excluding Screen2Words.VizWiz-Cap and Widget-Cap improve in the multitask setting, while COCO decreases slightly.
  • Few-shot Evaluation: +4.4 CIDEr points over previous state of the art are achieved on COCO with 4-shot captioning, while PaLI-X reports multilingual few-shot captioning results across 35 XM3600 languages.The model also performs decently on VQAv2, but trails Flamingo, consistent with a reported tension between few-shot and fine-tuning abilities.
  • Video and detection results: PaLI-X achieves new state-of-the-art performance on 5 of 7 video tasks, with 16 frames sometimes improving substantially over 8 frames, including nearly 6 CIDEr points on ActivityNet-Captions.On LVIS, the detection-tuned model reaches AP 31 overall and 31.4 on rare classes, versus about 12 for both in zero-shot evaluation.

5 Model Fairness, Biases, and Other Potential Issues

PaLI-X is evaluated for harmful associations, demographic parity, and subgroup performance, while the authors emphasize that statistical metrics cannot capture fairness’s full societal scope. The evaluation finds low toxicity, some occupation-linked gender disparities, and high accuracy across subgroups.

  • The RAI evaluation covers toxicity and profanity, demographic parity, and performance disparity across demographic subgroups.
  • Harmful associations: PaLI-X shows low toxicity and profanity across all FairFace subgroup slices.
  • Demographic parity: PaLI-X assigns higher occupation likelihood to men across most occupations, with significant associations for secretary, actor, guard, and plumber.
  • Demographic parity: Absolute WebLI correlations between perceived gender presentation and occupations are all below 0.2, with 99% below 0.1.
  • Performance disparity: PaLI-X maintains high accuracy across MIAP subgroups when detecting whether an image contains a person.
  • Limitations: The authors caution that fairness cannot be reduced to statistical metrics and that automated attribute inference may misidentify people.

6 Conclusions

Scaling and an improved training recipe produce a model that substantially outperforms prior state-of-the-art systems across diverse vision-and-language tasks, while also exhibiting emergent behaviors.

  • PaLI-X substantially outperforms previous state-of-the-art models across document, chart, infographic, captioning, VQA, counting, few-shot, video, and object-detection tasks.

A.1 PaLI-X Architecture Illustration

The appendix describes PaLI-X’s vision-language processing and evaluates its broad capabilities, including OCR-enhanced understanding, multilingual captioning, counting, and few-shot inference. Scaling produces especially notable gains for complex counting.

  • Architecture: Each video frame is independently processed by ViT, then patch embeddings are flattened and concatenated into one visual representation.
  • OCR capabilities: OCR co-training improves scene-text understanding on ST-VQA and TextVQA while also improving regular VQA performance.
  • Illustrative capabilities: PaLI-X’s examples demonstrate simple and complex counting, contextual text reading, and spatial awareness without an external OCR system.
  • Multilingual captioning: PaLI-X improves multilingual captioning over prior state-of-the-art performance in 5 of 7 reported languages and 14 of 35 total languages.
  • Counting: Complex counting shows emergence with scaling, unlike simple counting’s more linear progression across model sizes.
  • Few-shot architecture: In few-shot inference, exemplar texts and images are encoded separately, concatenated, and used by the decoder to autoregressively predict the target output.
  • Few-shot architecture: Attention re-weighting boosts cross-attention to target-image and target-prompt tokens, helping when many shots are provided.

B.5.2 Additional Few-shot Results

Additional few-shot results show that PaLI-X benefits from more multilingual exemplars and can transfer capabilities across languages while using examples to calibrate contextual concepts.

  • Multilingual captioning: 45.1 average CIDEr with 4 shots and 47.1 with 32 shots demonstrate improved few-shot multilingual captioning over the 22.7 no-shot baseline.
  • Multilingual captioning: The 32-shot average CIDEr score is only 6 points below the fully fine-tuned model.
  • Qualitative results: Few-shot captioning transfers across languages, including captions generated in images’ original languages.
  • Qualitative results: Three input shots calibrate the concept of “new,” enabling a plausible explanation for classifying an old personal computer as new.

B.5.3 Few-shot ablation results

The ablations examine how attention, token pooling, shot placement, and training choices affect few-shot performance, while also assessing multitask and OCR-related design decisions.

  • Few-shot ablations: 61.7 CIDEr is achieved by the full model for 4-shot COCO captioning, compared with 56.3 after pooling per-image tokens.Keeping all ViT encoder tokens improves few-shot captioning despite greater computational overhead.
  • Few-shot ablations: 59.6 CIDEr is achieved with per-example image-text attention, 2.1 points below the full attention model’s 61.7 for 4-shot COCO captioning.Image queries attend only to corresponding text, while text queries attend to all tokens.
  • Few-shot ablations: 82.1, 84.3 and 84.5 CIDEr are obtained with attention re-weighting at 4, 8 and 16 shots, respectively.Without re-weighting, scores decrease at larger shot counts in the reported PaLI-2 3B experiments.
  • Few-shot ablations: 65.2 CIDEr is achieved by the best encoder-decoder shot mix, compared with 64.0 using only encoder shots for 4-shot COCO captioning.The remaining shots are assigned to the decoder, suggesting that splitting shots across encoder and decoder is beneficial.
  • Multitask finetuning: Multitask finetuning uses a shared checkpoint and avoids test-set leakage through deduplication, but tasks converge at different rates without per-task validation optimization.The setup is intended to produce one model performing multiple tasks rather than separately fine-tuned models.
  • Training ablations: Adding image token generation does not harm performance on the main image-and-language understanding tasks.This ablation uses a 5B model with UL2-3B and ViT-G components, approximately a 10:1 down-scale of PaLI-X 55B.

C.1 Datasets & Benchmarks

The evaluation covers diverse video captioning and video question-answering datasets, with dataset sizes and effective experimental splits affected by data availability.

  • Video captioning: MSR-VTT contains 10K open-domain video clips with 20 captions each, using standard splits and test-set evaluation.Each clip lasts between 10 and 30 seconds.
  • Video captioning: VATEX includes captions for 41K videos sampled from Kinetics-600, with 10 English captions per video.Results are reported on the English public test set.
  • Video captioning: ActivityNet Captions contains 100K temporally localized sentences for 20K videos, evaluated with ground-truth temporal proposals.The standard 50/25/25% train-validation-test split is used, with val_1 for validation and val_2 for testing.
  • Video captioning: SMIT contains long audio-derived captions for 500K short video clips and is used as a large manually annotated video-captioning benchmark.It was traditionally used for text-to-video retrieval.
  • Video question answering: ActivityNet-QA contains 58,000 video question-answer pairs, while NExT-QA contains 52,044 pairs across 5,440 videos.ActivityNet-QA uses exact-match accuracy, whereas NExT-QA uses WUPS for its open-ended QA task.
  • Data availability: The experiments freshly collected data, and missing videos meant that training used sometimes significantly fewer videos than the original dataset descriptions.Most datasets were missing 10% or more of their videos because some were no longer available online.

D Additional results: Image Classification

Image classification is evaluated through zero-shot and ImageNet finetuning settings, including distribution-shifted datasets, with PaLI-X compared against generative-model baselines.

  • Evaluation setup: Zero-shot classification generates the English class name directly from the captioning prompt and counts exact class-name matches as correct.ImageNet-REAL additionally accepts outputs whose corresponding class belongs to the correct-label set.
  • Zero-shot evaluation: Zero-shot PaLI-X results are similar to PaLI results on the reported ImageNet classification evaluation.The evaluation uses the first-stage PaLI-X checkpoint at 224 resolution without ImageNet training.
  • Finetuning: Finetuning evaluates ImageNet-REAL, ImageNet-R, ImageNet-A, ImageNet-Sketch and ImageNet-v2 in addition to ImageNet.The experiments use the first-stage 224-resolution and final-stage 756-resolution models with shared hyperparameters.
  • Finetuning: PaLI-X achieves close to state-of-the-art results for generative models on ImageNet and other evaluated datasets after finetuning.The comparison is against GiT2, described as the current state of the art for full-finetuning on ImageNet.

E.1 Object detection as a VLM task

Object detection is formulated as conditional text generation: class prompts specify which objects to detect, and the model outputs their bounding-box coordinates.

  • Prompt construction: Training prompts combine positives, known negatives, and global negatives sampled from a wider label space.The example in Figure 9 illustrates positives such as car and wheel alongside negative and global-negative classes.
  • Prompt construction: Known negatives are classes definitely absent from the image, while global negatives are classes not explicitly labeled as positive or negative.Known negatives depend on dataset annotation completeness, whereas global negatives come from a wider combined label space.
  • Prompt sampling: Known negatives are sampled up to f− × n+, where n+ is the number of sampled positives and f− is drawn from U(0, 5.0).Positive sampling first retains a proportion of definitely present object boxes.
  • Prompt sampling: Global negatives are sampled up to fGN × n+ from valid classes in the wider label space, with fGN drawn from U(0, 5.0).The default global label space combines Visual Genome, Objects365 and OpenImagesV4, with additional spaces for detection finetuning.
  • Prompt and image preprocessing: Prompts are truncated to nmax and then shuffled, with nmax and the positive limit tuned per dataset to meet sequence-length constraints.Detection preprocessing also includes random flips, resizing, square crops and padding.
Loading 2305.18565v1…