PaperlayerView exact revision on arXiv

Source-linked AI summary

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren

arXiv:2110.08484v2cs.CVcs.CL

TL;DR

Large vision-language few-shot learners are difficult to deploy because of their size, motivating a smaller approach to prompt-based low-resource learning. The paper introduces FewVLM, a sequence-to-sequence model trained with PrefixLM and MaskedLM and analyzed across prompt settings. FewVLM outperforms the 31× larger Frozen by 18.2% point on zero-shot VQAv2 and achieves comparable results to the 246× larger PICa, while prompt and objective effects vary across settings and tasks.

  • Problem

    Recent few-shot vision-language models are impractically large for real-world deployment, motivating low-resource learning with a smaller model.

  • Method

    FewVLM is a moderate-sized sequence-to-sequence vision-language model pre-trained with PrefixLM and MaskedLM and evaluated with diverse prompts.

  • Results

    18.2% point on zero-shot VQAv2 over Frozen, which is 31× larger, while achieving comparable results to PICa, a 246× larger model.

  • Takeaways & Limitations

    Prompt effects are strongest in zero-shot settings, noisy prompts catch up with more training data, and MaskedLM favors VQA while PrefixLM favors captioning.

Abstract

from arXiv · show

Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning. However, these VL models are hard to deploy for real-world applications due to their impractically huge sizes and slow inference speed. To solve this limitation, we study prompt-based low-resource learning of VL tasks with our proposed method, FewVLM, relatively smaller than recent few-shot learners. For FewVLM, we pre-train a sequence-to-sequence transformer model with prefix language modeling (PrefixLM) and masked language modeling (MaskedLM). Furthermore, we analyze the effect of diverse prompts for few-shot tasks. Experimental results on VQA show that FewVLM with prompt-based learning outperforms Frozen which is 31x larger than FewVLM by 18.2% point and achieves comparable results to a 246x larger model, PICa. In our analysis, we observe that (1) prompts significantly affect zero-shot performance but marginally affect few-shot performance, (2) models with noisy prompts learn as quickly as hand-crafted prompts given larger training data, and (3) MaskedLM helps VQA tasks while PrefixLM boosts captioning performance. Our code is publicly available at \url{https://github.com/woojeongjin/FewVLM}

1 Introduction

FewVLM targets practical low-resource vision-language learning with a smaller model, while investigating how prompts and pre-training objectives affect zero- and few-shot performance.

  • Motivation: Few-shot vision-language models can reduce labeled-data requirements, but recent models are too large for deployment on small or moderate computing machines.Collecting many quality training examples is also expensive and slow.
  • Approach: FewVLM is a moderate-sized vision-language model fine-tuned with no or a handful of examples and pre-trained with PrefixLM and MaskedLM.The setup is intended to support economical training and inference on standard hardware.
  • Research questions: The study examines how prompt design affects zero- and few-shot learning, whether prompt effects persist with more training data, and how pre-training objectives affect performance.It compares hand-crafted and noisy prompts and studies PrefixLM versus MaskedLM.
  • Results: 18.2% point improvement on zero-shot VQAv2 over Frozen is reported, although Frozen is 31× larger than FewVLM.FewVLM also achieves comparable results to PICa, a 246× larger model.
  • Findings: Prompts significantly affect zero-shot performance but only marginally affect few-shot performance on new tasks.The paper also reports that MaskedLM helps VQA while PrefixLM boosts captioning.

2 Related Work

Prior work established few-shot vision-language learning and prompt-based language modeling, but strong VL few-shot learners remain impractical because of their model sizes.

  • Vision-language few-shot learning: Frozen, PICa, and SimVLM demonstrate few-shot or zero-shot vision-language learning across VQA, image classification, and captioning tasks.Frozen incorporates images and text through soft prompting, PICa converts images into textual descriptions, and SimVLM uses PrefixLM.
  • Vision-language few-shot learning: These vision-language learners improve performance on few-shot tasks but are impractical for real-world applications because of their model sizes.This deployment limitation motivates smaller prompt-based VL learners.
  • Language model prompting: Prompting and task demonstrations have improved pre-trained language models in NLP, including small-model few-shot text classification.The paper extends these prompt-based ideas to vision-language tasks.
  • Language model prompting: CLIP shows that prompt templates can affect zero-shot image-classification performance, motivating prompt analysis for vision-language tasks.The study examines prompt effects on both zero-shot and few-shot VL performance.

3 Analysis Setup

The analysis evaluates zero- and few-shot vision-language learning across multiple tasks, datasets, prompts, objectives, and baselines under limited-data settings.

  • Experimental setup: Zero-shot evaluation uses no training or development examples, while few-shot evaluation constructs equally sized training and development sets from limited training data.The few-shot setup follows prior procedures for tuning hyperparameters and selecting models.
  • Analysis questions: The study tests hand-crafted and noisy prompts to measure how prompt design affects zero- and few-shot performance and whether effects change with training-data size.It separately compares models trained with different data amounts and prompts.
  • Analysis questions: The analysis compares PrefixLM and MaskedLM as pre-training objectives for zero- and few-shot performance.These objectives are evaluated on the downstream tasks in the experiments.
  • Tasks and datasets: The downstream tasks are visual question answering, image captioning, and categorical learning, evaluated on VQAv2, OK-VQA, GQA, NoCaps, Flickr30k, and miniImageNet.Categorical learning uses a 5-way k-shot setup with five classes and k examples per class.
  • Evaluation: Few-shot results are averaged over five randomly sampled training and development splits, with models fine-tuned for 200 epochs and selected using development performance.NoCaps uses COCO captioning training data because it has no training data.
  • Baselines: Comparisons include Frozen and PICa for VQA, SimVLM for captioning, Unified VLP for selected tasks, and fully fine-tuned models as upper bounds.The fully fine-tuned baselines are trained on entire datasets, unlike few-shot models.

4 Method

FEWVLM is an encoder-decoder vision-language model that represents images with object regions, generates target text from text-image inputs, and uses two pre-training objectives.

  • Architecture: FEWVLM uses an encoder-decoder architecture to encode visual and text inputs and generate target text.The model is not task-specific, supporting zero- and few-shot settings.
  • Architecture: Each input image is represented by 36 object regions from a Faster R-CNN trained on Visual Genome, appended to the text input for encoding.The region representations and text are jointly supplied to the encoder.
  • Training objective: The model trains by minimizing the negative log-likelihood of target-text tokens conditioned on input text and image representations.This objective defines generation over the target sequence.
  • Pre-training: FEWVLM is pre-trained with PrefixLM and MaskedLM using image-text data from MS COCO and Visual Genome.The pre-training corpus contains 9.18M image-text pairs and 180K distinct images.
  • Pre-training: PrefixLM uses an initial text span and image as encoder input and generates the remaining span, whereas MaskedLM replaces random spans with sentinel tokens.These objectives provide complementary pre-training formats for the model.

5 Low-resource Adaptation

FEWVLM adapts vision-language models to low-resource tasks through prompt-based generation, examining how prompt formats affect VQA, captioning, and categorical learning. The section compares hand-crafted and noisy prompts and reports zero-shot benchmark context.

  • Prompt-based adaptation: FEWVLM uses prompt templates to construct input and target text during training, then generates the label from the same prompt at inference.
  • Visual Question Answering: VQA is formulated as generation, allowing FEWVLM to produce answers without task-specific classification heads.
  • Prompt design: Hand-crafted prompts vary input templates, sentinel-token placement, and target formats to study how prompt design affects adaptation.
  • Prompt design: Noisy-prompt experiments test irrelevant instructions, random vocabulary tokens, and misleading captions as robustness conditions.
  • Captioning: Captioning experiments compare the input phrases “a picture of,” “a photo of,” and “an image of,” while training targets remain unprompted captions.
  • Evaluation: Zero-shot VQA results compare FEWVLM models with larger Frozen and PICa-Full systems, while captioning uses CIDEr and SPICE evaluation.

6 Results and Discussion

FEWVLM delivers strong zero-shot and few-shot vision-language results with prompt-based learning, while prompt effects and pre-training objectives vary by task and data regime.

  • 6.2 Performance on Zero-shot Learning: FEWVLMbase significantly outperforms Frozen on zero-shot VQA, while FEWVLMlarge achieves comparable performance to the much larger PICa model.PICa is described as 246× larger than FEWVLMlarge, and Frozen as about 31× larger than FEWVLMbase.
  • 6.2 Performance on Zero-shot Learning: FEWVLMbase improves VQAv2 performance by about 30% point and significantly improves over VL-T5no-vqa on NoCaps captioning.
  • 6.3 Performance on Few-shot Learning: On few-shot VQA, FEWVLMlarge achieves a comparable result to PICa on VQAv2, while FEWVLMbase trained with four examples outperforms Frozen.PICa uses 16 in-context examples, whereas FEWVLM uses 16 training and validation examples in the reported setup.
  • 6.3 Performance on Few-shot Learning: On few-shot captioning, FEWVLMbase outperforms VL-T5no-vqa by 31.1% point on NoCaps CIDEr and slightly exceeds Unified VLP on Flickr30k.
  • 6.4 MiniImageNet: FEWVLM significantly outperforms Frozen across all miniImageNet shot settings, with hand-crafted prompts helping much more at 1 shot than at 5 shots.
  • 6.5 Prompt Effects: Input prompts strongly affect zero-shot results, whereas noisy or absent prompts approach hand-crafted-prompt performance with larger few-shot training data.On Flickr30k, “an image of” outperforms no prompt by 21.4 point; random-sentence prompts degrade performance most among noisy prompts.
  • 6.5 Prompt Effects: Adding “<text_1” to the VQA target prompt reduces performance variance, although both target prompts show similar results.
  • 6.6 Pre-training Objectives: MaskedLM helps VQA, while PrefixLM helps captioning in both zero-shot and few-shot settings; combining objectives improves cross-task generalization.

7 Conclusion

The paper presents FEWVLM as a prompt-based few-shot learner for vision-language tasks and reports strong results across diverse datasets. It finds that prompt sensitivity decreases with more training data and that pre-training objectives help different downstream tasks.

  • FEWVLM is a few-shot prompt-based learner for vision-language tasks that outperforms baselines and approaches PICa despite its much smaller size.
  • Prompts are vital in zero-shot and few-shot tasks, but models with larger training data are not significantly affected by noisy prompts.
  • MaskedLM and PrefixLM help different few-shot tasks, supporting task-dependent effects of pre-training objectives.

A Model Architectures

FEWVLM uses architectures based on VL-T5 and T5 for its base and large variants.

  • FEWVLMbase is based on VL-T5, while FEWVLMlarge is based on T5.

B COCO Captioning

The COCO captioning evaluation uses the Karpathy split with separate training, validation, and test image sets.

  • The MS COCO captioning data use the Karpathy split with 113,287 training, 5,000 validation, and 5,000 test images.

C Prompt Study

The prompt study evaluates hand-crafted prompts and a target prompt on VQAv2, with hand-crafted prompts also tested on Flickr30k. Results are organized across multiple prompt configurations and training settings.

  • Tables 7–9 report prompt results on VQAv2 and Flickr30k under various training settings.
  • VQAv2 results compare hand-crafted prompts with the target prompt “[A]”.
  • Flickr30k results evaluate hand-crafted prompts.

D Effect of Pre-training Data

The study examines how pre-training data affects few-shot downstream performance and compares MS COCO, Visual Genome, and Conceptual Captions. MS COCO and Visual Genome are reported as more helpful than Conceptual Captions.

  • The study compares MS COCO, Visual Genome, and Conceptual Captions as pre-training datasets for few-shot downstream tasks.
  • MS COCO and Visual Genome are more helpful to downstream tasks than Conceptual Captions.
  • The prompt-template experiments use input question text, target text “<text_1> [A]”, and appended image features.
  • Table 12 reports few-shot results for different pre-training datasets across downstream tasks.
Loading 2110.08484v2…