PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Few-Shot Learning with Frozen Language Models

Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill

arXiv:2106.13884v2cs.CVcs.CLcs.LG

TL;DR

Frozen addresses how to transfer language models’ few-shot learning to vision without changing their weights. It trains a vision encoder to produce continuous prefixes for a frozen language model, enabling interleaved image-text prompting. The resulting system performs open-ended multimodal interpretation and few-shot adaptation across visual question answering, outside-knowledge answering, and visual concept learning.

  • Problem

    Large language models can learn new language tasks from a few examples but are blind to visual tasks, questions, and concepts.

  • Method

    Frozen trains a vision encoder on image-caption pairs to represent images as continuous language-model prefixes while keeping the pretrained language model frozen.

  • Results

    Frozen transfers rapid task adaptation, encyclopedic knowledge, and fast concept binding to multimodal settings, including open-ended interpretation and genuinely multimodal few-shot learning.

  • Takeaways & Limitations

    Interleaved image-text prompting can extend language-model few-shot capabilities to visual inputs, including learning new visual-category names and using outside knowledge.

  • Takeaways & Limitations

    Frozen remains far from state-of-the-art on specific few-shot tasks and is presented as a starting point for multimodal few-shot learning.

Abstract

from arXiv · show

When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we train a vision encoder to represent each image as a sequence of continuous embeddings, such that a pre-trained, frozen language model prompted with this prefix generates the appropriate caption. The resulting system is a multimodal few-shot learner, with the surprising ability to learn a variety of new tasks when conditioned on examples, represented as a sequence of multiple interleaved image and text embeddings. We demonstrate that it can rapidly learn words for new objects and novel visual categories, do visual question-answering with only a handful of examples, and make use of outside knowledge, by measuring a single model on a variety of established and new benchmarks.

1 Introduction

Frozen extends the few-shot capabilities of large language models to multimodal inputs without changing the language model’s weights. It supports open-ended image interpretation, task adaptation, visual concept learning, and use of language-only knowledge.

  • Motivation and approach: Frozen gives a pretrained language model access to visual information by encoding images into its word-embedding space while keeping language-model weights unchanged.The vision encoder is trained so the language model can generate captions from image-derived embeddings.
  • Results: The system can use facts acquired during language-only pre-training when generating open-ended responses about images.Curated examples include image-conditioned outputs that draw on learned factual knowledge.
  • Multimodal prompting: After captioning training on single image-text pairs, Frozen can process ordered sequences of multiple images and words for multimodal prompting.These sequences can provide examples of new tasks or teach names for new visual categories.
  • Results: Frozen shows strong zero-shot and improved in-context performance on visual question answering, while also performing above chance on fast visual-category learning.Comparisons with blind baselines indicate adaptation to both task language and image-language relationships.
  • Scope and limitation: The authors state that Frozen is often far from state-of-the-art because the work prioritizes a proof of concept over maximizing performance on any specific task.Nevertheless, it performs above trivial baselines across a wide range of tasks with only a handful of benchmark examples.
  • Contributions: Frozen is presented as a modular, scalable, efficient approach that retains language-model capabilities while accepting arbitrary sequences of text and image inputs.The paper evaluates rapid task adaptation, encyclopedic knowledge, and fast concept binding across existing and new benchmarks.

2 Related Work

Frozen combines a frozen text-pretrained language model with an image-conditional continuous prefix, distinguishing it from task-specialized multimodal systems and approaches that update model weights.

  • Frozen language models: Unlike related approaches that fine-tune language-model weights for downstream applications, Frozen adapts to multimodal tasks without transformer weight updates.Its design preserves the pretrained model while using visual conditioning for adaptation.
  • Prefix tuning: Frozen extends prefix tuning by replacing a static continuous prompt with an image-conditional activation produced by an external neural network.This makes the visual prefix dynamic and dependent on the input image.
  • Task specialization: Many multimodal representation-learning systems use aligned data and task-specific fine-tuning to achieve strong classification performance, whereas Frozen targets broader task transfer.Those systems are described as highly specialized to individual tasks.
  • Task specialization: Other task-general multimodal generators update all system weights for each task and do not study zero- or few-shot learning, unlike Frozen.Frozen instead uses a text-only pretrained model and evaluates adaptation from limited examples.
  • Generative multimodal learning: Contrastive multimodal pretraining has enabled zero-shot generalization in discriminative settings, while Frozen studies zero- and few-shot generalization with open-ended generative output.The distinction is between discriminative prediction and unconstrained text generation.

3 The Frozen Method

Frozen trains only a vision encoder to turn images into language-model-compatible continuous prefixes, then uses interleaved image-text prompts for multimodal inference and few-shot task induction.

  • Base language model: Frozen starts from a pretrained autoregressive Transformer language model whose weights remain fixed during visual grounding.The base model is a 7-billion-parameter Transformer pretrained on C4.
  • Visual prefix: The vision encoder maps each image to a sequence of continuous embeddings with the same dimensionality as token embeddings.A linear projection reshapes the encoder output into n visual-prefix embeddings; two tokens performed best among the tested choices.
  • Training: Training updates only the vision-encoder parameters using paired image-caption data and maximizes caption likelihood.Gradients pass through the frozen language model to optimize the visual encoder with backpropagation and SGD.
  • Prompt structure: Relative positional encoding lets the Transformer handle prompts in which images occur in varying positions and multiple images are interleaved with text.This supports ordered multimodal sequences rather than requiring a single image at the beginning.
  • Inference interface: At inference, image embeddings can be placed beside text embeddings in arbitrary order to support visual question answering, outside-knowledge answering, and few-shot image classification.The same interface supplies task examples and evaluated inputs as a multimodal prompt.
  • Few-shot definitions: Task induction is explanatory text preceding image-text examples, while the number of shots counts complete task examples shown before evaluation.For visual concept binding, the number of ways counts classes and inner-shots count exemplars per category.
  • Evaluation setup: Table 1 compares transfer from Conceptual Captions to VQAv2 across models using or not using VQAv2 training data.The table includes Frozen train-blind and a Frozen VQA baseline that mixes VQAv2 training data.
  • Evaluation setup: Table 2 compares transfer from Conceptual Captions to OKVQA, with Frozen never trained on OKVQA and using VQAv2 only in the baseline row.The τ column identifies whether OKVQA training data is used.

4 Experiments: A Multi-Modal Few-Shot Learner

Frozen is evaluated as a multimodal few-shot learner across rapid task adaptation, encyclopedic knowledge transfer, and fast concept binding. It generalizes from captioning to visual question answering, retrieves language-model knowledge for visual questions, and learns novel visual-word associations from examples, with clear limits in harder settings.

  • Rapid Task Adaptation: Keeping the language model frozen generalizes substantially better to visual question answering than fine-tuning it.The experiments compare Frozen with fine-tuned and from-scratch systems.
  • Rapid Task Adaptation: Frozen transfers from captioning to visual question answering, supporting zero-shot generalization and improved performance from in-context examples.With four examples, performance reaches 38.2%, compared with 29.5% zero-shot and 48.4% with full SGD training.
  • Encyclopedic Knowledge: Frozen uses visual recognition to retrieve factual knowledge from language-model pretraining, exemplified by identifying an airplane and answering who invented it.The Conceptual Captions setup replaces named entities with general terms, so named-entity knowledge comes from language pretraining.
  • Fast Concept Binding: In two-way Open-Ended miniImageNet, Frozen learns novel names for presented visual categories and uses them above chance, improving with more examples.Performance improves more with different category exemplars than with repetitions of a single exemplar.
  • Fast Concept Binding: Richer and more varied visual support improves fast binding, while five-way Open-Ended miniImageNet remains not significantly above chance.Real-Name miniImageNet has higher absolute performance than the open-ended novel-name version, and the five-way setting remains beyond current capabilities.
  • Fast Concept Binding: Frozen improves with more shots on both Fast-VQA and Real-Fast-VQA, indicating that novel words can be integrated into multimodal language processing.The blind prefix-tuned model also improves moderately on Real-Fast-VQA from additional linguistic cues, but model outputs indicate multimodal support often contributes.

5 Discussion

Frozen demonstrates multimodal few-shot learning by extending soft prompting to ordered image-text inputs while preserving the language model’s text-prompting abilities. The authors frame it as an open-ended proof of concept whose task-specific performance remains below state-of-the-art systems trained on full datasets.

  • 5 Discussion: Frozen achieves far-from-state-of-the-art performance on specific few-shot tasks compared with systems using the full training set.The authors characterize the work as a starting point or baseline for multimodal few-shot learning research.
  • 5 Discussion: Further improvement is needed to make the observed zero-shot and few-shot generalization more robust, including higher accuracy and fewer seeds for compelling samples.The paper also leaves more elaborate vision-language mixing architectures for future investigation.
  • 5 Discussion: Frozen extends soft prompting to ordered sets of images and text while preserving the language model’s text-prompting abilities.The method transforms a large language model into a multimodal few-shot learning system.
  • 5 Discussion: The resulting system supports open-ended image interpretation and multimodal few-shot learning despite being trained only for captioning.The authors connect this result to rapid binding between language and visual elements across ordered inputs.

A.1 Compute Usage

The seven-billion-parameter language model was distributed across four accelerators and trained with additional data parallelism to reach a batch size of 128.

  • A.1 Compute Usage: The seven-billion-parameter language model used model parallelism across four accelerators.Each model-parallel instance had a batch size of 8.
  • A.1 Compute Usage: Sixteen synchronous data-parallel replicas increased the configuration’s batch size from 8 to 128.The system was trained on a 4x8 TPUv3 topology for about 12 hours.

A.2 Frozen Architecture Details

Frozen uses a GPT-like transformer with residual self-attention and positionwise MLP layers, differing from GPT-2 through relative position encodings. Its seven-billion-parameter configuration has 32 layers, 4096 hidden units per layer, and 32 attention heads.

  • A.2 Frozen Architecture Details: The pretrained transformer has GPT-like residual layers, each combining self-attention with a positionwise MLP.Its only stated architectural deviation from GPT-2 is the use of relative position encodings.
  • A.2 Frozen Architecture Details: The seven-billion-parameter configuration uses 32 layers with 4096 hidden units in each layer.The attention operations use 32 heads per layer.

A.3 Few-Shot Learning Definitions

Frozen can be conditioned on interleaved image-text sequences, so the same task may be induced through multiple input arrangements. The paper formalizes terminology for distinguishing these induction settings.

  • A.3 Few-Shot Learning Definitions: Frozen accepts sequences of interleaved images and text for multimodal tasks.This conditioning supports multiple ways of inducing the same task.
  • A.3 Few-Shot Learning Definitions: The paper formalizes terminology to distinguish different ways of inducing a task in Frozen.The terminology is described in section 3.4 of the main text.

A.4.1 Open-Ended MiniImageNet

Open-Ended MiniImageNet constructs few-shot visual-category questions by interleaving labeled support images with captions using randomly assigned nonsense words, then querying a held-out image.

  • Evaluation construction: The evaluation samples classes from a MiniImageNet class subset and uses validation images.
  • 2-way questions: Each 2-way question interleaves n labeled support images from two classes and assigns each class a random nonsense word.
  • 2-way questions: The query presents a held-out image with the truncated caption “this is a,” requiring the corresponding nonsense word.
  • 2-way questions: Class-word assignments are balanced because either class appears first or second in support, and either answer word is equally likely.
  • 5-way questions: The 5-way version extends the procedure to five classes and uses five nonsense words selected to decompose into two model tokens.

A.4.2 Real-Name miniImageNet

Real-Name miniImageNet and Fast-VQA evaluate whether Frozen can bind visual categories or question-relevant objects to names and answers in few-shot multimodal prompts. The encyclopedic-knowledge analysis further shows that visual input and demonstrations can elicit factual knowledge from language pre-training, though sampled outputs may require curation.

  • A.4.2 Real-Name miniImageNet: Real-Name miniImageNet replaces nonsense labels with ImageNet class names while retaining the few-shot support procedure.
  • A.4.3 Fast-VQA: Fast-VQA uses ImageNet images from all 1,000 classes and constructs questions from Visual Genome examples containing two relevant object names.
  • A.4.3 Fast-VQA: Fast-VQA replaces the two object names with randomly assigned nonsense words in support captions and modified questions, while Real-Fast-VQA keeps ImageNet class names without replacement.
  • A.5 Encyclopedic Knowledge: Few-shot demonstrations recover named-entity outputs for image questions after captioning training otherwise encourages vague descriptions without names.
  • A.5 Encyclopedic Knowledge: The model can use an image to identify an object and retrieve related factual knowledge from language pre-training, including who invented an airplane.
  • A.5 Encyclopedic Knowledge: The encyclopedic-knowledge behavior was observed across model versions and examples, but successful samples could require three to four tries because of language-model failure modes.
Loading 2106.13884v2…