PaperlayerView exact revision on arXiv

Source-linked AI summary

Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, Karen Simonyan

arXiv:2204.14198v2cs.CVcs.AIcs.LG

TL;DR

Few-shot adaptation of multimodal models remains difficult because conventional fine-tuning often needs many annotated examples and task-specific tuning. Flamingo bridges pretrained vision and language models for interleaved visual-text inputs, achieving state-of-the-art few-shot performance across diverse tasks and surpassing fine-tuned systems on 6 of 16 tasks with far less task-specific data.

  • Problem

    Multimodal models often require many annotated examples, task-specific hyperparameter tuning, and substantial resources to adapt to new tasks.

    Source: §1 · p. 3

  • Method

    Flamingo bridges pretrained vision and language models with cross-attention layers to process arbitrarily interleaved images, videos, and text for open-ended generation.

    Source: §2 · p. 4, §2.2 · p. 5

  • Results

    Flamingo achieves state-of-the-art few-shot performance across diverse vision-language tasks and surpasses fine-tuned state-of-the-art systems on 6 of 16 tasks using orders of magnitude less task-specific data.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Flamingo demonstrates that a single multimodal model can adapt to varied image and video tasks from only a few input/output examples.

    Source: §1 · pp. 3–4, §5 · p. 10

    Flamingo inherits pretrained language-model weaknesses, including occasional hallucinations and ungrounded guesses, poor generalization to longer sequences, and limited training sample efficiency.

    Source: §5 · p. 10

Abstract

from arXiv · show

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.

1 Introduction

Flamingo is a visual language model designed for few-shot adaptation across open-ended vision-language tasks using prompts containing interleaved visual and textual examples. It combines frozen vision and language models with architectural and training innovations, achieving strong few-shot performance without task-specific tuning.

Motivation

  • Flamingo addresses the limitations of fine-tuning and contrastive vision-language models by generating language for open-ended captioning and visual question-answering from few-shot prompts.Conventional fine-tuning often needs thousands of annotations, per-task hyperparameter tuning, and substantial resources, while contrastive models mainly support finite-outcome tasks.Source: §1 · p. 3

Model

  • Flamingo is a visually conditioned autoregressive model that ingests arbitrarily interleaved text, images, and videos and generates free-form text.It connects complementary frozen vision and language models while using novel components to preserve their pretrained knowledge and a Perceiver-based architecture for variable-resolution visual inputs.Source: §1 · p. 3, Figure 3 · p. 4

Training

  • Flamingo is trained on a carefully chosen mixture of complementary large-scale multimodal web data without machine-learning annotations, enabling direct few-shot adaptation.The paper emphasizes that this training strategy is crucial for final performance.Source: §1 · pp. 3–4

Contributions

  • Flamingo sets a new state of the art on 16 multimodal tasks and surpasses the fine-tuned state of the art on 6 using only 32 task-specific examples.With a larger annotation budget, it also sets a new state of the art on five additional benchmarks.Source: §1 · pp. 3–4

Evaluation

  • The evaluation reserves held-out benchmarks that were excluded from design and hyperparameter decisions to estimate unbiased few-shot performance across diverse tasks.The model is evaluated on multimodal language and image/video understanding tasks including captioning, visual dialogue, and visual question-answering.Source: §1 · p. 4

2 Approach

Flamingo bridges frozen vision and language models through fixed visual tokens and gated cross-attention, supporting interleaved image/video inputs and few-shot task adaptation. It is trained on diverse web-scale multimodal data and evaluated through multimodal in-context prompts.

2 Approach

  • Flamingo combines a frozen vision encoder and language model by converting visual features into fixed visual tokens and inserting trainable cross-attention layers between language-model blocks.The Perceiver Resampler outputs 64 visual tokens, while the new layers condition next-token prediction on visual information.Source: §2 · p. 4, §2.1 · p. 5, §2.2 · p. 5

2.2 Conditioning frozen language models on visual representations

  • GATED XATTN-DENSE layers insert gated visual cross-attention and feed-forward blocks into the frozen language model while preserving its initial behavior.A tanh gate with a zero-initialized, layer-specific scalar makes the conditioned model match the pretrained language model at initialization, improving stability and final performance.Source: Figure 4 · p. 5, §2.2 · p. 5

2.1 Visual processing and the Perceiver Resampler

  • The vision pipeline uses a pretrained frozen NFNet, samples video frames at 1 FPS, adds temporal embeddings, and resamples variable-size feature maps into 64 outputs.The Perceiver Resampler uses learned latent queries that cross-attend to flattened image or video features, reducing vision-text attention complexity.Source: §2.1 · p. 5

2.3 Multi-visual input support: per-image/video attention masking

  • Per-image/video attention masking lets each text token attend directly to only the immediately preceding visual input while retaining dependencies on earlier inputs through language-model self-attention.Although training uses at most five images per sequence, this design supports evaluation sequences containing up to 32 image/video-text pairs.Source: §2.3 · p. 6

2.4 Training on a mixture of vision and language datasets

  • Flamingo is trained with a weighted multi-objective loss over interleaved image-text, image-text-pair, and video-text-pair datasets scraped from the web.The mixture includes M3W from approximately 43 million webpages, ALIGN with 1.8 billion image-text pairs, LTIP with 312 million pairs, and VTP with 27 million videos.Source: §2.4 · p. 6

2.5 Task adaptation with few-shot in-context learning

  • For task adaptation, Flamingo receives interleaved support image/video-text pairs followed by a query visual input and generates or scores the answer without updating model weights.Open-ended tasks use beam-search decoding, close-ended tasks use log-likelihood scoring, and zero-shot generalization uses text-only task examples.Source: §2.5 · pp. 6–7
  • A single Flamingo reaches state-of-the-art on many image and video tasks with few-shot learning, outperforming fine-tuned methods on seven tasks using 32 examples without weight updates.Table 1 reports that Flamingo can surpass methods fine-tuned on thousands of annotated examples, with as few as four examples for state-of-the-art comparisons.Source: Table 1 · p. 7

3 Experiments

Flamingo achieves strong few-shot adaptation across 16 multimodal benchmarks, generalizes beyond development tasks, and improves further through fine-tuning. Experiments also show that performance depends on diverse training data, gated visual conditioning, efficient architecture choices, a strong vision encoder, and a frozen language model.

3.1 Few-shot learning on vision-language tasks

  • With as few as four examples, Flamingo outperforms prior zero-shot or few-shot methods across 16 benchmarks and beats fine-tuned state-of-the-art systems on six tasks using 32 examples.The results generalize well beyond the five development benchmarks used for design decisions.Source: §3.1 · pp. 7–8

3.2 Fine-tuning Flamingo as a pretrained vision-language model

  • Fine-tuning Flamingo improves over in-context few-shot results and establishes new state of the art on VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes.The procedure uses a short schedule, small learning rate, and an unfrozen vision backbone for higher input resolution.Source: Table 2 · p. 8, §3.2 · p. 8

3.1 Few-shot learning on vision-language tasks

  • Few-shot performance improves with model size and shot count, while Flamingo benefits from up to 32 inference images or videos despite training sequences limited to five images.The largest model makes better use of larger numbers of shots, demonstrating flexible processing of variable-length visual inputs.Source: §3.1 · p. 8

3.3 Ablation studies

  • Removing M3W decreases performance by more than 17%, removing paired image-text data decreases it by 9.8%, and removing paired video-text data harms all video tasks.Replacing image-text pairs with LAION-400M [96] slightly degrades performance, while gradient accumulation outperforms round-robin updates.Source: §3.3 · pp. 8–9
  • Removing 0-initialized tanh gating reduces the overall score by 4.2% and causes training instability, while GATED XATTN-DENSE performs best among conditioning architectures.The comparison includes vanilla cross-attention and GRAFTING [68], which learns interleaved self-attention and cross-attention layers from scratch.Source: §3.3 · p. 9
  • Adding GATED XATTN-DENSE every fourth block accelerates training by 66% for a 1.9% overall-score decrease, while the Perceiver Resampler outperforms slower MLP and vanilla Transformer alternatives.Under hardware constraints, Flamingo-9B uses every fourth layer and Flamingo-80B every seventh layer.Source: §3.3 · p. 9
  • The NFNet-F6 vision encoder exceeds CLIP ViT-L/14 [85] by 5.8% and NFNet-F0 by 8.0%, while fine-tuning the language model causes performance drops of 8.0% or 12.9% from scratch.These results support a strong vision backbone and freezing the language model to prevent catastrophic forgetting.Source: §3.3 · p. 9

4 Related work

Flamingo builds on pretrained language and vision-language modeling by using in-context few-shot adaptation with a frozen 70B Chinchilla LM [42]. It differs from prior approaches through arbitrary interleaving of images, videos, and text, and by training on multimodal webpages.

Language modelling and few-shot adaptation

  • Flamingo uses the 70B Chinchilla language model [42] and follows in-context few-shot learning rather than adapter, partial fine-tuning, prompt-optimization, metric-learning, or meta-learning approaches.The model builds on the standard pretraining-then-downstream-adaptation paradigm while selecting prompting-based adaptation inspired by.Source: §4 · p. 9

When language meets vision

  • Unlike prior vision-language models, Flamingo requires no fine-tuning on new tasks, generates text autoregressively, and uses a vision encoder based on contrastive models.It freezes Chinchilla LM layers and adds learnable layers within the frozen language model [42], following work aimed at preventing catastrophic forgetting.Source: §4 · pp. 9–10
  • Flamingo introduces the first language model able to ingest arbitrarily interleaved images, videos, and text.This distinguishes it from prior vision-language systems and supports multimodal sequences beyond fixed image-text pairings.Source: §4 · p. 10

Web-scale vision and language training datasets

  • Flamingo supplements scraped paired vision-text data with entire multimodal webpages containing interleaved images and text as a single training sequence.Unlike CM3 [1], which generates HTML markup, Flamingo predicts only plain text and emphasizes few-shot vision tasks rather than language-only zero-shot or fine-tuned evaluation.Source: §4 · p. 10

5 Discussion

Flamingo is presented as a general-purpose visual model for image and video tasks with minimal task-specific data, but its limitations and societal risks motivate further research and mitigation.

Limitations

  • Flamingo inherits pretrained language-model weaknesses, including occasional hallucinations, poor generalization to longer sequences, and low training sample efficiency.Addressing these issues could accelerate progress and improve visual-language models.Source: §5 · p. 10
  • Flamingo’s classification performance lags behind state-of-the-art contrastive models [82] [85], motivating unified methods that combine broad task coverage with stronger text-image retrieval.Contrastive models directly optimize text-image retrieval, while Flamingo supports a wider range of tasks, including open-ended ones.Source: §5 · p. 10
  • In-context learning works with only a few dozen examples and enables simple inference-only deployment, but is sensitive to demonstrations and scales poorly in compute and performance beyond low-shot regimes.Combining few-shot learning methods may leverage their complementary benefits.Source: §5 · p. 10

Societal impacts

  • Flamingo can lower barriers to beneficial and malicious applications while risking offensive outputs, social bias, privacy leakage [42] [126], and visual gender or racial bias.Its few-shot capabilities may also help mitigate some risks, alongside mitigation strategies discussed in Appendix D.2.Source: §5 · p. 10

Conclusion

  • Flamingo is a general-purpose family of models for image and video tasks requiring minimal task-specific training data, with interactive abilities extending beyond traditional vision benchmarks.The results suggest that connecting pretrained large language models with powerful visual models is an important step toward general-purpose visual understanding.Source: §5 · p. 10

Checklist … A.1.4 Transformer architecture

The paper supplements its method with implementation details, architectural explanations, evaluation procedures, and documentation of limitations, reproducibility, compute, data practices, and societal impacts. The model details specify how visual inputs are compressed, integrated with a frozen language model, selectively aligned to interleaved text, and configured across transformer components.

Checklist

  • The checklist states that limitations, negative societal impacts, consent, personally identifiable information, and offensive content are discussed in the paper’s designated sections and datasheets.The data were automatically scraped from millions of webpages, with datasheets provided in Appendix F.Source: Checklist · p. 21
  • The checklist reports training details and compute resources, but not code, data, repeated-run error bars, or new released assets.The largest run used 1536 TPU chips for 15 days; code and data are proprietary, and repeated runs were considered computationally unjustified.Source: Checklist · p. 21

Appendix

  • The appendices provide expanded method, evaluation, dataset, experiment, qualitative-result, discussion, model-card, datasheet, and visual-attribution documentation.Appendix A covers model, few-shot evaluation, and training datasets; Appendix B covers additional experiments and ablations; Appendices C–G cover qualitative results, discussion, model card, datasheets, and credits.Source: Appendix · p. 22

A.1.1 Perceiver Resampler

  • The Perceiver Resampler converts variable-size spatio-temporal visual features into a fixed number of tokens using learned latent queries and temporal position encodings.It omits explicit spatial encodings, concatenates latent-derived keys and values with visual keys and values, and found this slightly better than alternatives.Source: Figure 5 · p. 23, §A.1.1 · p. 23

A.1.2 GATED XATTN-DENSE details

  • GATED XATTN-DENSE layers connect visual information to the frozen language model, with gating values becoming nonzero across layers during Flamingo-3B training.Text tokens selectively cross-attend only to visual tokens from the last preceding image or video through masking, while the optimization effects remain incompletely understood.Source: §A.1.2 · pp. 23–24

A.1.3 Multi-visual input support

  • For interleaved image, video, and text sequences, each text position uses preceding text and visual inputs determined by the last preceding visual item.The function φ maps each text position to that visual item’s index, or zero when no visual input precedes it.Source: §A.1.3 · p. 24

A.1.4 Transformer architecture

  • The transformer configurations specify layers, hidden dimensions, attention heads, activations, and feed-forward sizes for each Flamingo component.The feed-forward hidden size is 4D; the frozen language model uses GeLU, whereas trainable transformer layers use Squared ReLU, which outperformed GeLU.Source: §A.1.4 · pp. 24–25, Table 4 · p. 25

Appendix

  • Few-shot evaluation prompts interleave task-specific support examples with a query, using standardized output or question-answer formatting for vision-to-text and visual-question-answering tasks.The prompt prepends “Output:” for vision-to-text responses and uses “Question: {question} Answer: {answer}” for visual question answering.Source: Appendix · p. 22, Figure 8 · p. 25

A.2 In-context few-shot evaluation details … A.3.2 M3W image-placement augmentation

Flamingo evaluation uses multimodal prompts with flexible few-shot, zero-shot, retrieval, and ensembling strategies. Its M3W training data is filtered and augmented by randomized image-placement indices to address uncertain image-text correspondence.

A.2 In-context few-shot evaluation details

  • Few-shot prompts concatenate randomly ordered image/video-text support examples before a visual query, enabling in-context task adaptation.Each support example contains an input visual and expected response or task-specific information.Source: §A.2 · p. 25
  • Open-ended predictions stop at the first <EOC> token using beam search with beam size 3, whereas close-ended outputs are ranked by model log-likelihood.Close-ended scoring independently appends every candidate output to the query image.Source: §A.2 · p. 25
  • Zero-shot evaluation uses two text-only downstream examples without their visuals, avoiding prompt engineering and reducing bias from single-example prompting.One example is highly detrimental, while adding more than two helps only marginally; close-ended tasks need no text example.Source: §A.2 · p. 26
  • RICES retrieves visually similar support examples, orders them from least to most similar, and places the most similar example immediately before the query.This limits prompt length and supports classification settings with hundreds of classes.Source: §A.2 · p. 26
  • Prompt ensembling averages close-ended log-likelihoods across six random permutations of selected few-shot examples and can be combined with RICES.The permutations apply to ranked nearest neighbors selected by retrieval.Source: §A.2 · p. 26

A.3.1 M3W collection

  • M3W collection filters non-English, explicit, low-quality, repetitive, and unsuitable-image content, retaining documents with salient interleaved text and images.Images below 64 pixels in either dimension, extreme aspect ratios, and unambiguously low-quality images are removed.Source: §A.3 · p. 26, §A.3.1 · p. 27

A.3.2 M3W image-placement augmentation

  • M3W image-text correspondence is generally unknown, so training assumes each text token aligns with either the preceding or following image.This simplification can create semantically unnatural pairings roughly half the time.Source: §A.3.2 · p. 27
  • Randomizing image-placement indices provides a small advantage over always selecting the previous or next image, suggesting beneficial data augmentation.The method assumes the relevant image is either immediately before or after each text token, and randomization chooses between these alternatives.Source: §A.3.2 · p. 27

A.3.3 LTIP and VTP: Visual data paired with text … B.1.3 Contrastive model details

Flamingo is trained with large-scale paired image-text and video-text data, carefully deduplicated against evaluation benchmarks, and implemented in scaled model variants with contrastive visual pretraining. Training uses multimodal augmentation, fixed dataset mixtures, and a contrastive vision encoder whose weights initialize the main models.

A.3.3 LTIP and VTP: Visual data paired with text

  • LTIP provides 312 million image-caption pairs and VTP provides 27 million short videos with richer descriptions than ALIGN, whose captions average 12.4 tokens versus 20.5 for LTIP.ALIGN contains 1.8 billion images, while VTP videos average approximately 22 seconds; all paired datasets use image or video boundary tokens and are deduplicated against benchmarks.Source: §A.3.3 · p. 27

A.3.4 Dataset deduplication against evaluation tasks

  • M3W was not deduplicated, but among 1,314 potential benchmark duplicates from its 185 million images, only 125 were exact duplicates.LTIP and ALIGN were deduplicated against ImageNet, COCO, OK-VQA, VQAv2, Flickr30k, and VisDial; VTP was not deduplicated because its videos came from sources distinct from the video evaluation datasets.Source: §A.3.4 · p. 28

B.1.1 Models

  • The number of GATED XATTN-DENSE layers is selected by trading downstream performance against memory constraints, with small-scale findings transferred to the largest architecture.Flamingo-3B adds 1.4B learned parameters, Flamingo-9B adds 1.8B, and Flamingo-80B adds 10B.Source: §B.1.1 · pp. 28–29
  • Flamingo scales across 3B, 9B, and 80B models by increasing frozen language-model size from 1.4B to 70B while keeping the vision encoder frozen and the Resampler near 200M parameters.The models add GATED XATTN-DENSE layers before every block, every fourth block, or every seventh block respectively; the largest model uses the frozen Chinchilla 70B language model [42].Source: §B.1.1 · pp. 28–29

B.1.2 Training details for the Flamingo models

  • Stochastic space-prefixing improves performance by making the tokenizer invariant to whether paired text begins with a space, while visual inputs use 320 × 320 resizing, flips, and color augmentation.Video training samples 8 frames at 1 FPS, whereas inference accepts 30 frames at 3 FPS through interpolated Resampler temporal embeddings.Source: §B.1.2 · p. 29
  • Training uses M3W, ALIGN, LTIP, and VTP with fixed weights of 1.0, 0.2, 0.2, and 0.03, respectively, for 500k steps unless otherwise specified.AdamW uses a learning rate warmed linearly to 10^-4 over 5,000 steps, global norm clipping of 1, and weight decay of 0.1 except for the Resampler.Source: §B.1.2 · p. 29

B.1.3 Contrastive model details

  • The contrastive vision encoder jointly trains vision and language encoders by maximizing paired-image/text similarity and minimizing in-batch unpaired similarity with bidirectional CLIP-style losses [85].Embeddings are L2-normalized and mean-pooled before projection; a learnable inverse-temperature parameter is used, and only the contrastive vision-encoder weights initialize Flamingo.Source: §B.1.3 · p. 30
  • The vision encoder is pretrained on ALIGN and LTIP at 288 × 288 resolution with a 1376-dimensional joint space, batch size 16,384, and 1.2 million update steps.Evaluation tracks zero-shot image classification and retrieval using image-text matching and prompt ensembling with multiple class-name templates.Source: §B.1.3 · p. 30

B.1.4 Evaluation benchmarks … B.3 Extended ablation studies

Flamingo is evaluated across diverse few-shot multimodal, classification, fine-tuning, and zero-shot retrieval settings, using controls intended to separate design validation from unbiased performance estimation. Results show strong gains from larger support sets, fine-tuning, and contrastive pretraining, while revealing limitations on classification and computational cost.

B.1.4 Evaluation benchmarks

  • The evaluation covers 16 multimodal image/video and language benchmarks spanning visual question answering, captioning, visual dialogue, and ImageNet and Kinetics classification.Five multimodal benchmarks plus ImageNet and Kinetics form the development set, while 11 remaining benchmarks provide unbiased few-shot estimates without design validation.Source: §B.1.4 · pp. 30–32, Table 6 · p. 31
  • Benchmark evaluation uses disjoint validation-support, validation-query, test-support, and test-query subsets; test-query results follow prior-work reporting for apples-to-apples comparison, while training data supplies validation-query and support subsets.Source: §B.1.4 · pp. 30–31

B.1.5 Few-shot learning evaluation hyperparameters

  • The evaluation uses one jointly optimized hyperparameter and prompt set across development benchmarks, avoiding benchmark-specific cross-validation except for specified exceptions.Most non-question-answering tasks use “Output: {output}”, while question answering and visual dialogue use “Question: {question} Answer: {answer}”; TextVQA relies on Flamingo’s OCR rather than provided transcripts.Source: §B.1.5 · p. 32

B.1.6 Dialogue prompt

  • The dialogue examples use a conversational prompt in which a human User sends images and Flamingo describes them, answers follow-up questions, and can express uncertainty.The prompt frames Flamingo as an intelligent visual AI conversing with a user.Source: §B.1.6 · p. 32

B.2.1 Few-shot learning on classification tasks

  • On ImageNet, RICES improves classification by 9.2% over random selection of the same 16 support examples, while prompt ensembling provides further gains.Larger models and more shots also improve classification, although Flamingo remains below dominant contrastive classifiers.Source: §B.2.1 · p. 33, Table 7 · p. 33

B.2.2 Fine-tuning Flamingo as a pretrained vision-language model

  • Fine-tuning freezes the language-model layers, trains Flamingo layers and the visual encoder at 480 × 480 resolution, and improves results over few-shot learning at substantial computational cost.Task-specific fine-tuning sets new state of the art on five tasks, but hyperparameter tuning and large datasets increase cost; metric-specific optimization could further improve some results.Source: §B.2.2 · pp. 33–34
  • Flamingo achieves new state of the art on five fine-tuned tasks, including 82.0% on VQAv2 versus 67.3% with 32-shot prompting and 81.3% for the prior state of the art.The five tasks are VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes.Source: §B.2.2 · p. 34, Table 8 · p. 34

B.2.3 Zero-shot performance of the pretrained contrastive model

  • The frozen contrastively pretrained vision encoder outperforms CLIP [85], ALIGN [50], and Florence [140] on retrieval, but underperforms zero-shot classification leaders on Kinetics700 and ImageNet.The contrastive model favors rich scene descriptions for retrieval, whereas label-oriented training benefits ImageNet classification; additional ablations compare variants against the baseline Flamingo run and track training time.Source: §B.2.3 · p. 34, Table 9 · p. 35, Table 10 · p. 35

B.3.1 Flamingo

Table 10 ablations show that Flamingo’s strongest design choices are single-image cross-attention, high-quality frozen language and vision pretraining, randomized image placement, and a medium Resampler.

Experimental setup

  • The ablations report per-task and Overall scores for Flamingo-3B on five validation benchmarks with four shots, alongside additional LAION400M and CLIP ViT L-14 reference experiments.The main study trains for 1 million gradient steps under task-specific batch sizes.Source: §B.3.1 · pp. 35–36

Resampler size

  • The medium Resampler achieves the best performance, while larger Resamplers can destabilize training when scaled with the frozen language model.Flamingo therefore uses the same medium Resampler size across models.Source: §B.3.1 · p. 35

Image cross-attention

  • A 7.2% higher Overall score favors attending only to the single most recent image, while language self-attention still propagates information from earlier images.This strategy generalizes from N=5 training images to up to N=32 at test time; explicit image disambiguation alternatives were less robust.Source: §B.3.1 · pp. 35–36

M3W image placement

  • Randomizing whether webpage text attends to the previous or following image improves performance over always selecting either image.The augmentation samples the direction uniformly, with p_next=1/2 as the default.Source: §B.3.1 · p. 36

Pretraining and freezing

  • Using C4-pretrained language models reduces Overall performance by −7.9%, while training the vision encoder from scratch or from pretrained weights causes −9.3% or −2.6% drops.The language-model degradation is strongest on language-understanding tasks, whereas COCO and VATEX remain comparable; vision pretraining also raises compute cost.Source: §B.3.1 · p. 36

Language-model freezing

  • Freezing the pretrained language model outperforms co-training it on MassiveText and is computationally cheaper because it avoids language-model updates and extra text-only training.This benefit is especially relevant for Flamingo-80B, where almost 90% of weights are frozen.Source: §B.3.1 · p. 36

B.3.2 Dataset mixing strategies for the contrastive pretraining

LTIP improves contrastive pretraining despite being smaller than ALIGN, and combining both datasets performs best when gradients are accumulated rather than batches merged or alternated.

Dataset mixing strategies

  • LTIP-only training outperforms ALIGN-only training despite LTIP being smaller than ALIGN by a factor of 6.This suggests dataset quality can matter more than scale in the evaluated regime.Source: §B.3.2 · p. 36
  • A model trained on both ALIGN and LTIP outperforms models trained on either dataset individually.The comparison uses a small model evaluated after training on the individual datasets and their combinations.Source: §B.3.2 · p. 37, Table 11 · p. 37
  • The ablation trains a small NFNet-F0/BERT-mini model with batch size 2048 for 1 million gradient-calculation steps.Models are trained on ALIGN, LTIP, or a mixture using three combination regimes.Source: §B.3.2 · p. 37
  • Across all evaluation metrics, Accumulation outperforms the other dataset-combination strategies.The comparison covers Data merged, Round-robin, and Accumulation training regimes.Source: §B.3.2 · p. 37, Table 11 · p. 37

C Qualitative results · D Discussion

Qualitative results show that Flamingo adapts to image-question and caption interactions, sustains multimodal dialogue, separately attends to multiple images, and integrates video frames for temporally grounded answers. These samples use greedy decoding rather than the beam search used for quantitative benchmarks.

C Qualitative results

  • Qualitative samples use greedy decoding for faster sampling, whereas quantitative benchmark results use beam search with beam width 3.Source: C Qualitative results · p. 37
  • Flamingo answers image-based questions and caption prompts despite not being specifically trained for question answering, using implicit inference over visually referred objects.It can recall relevant knowledge for unnamed objects and answer property-based queries.Source: C Qualitative results · p. 37
  • Flamingo preserves contrastive vision networks’ character-recognition capability, including for text that is relatively small in the image [85].Source: C Qualitative results · p. 37
  • Across several interaction rounds, Flamingo can attend to interleaved images and answer questions that cannot be guessed from language alone.The examples use a brief dialogue followed by user interaction with image insertions.Source: C Qualitative results · p. 37
  • Flamingo separately attends to multiple images and handles simple comparisons and inferences properly.Source: C Qualitative results · p. 37
  • Flamingo integrates information from multiple video frames, including frames scanning through a scene or text.Source: C Qualitative results · p. 38
  • Flamingo answers video questions involving temporal understanding, including relations expressed by “after.”Source: C Qualitative results · p. 38

D.1 Limitations, failure cases and opportunities … D.2.1 Benefits

The discussion identifies important limitations of Flamingo’s few-shot approach, including classification gaps, inherited language-model weaknesses, prompt and compute sensitivity, and diminishing returns from additional shots. It also outlines opportunities spanning broader interfaces, scaling laws, accessibility, and recycling pretrained models.

D.1 Limitations, failure cases and opportunities

  • Flamingo lags contrastive models on classification because language modeling does not directly optimize image-to-text retrieval, while prompt and data-distribution biases complicate few-shot calibration.Calibration can mitigate these biases only when a prior over labels, such as uniformity, is assumed; objectives, architectures, and evaluation procedures that bridge the gap remain open directions.Source: §D.1 · p. 38
  • Causal language-model inheritance limits bidirectional expressiveness, long-context generalization, sample efficiency, and grounding, with VisDial prompts reaching 4096–8192 tokens beyond the 2048-token training limit.A 32-shot VisDial prompt contains at least 672 sentences, so reported results were capped at 16 shots; inherited priors may also contribute to hallucinations and ungrounded guesses.Source: §D.1 · p. 38, Figure 13 · p. 42

−→ Flamingo.

  • In-context inference cost grows linearly with shots when keys and values are cached, quadratically otherwise, while gradient-based methods have constant inference complexity with respect to shots.In-context learning is also sensitive to demonstration order and format.Source: §D.1 · p. 38, −→ Flamingo · p. 41
  • Performance plateaus rapidly beyond 32 shots, although RICES mitigates this limitation for classification; task location and example format may explain why additional demonstrations often provide little benefit.The authors observe non-trivial zero-shot performance with prompts containing no images, suggesting that recognizing the task and its format can matter more than learning input-output mappings from extra examples.Source: −→ Flamingo · p. 41
  • In-context learning is highly effective with 32 samples or fewer, but no few-shot method works universally; combining methods may help when hundreds of samples are available.Its advantages include minimal tuning, strong performance in very low-data regimes, and inference-only deployment, whereas gradient-based methods often require thousands of examples and careful optimization.Source: §D.1 · p. 38, −→ Flamingo · p. 42
  • Extending Flamingo to structured, dense, continuous-space, temporal, spatio-temporal, or audio-conditioned tasks could broaden its capabilities beyond natural-language interfaces.The paper also calls for vision-language scaling laws, potentially relating aggregate downstream task performance to model scale rather than relying only on perplexity.Source: −→ Flamingo · p. 42

D.2.1 Benefits

  • Flamingo’s task generalization and chat-like interface could make visual recognition more accessible to non-experts and low-resource applications, including assistance for visually impaired people through VizWiz.Dialogue may also improve understanding of model bias, fairness, and toxicity.Source: §D.2.1 · p. 43
  • Recycling pretrained frozen language models and visual encoders can introduce new modalities without expensive retraining, offering practical and environmental benefits as models grow larger.This approach motivates further research into efficiently repurposing existing models rather than training them from scratch.Source: §D.2.1 · p. 43

D.2.2 Risks and mitigation strategies … F Datasheets

Flamingo’s preliminary risk assessment identifies inherited language-model risks and additional visual-input risks, while proposing mitigation and safe-deployment strategies. The accompanying model card documents its architecture, intended research scope, evaluation coverage, limitations, and datasheet references.

D.2.2 Risks and mitigation strategies

  • Flamingo inherits large-language-model risks, including offensive language, social biases, stereotypes, and private-information leakage, even though its unprompted toxicity propensity is reported as rather low.Gender bias remains present despite being lower than in previous models; suggested mitigations include policy, product design, better benchmarks, and prompting.Source: §D.2.2 · p. 43
  • COCO captioning showed no statistically significant gender or skin-tone differences in CIDEr scores, with the smallest p-value equal to 0.25, although larger samples could reveal effects.The comparison used two-tailed unequal-variance t-tests across per-sample CIDEr scores.Source: §D.2.2 · p. 44, Table 12 · p. 44
  • Image-conditioned captions were judged appropriate despite some Perspective API toxicity flags on COCO safe-for-work images, but toxic outputs remain possible with unsafe images or toxic text.The authors call for more thorough study before production deployment.Source: §D.2.2 · p. 44
  • Flamingo could support safety mitigation by filtering harmful training data, down-ranking offensive outputs, generating red-team test cases, and providing visual or external-quote references.Qualitative examples also suggest that its text interface can explain its own outputs.Source: §D.2.2 · pp. 44–45

Model Details

  • Flamingo is out of scope for harmful or deceitful visually conditioned generation and should not be used downstream without application-specific safety and fairness analysis; it is trained on English data.The model card identifies language and downstream application factors as relevant considerations.Source: Model Details · p. 45

F Datasheets

  • The model card reports evaluation across 18 vision-language benchmarks spanning classification, captioning, visual question answering, and visual dialogue, while noting that intersectional biases were not investigated.It points to Appendix D.2.2 for image-conditioned toxicity analysis and to the datasheets in Appendix F.1, F.2.1, and F.2.2.Source: Model Details · pp. 45–46, E Flamingo Model Card · p. 45, Table 13 · p. 47

Model Details

  • Flamingo sets a new few-shot state of the art across open-ended vision-language tasks and surpasses fine-tuned state of the art on 6 of 16 tasks with far less task-specific data.The model card refers to Section 3 for the full quantitative study.Source: Model Details · p. 46

F.1 M3W dataset … G Credit for visual content

The paper documents its multimodal datasets using datasheets and describes licensing and provenance for visual content used throughout the paper. M3W supports pretraining multimodal language and vision models, while LTIP and VTP use random training and development splits.

F.1 M3W dataset

  • M3W uses random training and development splits and filters explicit content, but does not otherwise remove potentially offensive web content.The authors note that offensive, insulting, or threatening material may remain because such content is prevalent on the web.Source: §F.1 · p. 48

F.2 Image and video text pair datasets

  • The image and video text-pair datasets are presented as datasheet-documented resources supporting multimodal pretraining, with random training and development splits reported for their datasets.The supplied passages specifically substantiate these practices for M3W, LTIP, and VTP.Source: §F.1 · p. 48, §F.2.1 · p. 50, §F.2.2 · p. 52

F.1 M3W dataset

  • M3W is documented using the datasheet framework of Gebru et al., with its datasheet provided in Table 14.The dataset is used to pretrain multimodal language and vision models.Source: §F.1 · pp. 47–48, Table 14 · p. 49

F.2.1 Datasheet for LTIP

  • LTIP is documented in Table 15 under the datasheet framework of Gebru et al. [30] and uses random training and development splits.The supplied passage identifies the recommended split procedure but does not provide further dataset statistics.Source: §F.2.1 · p. 50, Table 15 · p. 51

F.2.2 Datasheet for VTP

  • VTP is documented in Table 16 under the datasheet framework of Gebru et al. [30] and uses random training and development splits.The datasheet passage supplies the documentation framework, while the split procedure is stated separately.Source: §F.2.2 · p. 52, Table 16 · p. 53

G Credit for visual content

  • For Figure 1, the cited sources include licensed Unsplash images, public-domain images, Wikimedia Commons material under CC BY-ND 2.0, YFCC100M video, and DALL·E 2 [89].Source: G Credit for visual content · pp. 53–54
  • Additional visuals come from Flaticon, Sketchfab, CLIP [85], Chicago and Tokyo Unsplash pictures, COCO, and Wikimedia Commons under CC BY-ND 2.0.The credits cover model figures and qualitative figures as well as dialogue illustrations.Source: G Credit for visual content · p. 54

Related papers

Loading 2204.14198v2…