PaperlayerView exact revision on arXiv

Source-linked AI summary

Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, Karen Simonyan

arXiv:2204.14198v2cs.CVcs.AIcs.LG

TL;DR

Adapting multimodal models to new tasks from only a few annotated examples remains difficult because conventional fine-tuning is data- and resource-intensive. Flamingo bridges pretrained vision and language models to process interleaved images, videos, and text, achieving few-shot state-of-the-art performance across diverse vision-language tasks and surpassing fine-tuned systems on 6 of 16 tasks.

  • Problem

    Multimodal models still commonly require many annotated examples, task-specific tuning, and substantial resources to adapt to new tasks.

  • Method

    Flamingo bridges pretrained vision and language models with cross-attention architecture that accepts arbitrarily interleaved images, videos, and text and generates open-ended text.

  • Results

    Flamingo achieves few-shot state-of-the-art performance across a wide range of vision-language tasks and surpasses the fine-tuned state of the art on 6 of 16 tasks.

  • Takeaways & Limitations

    Flamingo demonstrates that connecting pretrained large language models with powerful visual models supports general-purpose visual understanding with minimal task-specific training data.

  • Takeaways & Limitations

    In-context learning is highly sensitive to demonstrations, and its inference cost and absolute performance scale poorly beyond the low-data regime.

Abstract

from arXiv · show

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.

1 Introduction

Flamingo addresses the need for rapid multimodal adaptation by introducing VLMs that learn new vision-language tasks from a few prompted examples, without task-specific tuning. Its architecture accepts arbitrarily interleaved visual and textual inputs and generates open-ended text, while achieving strong few-shot performance across diverse tasks.

  • Motivation: Existing vision systems commonly require pretraining followed by fine-tuning on many thousands of annotated examples for each target task.The introduction identifies this data-intensive adaptation paradigm as a central limitation motivating Flamingo.
  • Contributions: On 16 evaluated tasks, Flamingo surpasses the fine-tuned state of the art on 6 tasks using orders of magnitude less task-specific training data.The model achieves this by being prompted with a few input/output examples.
  • Method: Flamingo is trained on a carefully chosen mixture of complementary large-scale multimodal web data without machine-learning annotations.This training enables direct few-shot adaptation to vision tasks without task-specific tuning.
  • Architecture: Flamingo takes visual data interleaved with text as input and produces free-form text as output.This input-output format is central to the model family’s visual-language design.
  • Contributions: Flamingo introduces VLMs that perform captioning, visual dialogue, and visual question-answering from only a few input/output examples.The models generate text in an open-ended manner while accepting interleaved visual and textual data.

2 Approach

Flamingo bridges frozen vision and language models to process arbitrarily interleaved images, videos, and text, producing free-form text. Its architecture uses a Perceiver Resampler and gated cross-attention within a frozen language model, enabling in-context learning from multimodal prompts.

  • 2 Approach: Flamingo accepts interleaved text with images or videos and generates free-form text through a Transformer decoder conditioned on visual representations.The decoder interleaves pretrained, frozen text-only language-model blocks with newly trained blocks that cross-attend to Perceiver outputs.
  • Vision Encoder: The frozen NFNet F6 vision encoder converts images into spatial feature sequences and samples video frames at 1 FPS for independent encoding.Its final-stage 2D spatial grid is flattened into a 1D sequence, while video frames produce a 3D feature representation.
  • Perceiver Resampler: The Perceiver Resampler converts variable-size image or video features into 64 visual tokens, reducing the computational complexity of vision-text cross-attention.It connects the vision encoder to the frozen language model using a predefined number of learned latent queries.
  • GATED XATTN-DENSE Layers: Flamingo inserts gated cross-attention dense blocks between frozen language-model layers, using tanh-gating so initialization preserves the original model’s outputs.The new blocks are trained from scratch, with keys and values from vision features and queries from language inputs.
  • Interleaved Inputs: Image-causal masking makes each text token attend directly to the visual tokens of the immediately preceding image rather than all previous images.The model can nevertheless depend on earlier images through the language representations propagated across the sequence.
  • Training and In-Context Learning: Flamingo is trained on web-scraped interleaved image-text, image-text pair, and video-text pair datasets, then evaluated with multimodal prompts containing support examples and a query input.Training minimizes a weighted sum of per-dataset text negative log-likelihoods, while task adaptation uses interleaved image/video-text examples for in-context learning.

3 Experiments

Flamingo achieves strong few-shot adaptation across 16 multimodal benchmarks, generalizes beyond development tasks, and improves further through scaling and fine-tuning. Ablations identify training-data mixture, visual conditioning, compute allocation, vision encoders, and language-model freezing as important design factors.

  • Few-shot results: With as few as four examples per task, Flamingo outperforms prior zero-shot or few-shot methods across 16 benchmarks and is often competitive with extensively fine-tuned systems.Some comparisons involve methods fine-tuned on up to hundreds of thousands of annotated examples.
  • Scaling with respect to parameters and shots: Few-shot performance improves with model size and shot count, with the largest model benefiting most from more shots and supporting up to 32 inference images or videos.The models were trained with sequences limited to five images on M3W.
  • Fine-tuned results: Fine-tuning the largest Flamingo model sets a new state of the art on five additional tasks: VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes.This fine-tuning uses a short schedule, a small learning rate, and an unfrozen vision backbone for higher input resolution.
  • Importance of the training data mixture: Removing M3W decreases performance by more than 17%, removing paired image-text data decreases it by 9.8%, and removing paired video-text data harms all video tasks.Replacing the image-text pairs with LAION-400M causes a slight degradation.
  • Visual conditioning and compute/memory trade-offs: Removing 0-initialized tanh gating drops the overall score by 4.2% and causes training instabilities, while inserting GATED XATTN-DENSE blocks every fourth layer accelerates training by 66% for a 1.9% score decrease.The trade-off concerns trainable parameters, time complexity, and overall performance.
  • Vision encoder and language-model freezing: NFNet-F6 outperforms CLIP ViT-L/14 by 5.8% and NFNet-F0 by 8.0%, while training the language model from scratch or fine-tuning it reduces performance by 12.9% or 8.0%, respectively.The language-model results indicate catastrophic forgetting, supporting freezing the pretrained language-model layers.

4 Related work

Flamingo builds on transformer-based language modelling, few-shot adaptation, and prior vision-language approaches while differing in its adaptation requirements and multimodal input flexibility. It also emphasizes training on entire multimodal webpages with interleaved images and text, beyond paired vision-text data.

  • Language modelling and few-shot adaptation: Flamingo uses the 70B Chinchilla language model as its base language model.
  • When language meets vision: Unlike prior BERT-inspired vision-language approaches, Flamingo models do not require fine-tuning on new tasks.
  • When language meets vision: Flamingo introduces the first language model that can ingest arbitrarily interleaved images, videos, and text.
  • Web-scale vision and language training datasets: Manually annotated vision-language datasets are costly and relatively small, typically 10k-100k examples.
  • Web-scale vision and language training datasets: Beyond automatically scraped paired vision-text data, Flamingo trains on entire multimodal webpages containing interleaved images and text as a single sequence.

5 Discussion

Flamingo is presented as a flexible, general-purpose model family for image and video tasks with minimal task-specific data, while the discussion highlights inherited language-model limitations, classification gaps, in-context-learning tradeoffs, and societal risks.

  • Limitations: Flamingo inherits pretrained language models’ weaknesses, including occasional hallucinations, poor generalization to longer sequences, and poor training sample efficiency.The authors identify addressing these issues as a way to improve Flamingo and visual-language-model capabilities.
  • Limitations: Flamingo’s classification performance lags behind state-of-the-art contrastive models, which directly optimize text-image retrieval.The authors frame combining contrastive strengths with Flamingo’s broader open-ended task coverage as an important research direction.
  • Limitations: In-context learning works with only a few dozen examples and simplifies deployment through inference-only use, generally without hyperparameter tuning, but has application-dependent drawbacks.The passage also notes that in-context learning is highly sensitive to various factors, though the supplied text ends before specifying them.
  • Societal impacts: Flamingo may lower barriers to beneficial and malicious applications for non-experts in data-starved settings, while posing risks including offensive language, social biases, stereotypes, and information leakage.These societal risks are described as overlapping with those of large language models.
  • Conclusion: The authors propose Flamingo as a general-purpose family for image and video tasks with minimal task-specific data and report qualitative interactive abilities beyond traditional vision benchmarks.They conclude that connecting pretrained large language models with powerful visual models is an important step toward general-purpose visual understanding.

Checklist … A.1.4 Transformer architecture

The appendices document the model architecture, multimodal input handling, evaluation and training details, supplementary results, limitations, and reproducibility considerations. They also specify key implementation choices, including visual resampling, gated cross-attention, selective masking, and transformer configurations.

  • Checklist: The checklist reports that training details and compute resources were provided, but proprietary code and data and repeated-run error bars were not.The largest run used 1536 TPU chips for 15 days; the authors attribute the absence of error bars to low observed variance and computational cost.
  • Appendix: The appendix organizes additional method, evaluation, dataset, experiment, qualitative-result, discussion, model-card, datasheet, and visual-attribution material across Appendices A–G.It includes few-shot prompting, prediction procedures, zero-shot numbers, retrieval, ensembling, dataset collection and deduplication, further results, ablations, qualitative examples, limitations, and societal impacts.
  • A Method: Appendix A provides implementation details for the Perceiver Resampler, GATED XATTN-DENSE layer, multi-image/video attention, and model hyperparameters.Appendix B adds training, evaluation, classification, fine-tuning, zero-shot, and ablation details.
  • Appendix: The appendices provide additional qualitative examples and a fuller discussion of limitations, failure cases, broader impacts, and societal impacts.Qualitative material includes single-image, dialogue, and video examples.
  • A.1.1 Perceiver Resampler: The Perceiver Resampler cross-attends learned latent queries to flattened spatio-temporal visual features and outputs a fixed number of visual tokens.Its output count is independent of image resolution or the number of video frames.
  • A.1.2 GATED XATTN-DENSE details: GATED XATTN-DENSE layers connect visual representations to a frozen language model, whose layers increasingly use visual information during training.Selective masking ensures each text token cross-attends only to the visual tokens from the last preceding image or video; the optimization effects require further study.
  • A.1.3 Multi-visual input support: For interleaved visual-text sequences, a position function assigns each text token to the index of the last preceding image or video, enabling selective visual attention.The implementation inserts image tags and uses masking to restrict accessible visual tokens.
  • A.1.4 Transformer architecture: Table 4 lists layers, hidden dimensions, attention heads, activations, and key/value dimensions for each Flamingo transformer component, with feed-forward hidden size 4D.The frozen language model uses GeLU, while trainable transformer layers use Squared ReLU, which the authors found to outperform GeLU.

A.2 In-context few-shot evaluation details … A.3.2 M3W image-placement augmentation

Flamingo evaluates in-context adaptation by conditioning on visual-text support examples and scoring open- or close-ended predictions with task-specific procedures. Its training data includes filtered, interleaved web content, while M3W image placement is randomized between adjacent images to address uncertain webpage correspondence and improve augmentation.

  • A.2 In-context few-shot evaluation details: Flamingo evaluates few-shot adaptation from support pairs of images or videos with expected textual responses, followed by a single visual query.Support examples may include task-specific information such as a question.
  • A.2 In-context few-shot evaluation details: Open-ended predictions stop at the first <EOC> token, while close-ended evaluation scores each candidate output by the model’s estimated log-likelihood.Unless specified otherwise, open-ended generation uses beam search with beam size 3.
  • A.2 In-context few-shot evaluation details: Zero-shot evaluation avoids validated prompts because prompt validation requires annotated examples and can be non-robust with limited access to them.The passage contrasts this with prompt engineering used to condition models without few-shot examples.
  • A.2 In-context few-shot evaluation details: Retrieval-based example selection limits support-set prompts when including all examples is expensive or exceeds the sequence length used during training.The method is motivated by concerns about prompt cost and generalization for oversized prompts.
  • A.2 In-context few-shot evaluation details: Prompt ensembling averages answer log-likelihoods over 6 random permutations of selected few-shot examples and can be combined with RICES.The ensembling procedure is described for close-ended evaluation.
  • A.3 Training dataset details: Flamingo training uses a carefully chosen mixture of datasets, including M3W web documents containing filtered plain text interleaved with images.M3W collection removes non-English, explicit, low-quality, repetitive, and unsuitable-image content, including images below 64 pixels in width or height.
  • A.3.2 M3W image-placement augmentation: M3W image-text correspondence is generally unknown, so the method assumes the most relevant image at each text position is either the previous or next image.This can produce semantically unnatural associations when webpage layout does not reveal correspondence.
  • A.3.2 M3W image-placement augmentation: Ablation results show a small advantage for setting p_next = 1⁄2 over always using the previous or next image, suggesting randomization provides beneficial data augmentation.The comparison covers p_next = 0, p_next = 1, and p_next = 1⁄2.

A.3.3 LTIP and VTP: Visual data paired with text … B.1.3 Contrastive model details

Flamingo is trained with large paired image-text and video-text datasets, then evaluated using scalable model architectures, carefully specified training procedures, deduplication safeguards, and contrastive vision-language pretraining. The approach combines broad multimodal data with frozen vision and language components, efficient video handling, and shared embedding objectives.

  • A.3.3 LTIP and VTP: Visual data paired with text: ALIGN provides 1.8 billion image–alt-text pairs, augmented by LTIP’s 312 million images and VTP’s 27 million short videos to improve paired visual-text training data.ALIGN is large but noisy and limited to images, motivating the additional datasets.
  • A.3.4 Dataset deduplication against evaluation tasks: M3W deduplication found 1314 potential duplicates among 185M images from six benchmark datasets, supporting the assessment that benchmark overlap was limited.The passage reports near-duplicate statistics for ImageNet, COCO, OK-VQA, VQAv2, Flickr30k, and VisDial.
  • B.1.1 Models: The models scale frozen language backbones from 1.4B to 7B and 70B while keeping the frozen vision encoder fixed and using a Perceiver Resampler of approximately 200M parameters.The vision encoder is an NFNet-F6 trained contrastively, unless otherwise stated in ablations.
  • B.1.1 Models: Flamingo-3B adds a GATED XATTN-DENSE layer before every transformer block, while Flamingo-9B inserts one before every fourth block and adds 1.8B learned parameters.The 3B configuration adds 1.4B additional learned parameters; layer frequency balances memory constraints and downstream performance.
  • B.1.2 Training details for the Flamingo models: Training resizes visual inputs to 320 × 320, samples eight video frames at 1 fps, and uses 30 frames at 3 FPS during inference via interpolated temporal embeddings.The higher training resolution exceeds the 288 × 288 contrastive-pretraining resolution.
  • B.1.2 Training details for the Flamingo models: Optimization uses AdamW with global norm clipping of 1, a learning rate warming to 10^-4 over 5000 steps, and 500k training steps unless specified otherwise.Weight decay is 0.1 for other trainable parameters and omitted for the Perceiver Resampler.
  • B.1.3 Contrastive model details: The contrastive vision-language model separately encodes images and text into L2-normalized shared embeddings, maximizing paired similarity and minimizing unpaired similarity with multi-class cross-entropy.The language encoder uses BERT, and outputs are mean-pooled before projection.
  • B.1.3 Contrastive model details: Contrastive pretraining uses ALIGN and LTIP at 288 × 288 resolution with a 1376-dimensional embedding space, batch size 16,384, and 1.2 million update steps on 512 TPUv4 chips.The learning rate decays linearly from 10^-3 to zero, with random color augmentation and horizontal flips.

B.1.4 Evaluation benchmarks … B.3 Extended ablation studies

Flamingo is evaluated across diverse few-shot multimodal and classification benchmarks using carefully separated evaluation subsets and standardized prompts. Additional results show benefits from larger support sets, task-specific fine-tuning, and pretrained visual representations, alongside computational and metric-optimization limitations.

  • B.1.4 Evaluation benchmarks: Flamingo evaluates 16 multimodal image/video-language benchmarks spanning visual question answering, captioning, dialogue, and ImageNet and Kinetics classification.The benchmark suite targets rapid adaptation to diverse image and video tasks.
  • B.1.4 Evaluation benchmarks: The DEV set selects five challenging multimodal benchmarks plus ImageNet and Kinetics to validate design decisions and hyperparameters.These benchmarks were chosen for broad study across captioning, visual question answering, and image/video classification.
  • B.1.4 Evaluation benchmarks: Few-shot evaluation separates validation support, validation query, test support, and test query subsets for adaptation, validation, and final performance estimation.Test-query choices follow prior-work reporting conventions where official validation or test splits are unavailable or used differently.
  • B.1.5 Few-shot learning evaluation hyperparameters: Standardized prompts reduce task-specific tuning, using “Output: {output}” for non-question-answering tasks and “Question: {question} Answer: {answer}” for question answering and dialogue.HatefulMemes, TextVQA, and RareAct receive task-specific handling.
  • B.1.6 Dialogue prompt: Dialogue examples frame Flamingo as an intelligent visual AI that describes user-provided images and answers follow-up questions.The supplied examples include descriptions of a kitten, a dinner gathering, and possible celebrations.
  • B.2.1 Few-shot learning on classification tasks: Larger Flamingo models perform better on ImageNet and Kinetics700, while RICES and prompt ensembling yield large gains when using up to 5000 support examples.The classification experiments examine support sets larger than the current prompt capacity.
  • B.2.2 Fine-tuning Flamingo as a pretrained vision-language model: Fine-tuning on large task-specific datasets sets a new state of the art on VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes, including 82.0% on VQAv2.These gains exceed prior in-context few-shot results but require substantially higher computational cost and hyperparameter tuning.
  • B.2.3 Zero-shot performance of the pretrained contrastive model: The separately contrastively pretrained, frozen Vision Encoder supports zero-shot ImageNet and Kinetics700 classification plus Flick30K and COCO retrieval, outperforming current state-of-the-art dual encoders on retrieval.Extended ablations compare training configurations against the baseline Flamingo run and account for TPU-related accumulation-step differences.

B.3.1 Flamingo

Flamingo ablations identify design choices that improve performance and training stability, including single-image cross-attention, randomized image placement, MassiveText pretraining, and pretrained frozen vision encoders. The model uses a medium Resampler and trains with five images per sequence while generalizing to 32 at test time.

  • Resampler size: The medium Resampler is the best-performing option, while enlarging it alongside the frozen language model can destabilize training.Flamingo uses the medium Resampler as the default across models.
  • Cross-attention strategy: Single-image cross-attention performs 7.2% better in the overall score than attending to all previous images.Flamingo therefore retains the single-image masking strategy, while language self-attention can still propagate information from earlier images.
  • Sequence length: Flamingo uses N=5 images per training sequence but generalizes to N=32 images during few-shot evaluation.The single-image cross-attention strategy reduces training-time image requirements while preserving access to information from earlier images through language self-attention.
  • Image placement augmentation: Randomizing whether text attends to the previous or next image is beneficial on M3W.The default p_next = 1/2 selects the previous or following image uniformly at random.
  • Language model pretraining: Replacing MassiveText with the smaller, less filtered C4 dataset causes a −7.9% overall performance loss.The decrease is especially notable on tasks involving more language understanding.
  • Vision encoder freezing: Training the vision encoder from scratch decreases performance by −9.3%, supporting the use of pretrained frozen vision components.The ablation compares scratch initialization against initialization from contrastive vision-language pretraining.

B.3.2 Dataset mixing strategies for the contrastive pretraining

Combining LTIP with ALIGN improves contrastive pretraining, despite LTIP being substantially smaller, indicating that dataset quality can matter more than scale. Among combination strategies, Accumulation performs best across evaluation metrics.

  • Dataset effects: LTIP-only training outperforms ALIGN-only training on the evaluation metrics despite LTIP being 6× smaller than ALIGN.This suggests dataset quality may be more important than scale in the studied regime.
  • Dataset effects: Training on both ALIGN and LTIP outperforms training on either dataset individually.
  • Experimental setup: The study trains a small NFNet-F0/BERT-mini model for 1 million gradient-calculation steps with batch size 2048.Experiments compare ALIGN, LTIP, and mixtures of the two datasets.
  • Combination strategies: The combined datasets are evaluated using Data merged, Round-robin, and Accumulation regimes.Data merged combines examples within batches; Round-robin alternates dataset batches; Accumulation weights and sums per-dataset gradients before updating parameters.
  • Combination strategies: Accumulation outperforms the other dataset-combination methods across all evaluation metrics.The ablation also finds that LTIP is 5 × smaller than ALIGN, reinforcing the importance of training-data quality over abundance.

C Qualitative results · D Discussion

Flamingo’s qualitative results show flexible image, dialogue, and video interaction, including implicit inference, separate attention to multiple images, and temporal reasoning across frames.

  • C Qualitative results: Qualitative samples use greedy decoding, whereas quantitative benchmark results use beam search with beam width 3.Greedy decoding was selected for faster sampling.
  • C Qualitative results: Flamingo adapts to image-question and image-caption prompts despite not being specifically trained for question-answer formatting.This adaptation is attributed to the capabilities of the pretrained language model.
  • C Qualitative results: On single-image interactions, Flamingo often performs at least one step of implicit inference about properties queried indirectly.Some queried objects are not explicitly named in the prompt.
  • C Qualitative results: Flamingo sustains extended dialogue with interleaved image and text inputs across several rounds of interaction.It can attend to inserted images and answer questions that cannot be guessed from language alone.
  • C Qualitative results: Flamingo separately attends to multiple images and handles simple comparisons and inferences between them.These abilities are demonstrated in interleaved image-text dialogue samples.
  • C Qualitative results: With video inputs, Flamingo integrates information from multiple frames, including videos scanning through a scene or text.Video inputs introduce extra challenges compared with images.
  • C Qualitative results: Flamingo answers video questions involving temporal understanding, including relations expressed by “after.”The examples include temporal reasoning over information distributed across frames.

D.1 Limitations, failure cases and opportunities … D.2.1 Benefits

Flamingo’s limitations include weaker classification, inherited causal-language-model weaknesses, inference and prompt-sensitivity costs, and rapidly plateauing benefits from additional shots. Its opportunities and benefits include richer interfaces, scaling-law analysis, task generalisation, and recycling pretrained components.

  • D.1 Limitations, failure cases and opportunities: Flamingo trails contrastive models on classification because contrastive training directly optimizes text-image retrieval, which classification can approximate.
  • D.1 Limitations, failure cases and opportunities: Causal language modeling is less expressive than bidirectional modeling, so Flamingo inherits weaknesses from its pretrained causal language models.
  • D.1 Limitations, failure cases and opportunities: In-context learning simplifies deployment and works with dozens of examples, but its transformer inference cost scales linearly with shots when prompts are cached and quadratically otherwise.
  • −→ Flamingo.: In-context learning is sensitive to demonstration order and format, while performance plateaus beyond 32 shots; RICES mitigates this issue for classification.
  • −→ Flamingo.: No single few-shot method works universally: in-context learning is highly effective with 32 samples or fewer, while method choice depends on application characteristics and annotation volume.
  • −→ Flamingo.: Natural language provides a versatile interface, but structured outputs such as bounding boxes and spatio-temporal predictions can be cumbersome to express.
  • D.2.1 Benefits: Task generalisation broadens potential applications beyond narrow, task-specific vision systems, while recycling frozen pretrained language models and visual encoders can avoid expensive retraining.

D.2.2 Risks and mitigation strategies … F Datasheets

Flamingo’s preliminary risk assessment identifies inherited language-model risks, image-conditioned bias and toxicity concerns, while outlining mitigation and explainability directions. Its model card restricts use to research and documents the model, evaluation scope, limitations, and datasheet references.

  • D.2.2 Risks and mitigation strategies: Flamingo inherits large-language-model risks, including offensive language, social biases, stereotypes, and private-information leakage, especially when no images are provided.The risk assessment is preliminary, and further research is needed before deployment.
  • D.2.2 Risks and mitigation strategies: Across gender and skin-tone comparisons, Flamingo’s per-sample CIDEr scores showed no statistically significant differences, with the lowest p-value equal to 0.25.The comparisons used a two-tailed t-test with unequal variance on COCO captioning samples.
  • D.2.2 Risks and mitigation strategies: Some image-conditioned captions were labeled potentially toxic, although manual inspection found no clear toxicity; further study is needed, particularly for unsafe images or toxic text.The evaluation used the Perspective API on captions generated from COCO test images.
  • D.2.2 Risks and mitigation strategies: Flamingo’s few-shot adaptability could support mitigation through conditioning or fine-tuning, red-team test-case generation, visual or external quote references, and output explanation.Qualitative examples suggest that Flamingo can explain its own outputs through its text interface.
  • E Flamingo Model Card: A model card for Flamingo is presented using the framework of Mitchell et al.The card is provided in Table 13.
  • Model Details: Flamingo is a transformer-based autoregressive language model conditioned on visual features through a convnet encoder and transformer-based cross-attention layers.The cross-attention layers incorporate vision features into the language model’s text predictions.
  • Model Details: The model is intended primarily for visual-language-model research, while harmful or deceitful uses and downstream applications without application-specific safety and fairness analysis are out of scope.Relevant factors include the use of English data and the model’s research-only design.
  • Model Details: Flamingo’s model documentation covers evaluation on 18 benchmarks, dataset references, few-shot state-of-the-art performance across 16 tasks, and the limitation that intersectional biases were not investigated.It surpasses fine-tuned state-of-the-art results in 6 cases while using orders of magnitude less task-specific training data, and points to datasheets in Appendix F.1, F.2.1, and F.2.2.

F.1 M3W dataset … G Credit for visual content

The supplementary material documents M3W and related datasets using datasheets, records dataset-use and split practices, notes content-filtering limitations, and credits visual sources and licenses. LTIP and VTP datasheets likewise follow the Gebru et al. framework and use random training/development splits.

  • F.1 M3W dataset: M3W’s datasheet follows the framework defined by Gebru et al. and is presented in Table 14.The dataset is used for pre-training multimodal language and vision models.
  • F.1 M3W dataset: M3W uses random splits for its training and development sets.The dataset is used for pre-training multimodal language and vision models.
  • F.1 M3W dataset: M3W may contain offensive, insulting, or threatening web content, while explicit content is filtered using a dedicated filter.The authors do not otherwise try to filter out such content.
  • F.2.1 Datasheet for LTIP: LTIP uses random splits for its training and development sets, and its datasheet follows the Gebru et al. framework.This information appears in Table 15 and its accompanying description.
  • F.2 Image and video text pair datasets: The supplementary datasheets cover LTIP and VTP as image and video text pair datasets.Both datasheets state that the framework presented by Gebru et al. is followed.
  • F.2.2 Datasheet for VTP: VTP uses random splits for its training and development sets, and its datasheet follows the Gebru et al. framework.This information appears in Table 16 and its accompanying description.
  • G Credit for visual content: Visual content is credited to Unsplash, Wikimedia Commons, DALL·E 2, YFCC100M, Flaticon, Sketchfab, CLIP, and the COCO dataset under specified licenses or source terms.Credits include Unsplash-licensed images, public-domain images, Wikimedia Commons material under CC BY-ND 2.0, and visuals available from DALL·E 2 or CLIP.
Loading 2204.14198v2…