OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

1 Introduction

• The common (image, text) →text format supports tasks such as image classification and visual question answering (VQA).BLIP-2 [22] is given as an example.Source: §1 · p. 1
• Interleaved sequences of images and text support few-shot in-context learning [3] and multi-round multimodal chatbot interactions.Unlike a single-image input, the sequence can include demonstrations for a new task.Source: §1 · p. 1
• Across 7 evaluation datasets, OpenFlamingo-3B reaches 85% and OpenFlamingo-9B reaches 89% of their corresponding Flamingo models’ performance.Models and code are open-sourced.Source: Figure 1 · p. 1, §1 · p. 2
• Autoregressive vision-language models have shown generalization across diverse tasks, including Flamingo [3], CM3 [1], Kosmos-1 [12], PALM-E [8], and multimodal GPT-4 [28].Evaluations suggest these models can serve as performant foundation models [5].Source: §1 · p. 1
• Closed-source autoregressive vision-language models limit research on web-scraped data’s effects on performance and safety; cited open alternatives accept only single images and often train on curated datasets such as COCO [24].The open alternatives listed include LLaVA [25], LLaMA-Adapter [41], BLIP-2 [23], and mPLUG-Owl [39].Source: §1 · p. 2
• OpenFlamingo reproduces Flamingo [3] by augmenting frozen pretrained language-model layers to cross-attend to a frozen vision encoder while predicting the next token.The cross-modal module is trained on web-scraped image-text sequences from LAION-2B [32] and Multimodal C4 [45], using publicly available components including CLIP [30] and open-source language models.Source: §1 · p. 2
• The five OpenFlamingo models range from 3B to 9B parameters and use standard or instruction-tuned [37] language-model backbones.Models and code are open-sourced.Source: §1 · p. 2

Related work

• Autoregressive vision-language models accept interleaved image-text sequences for in-context learning, unlike one-image-context models such as BLIP-2 and LLaVa [6] [16] [22] [25] [39] [41].Generative vision-language models output text conditioned on an image-text sequence.Source: §2 · pp. 2–3
• 32 in-context examples mark Flamingo’s reported performance gains, whereas Kosmos-1 improves up to 4 examples and degrades at 8 on COCO and Flickr-30K captioning [3] [12] [24] [29].These Flamingo gains motivated OpenFlamingo’s replication.Source: §2 · p. 3

Open-source image-text datasets

• Proprietary autoregressive models use closed datasets; Flamingo relies on unavailable ALIGN image-text pairs [14] and M3W interleaved sequences [3], while OpenFlamingo trains on LAION-2B and Multimodal C4 [45].Recent replication efforts include LAION-2B, Multimodal C4, and OBELISC [18].Source: §2 · p. 3
• Laurençon et al. [18] also train 9B and 80B Flamingo-style models, using OBELISC rather than Multimodal C4 and LLaMA-9B [41] rather than MPT and RedPajama.Their models differ from OpenFlamingo in pretraining dataset and language model.Source: §2 · p. 3

3.1 Architecture

• OpenFlamingo matches Flamingo’s architecture [3], using image-conditioned text prediction with a frozen language model and vision encoder, dense cross-attention, and a trainable Perceiver resampler.It does not support video inputs.Source: §3.1 · p. 3

3.2 Training data

• Training replaces ALIGN [14] and M3W [3] with open LAION-2B [32] and Multimodal C4 [45], and also tests synthetic sequences for OpenFlamingo-4B.LAION-2B provides 2B image-text pairs; MMC4 provides 101M interleaved samples, CLIP-aligns images to sentences, filters low-similarity images, and probabilistically rejects some single-image sequences. The synthetic set contains 417K sequences with ChatGPT-generated text and retrieved LAION-5B images.Source: §3.2 · pp. 4–5

3.3 Training details

• Models train for 60M interleaved examples and 120M LAION-2B examples with next-token prediction, AdamW, a constant 1e-4 learning rate after warmup, and Flamingo-default loss weights.OpenFlamingo-4B mixes about 240K synthetic sequences into its 60M interleaved examples; other models use MMC4. MMC4 loss decreases slowly, which the authors speculate may reflect long text passages that can be generated without images.Source: §3.3 · p. 5

3.4 Evaluation method

• Evaluation covers seven datasets for captioning, VQA, and hateful-meme classification, measuring performance with 0, 4, 8, 16, or 32 in-context examples.Evaluations use the splits from Alayrac et al. [3], average scores across three seeds, and uniformly sample demonstrations from training data; retrieval-based selection is also evaluated in the appendix.Source: §3.4 · pp. 5–6
• Evaluation uses task-specific prompts and decoding, with CIDEr for captioning, exact-match accuracy for VQA, and AUC ROC for HatefulMemes.Captioning and VQA use three-beam search with task-specific token limits; HatefulMemes compares the log-likelihoods of “yes” and “no.” Zero-shot evaluations include two image-removed examples, and classification averages logits across six example permutations.Source: §3.4 · p. 6

Overall comparison

• More than 86% of corresponding Flamingo performance is attained on average by OpenFlamingo-3B, -3B (Instruct), -4B (Instruct), and -9B across 0, 4, and 32 shots.The comparison covers seven vision-language datasets.Source: §4 · p. 6, Table 4 · p. 8
• At 0 shots, OpenFlamingo-9B exceeds Flamingo-9B on VQAv2 (51.8% →52.7% VQA accuracy) and COCO (79.4 →79.5 CIDEr).OpenFlamingo-9B also approaches Flamingo-9B at 0 shots on Flickr-30K and VizWiz, and at 4 shots on COCO, VQAv2, and VizWiz. However, it trails Flamingo-9B by 6.9 percentage points on OK-VQA and 7.8 percentage points on TextVQA in 0-shot evaluations; the reason for generally low VQA performance is unclear.Source: §4 · pp. 6–7

Extrapolating to more in-context examples

• OpenFlamingo-3B and -9B generally improve with more in-context examples, but their improvement rate is lower than Flamingo-9B’s, widening the gap at higher shot counts.The authors speculate that the gap may stem from pre-training data containing mostly sequences with few images; OpenFlamingo-4B unexpectedly degrades after 4 or 8 shots and uses RedPajama backbones with frozen <image> and <|endofchunk|> embeddings.Source: §4 · p. 7

Trends by model size

• OpenFlamingo-9B generally outperforms smaller models, except on HatefulMemes and at large shot counts on Flickr-30K and TextVQA; OpenFlamingo-4B often underperforms 3B.The 4B models underperform 3B on Flickr-30K, HatefulMemes, TextVQA, and VizWiz.Source: §4 · pp. 7–8

Effect of language instruction-tuning

• Instruction-tuned variants of MPT-1B and RedPajama-3B outperform their respective base models on average, with the difference starkest for RedPajama-3B.Transfer of language instruction tuning to vision-language tasks had previously been reported in Huang et al. [12] and Li et al. [23].Source: §4 · p. 8

Comparison to fine-tuned state-of-the-art

• More than 62% of fine-tuned state-of-the-art performance is achieved by OpenFlamingo-9B with 32 RICES-selected examples, compared with 72% for Flamingo-9B.The comparison uses fine-tuned state-of-the-art results listed on Papers With Code on June 19, 2023.Source: §4 · p. 8, Figure 7 · p. 8

5.1 Frozen embeddings

• Freezing the <image> and <|endofchunk|> embeddings reduced 0-shot COCO by 4.6 CIDEr and 0-shot VQAv2 by 12.1% accuracy in small OPT-125M models.The models were trained on 20M interleaved samples, one-third of full training; the authors suggest frozen embeddings may affect downstream trends.Source: §5.1 · p. 9

5.2 VQAv2 validation trends

• OpenFlamingo-9B validation performance steadily improved on COCO during training, while VQAv2 progress was flatter.The authors note this matches trends reported by Li et al. [23].Source: §5.2 · p. 9, Figure 8 · p. 9
• Switching from OPT-1.3B to MPT-1B (Instruct) added nearly 10 percentage points in 0-shot VQAv2 performance, despite limited gains from additional training.The authors hypothesize language-model choice may similarly affect other VQA tasks.Source: §5.2 · p. 9, Table 7 · p. 10
• OpenFlamingo-9B scored 30.5% on VQAv2 questions with numerical answers versus 70.6% on yes / no questions, and models also struggled with verbosity and non-central objects.Exact-match VQA scoring favors concise answers; verbose outputs may be truncated, and models sometimes answer about the image's central item instead of the object asked about.Source: §5.2 · p. 9, Table 6 · p. 10

5.3 Applications of OpenFlamingo

• OpenFlamingo has supported downstream assistants: Li et al. [20] fine-tuned it on MIMIC-IT [19] to create Otter, and Gong et al. [10] released vision-and-language instruction-tuned Multimodal-GPT.The authors hope the community continues to use OpenFlamingo models.Source: §5.3 · pp. 9–10

5.4 Limitations

• OpenFlamingo inherits risks from its foundational language models: web-scraped training data and no safety-focused fine-tuning may lead to unexpected, inappropriate, or inaccurate outputs.The authors hope to further investigate safety properties of autoregressive vision-language models.Source: §5.4 · p. 10

6 Conclusion

• OpenFlamingo comprises five autoregressive vision-language models across the 3B, 4B, and 9B scales.Source: §6 · p. 10
• The project remains active, with ongoing work to train and release high-quality autoregressive vision-language models.Source: §6 · p. 10
• The authors hope OpenFlamingo enables more researchers to train and study autoregressive vision-language models.Source: §6 · p. 10

• Table 11 reports full evaluation results for 0, 4, 8, 16, and 32 in-context examples, with OpenFlamingo performance expressed as a fraction of corresponding Flamingo performance in Figure 11.Source: A Extended results · p. 14

A.1 Comparison to fine-tuned SoTAs

• Figure 9 compares OpenFlamingo models with fine-tuned state-of-the-art performance, averaged across datasets and evaluated with different numbers of in-context examples.The fine-tuned methods were drawn from PapersWithCode on 06/19/23 (Table 8).Source: Figure 9 · p. 14, §A.1 · p. 14
• RICES [38] often outperforms random demonstrations on OpenFlamingo-9B.Table 9 reports scores for OpenFlamingo-9B.Source: Table 9 · p. 14

A.2 Evaluations using RICES

• RICES selects the top-k most similar training examples as demonstrations using image cosine similarity from the frozen CLIP ViT-L/14 vision encoder.Source: §A.2 · p. 14
• Against random demonstrations, RICES improves most settings, including by 19.2 ROC AUC on HatefulMemes at 32 shots.Source: §A.2 · p. 14
• On Flickr-30K, RICES lowers CIDEr by 20.4 at 0 shots and 13.1 at 4 shots, possibly because the model parrots similar demonstrations.The paper hypothesizes that RICES retrieves demonstrations so similar to test examples that OpenFlamingo-9B repeats their captions, including incorrect details.Source: §A.2 · p. 14

B Additional notes on filtering MMC4

• Using CLIP ViT-L/14 embeddings, the filter removes images below a 0.24 cosine-similarity threshold while retaining text, and discards sequences with no remaining images.This aims to keep images relevant to the text following them, using a filtering approach similar to image-text pair filtering [31] [9].Source: B Additional notes on filtering MMC4 · p. 15
• At threshold 0.32, nearly 58% of sampled MMC4 sequences contain one image, versus 38% at 0.24 in Figure 4.Long interleaved sequences may matter for handling many in-context examples.Source: B Additional notes on filtering MMC4 · p. 15
• At threshold 0.32, an estimated 88.7% of MMC4 sequences are discarded completely, compared with 42.7% at threshold 0.24.Source: B Additional notes on filtering MMC4 · p. 15
• Future work will investigate balancing sequence length, data quality, and dataset size to improve OpenFlamingo models.Source: B Additional notes on filtering MMC4 · p. 15

C Synthetic data prompt

• The prompt used to generate ChatGPT-generated data is provided.The surrounding text identifies this as data described in §3.2.Source: C Synthetic data prompt · p. 15
• 10 candidate images are sought from LAION-5B for each unique caption, using index=laion5B-L-14, aesthetic score=9, and aesthetic weight=0.5.The retrieved images are candidates for interleaving.Source: C Synthetic data prompt · p. 15
• Retrieved images are reranked with CLIP ViT-L/16@336px, and the highest-similarity image is selected for interleaving.Source: C Synthetic data prompt · p. 15

D Image credits

• Links to the images used in Figure 2 are included, with Table 13 listing their sources in left-to-right presentation order for each row.Source: D Image credits · p. 15, Table 13 · p. 19

C Synthetic data prompt

• 5 interleaved human-authored sequences, a system prompt, and a query pattern guide ChatGPT’s synthetic sequence generation.The illustrated query pattern alternates image placeholders with sentences.Source: Figure 12 · p. 20
• ChatGPT first explains the kind of sequence it is attempting to generate, such as nature images followed by conclusion-leading statements.The figure gives this as an example of ChatGPT’s explanation.Source: Figure 12 · p. 20
• Generated “Photo X” descriptions are used to retrieve images that form the interleaved sequences.The figure describes this step after ChatGPT generates its sequence explanation.Source: Figure 12 · p. 20