Source-linked AI summary
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, Ludwig Schmidt
TL;DR
Closed-source autoregressive vision-language models limit research on interleaved image-text inputs and web-scraped training data. OpenFlamingo reproduces Flamingo’s approach with open-source models trained using web-scraped image-text sequences, reaching 85% and 89% of corresponding Flamingo performance for its 3B and 9B models across seven evaluation datasets.
Problem
Closed-source autoregressive vision-language models restrict academic study of their models, data, code, hyperparameters, performance, and safety, while existing open-source alternatives generally accept only single images.
Method
OpenFlamingo augments pretrained frozen language models to cross-attend to outputs from a frozen vision encoder while predicting tokens, training the cross-modal module on web-scraped image-text sequences.
Results
85% and 89% of corresponding Flamingo performance: OpenFlamingo-3B and OpenFlamingo-9B, respectively, averaged across 7 evaluation datasets.
Takeaways & Limitations
OpenFlamingo provides an open-source family of autoregressive vision-language models intended to enable more researchers to train and study such models.
Takeaways & Limitations
Because OpenFlamingo models train on web-scraped data without safety-focused fine-tuning, they may produce unexpected, inappropriate, or inaccurate outputs.
Abstract
from arXiv · showhide
We introduce OpenFlamingo, a family of autoregressive vision-language models ranging from 3B to 9B parameters. OpenFlamingo is an ongoing effort to produce an open-source replication of DeepMind's Flamingo models. On seven vision-language datasets, OpenFlamingo models average between 80 - 89% of corresponding Flamingo performance. This technical report describes our models, training data, hyperparameters, and evaluation suite. We share our models and code at https://github.com/mlfoundations/open_flamingo.
1 Introduction
OpenFlamingo is an open-source reproduction of Flamingo that supports interleaved image-text inputs and in-context learning. Its five models span 3B to 9B parameters and reach 85% and 89% of corresponding Flamingo performance at the 3B and 9B scales.
- Single-image vision-language formats support tasks such as image classification and visual question answering, but limit input flexibility.
- Interleaved image-text sequences enable few-shot in-context learning and multi-round multimodal chatbot interactions.
- Closed-source autoregressive vision-language models restrict research because their weights, data, code, and hyperparameters are proprietary.
- OpenFlamingo augments frozen language models with cross-attention to frozen vision-encoder outputs while predicting the next token.
- Five OpenFlamingo models span 3B to 9B parameters and use standard or instruction-tuned language-model backbones.
2 Related work
Related work contrasts single-image open-source models with autoregressive models that process interleaved image-text sequences and support in-context learning. OpenFlamingo follows Flamingo’s design and uses publicly available web-scraped datasets to pursue an open replication.
- BLIP-2 and LLaVA generally incorporate only one image, whereas autoregressive vision-language models accept interleaved image-text sequences.
- Table 1 defines the architecture comparison through language models, model density, a CLIP ViT-L/14 vision encoder, and cross-attention intervals.
- Flamingo was chosen for replication because its aggregated performance improves steadily with up to 32 in-context examples.
- Kosmos-1 shows captioning improvements up to 4 in-context examples but degrades with 8 on COCO and Flickr-30K.
- Flamingo relies on unavailable ALIGN and M3W datasets, while OpenFlamingo uses LAION-2B and Multimodal C4.
3 Approach
OpenFlamingo matches the Flamingo architecture by combining frozen vision and language components with trainable cross-modal modules, and trains on open image-text data. Its approach covers dataset construction, next-token training, distributed optimization, and in-context evaluation across seven vision-language datasets.
- 3.1 Architecture: OpenFlamingo predicts each next text token from previous text and the last preceding image using dense cross-attention in a frozen autoregressive language model.Image patch features come from a frozen vision encoder and pass through a trainable Perceiver resampler.
- 3.2 Training data: The models combine open-source LAION-2B image-text pairs with Multimodal C4 interleaved image-text sequences, plus ChatGPT-generated sequences for OpenFlamingo-4B.LAION-2B provides 2B image-text pairs, while MMC4 provides 101M interleaved samples; synthetic data contains 417K sequences.
- 3.3 Training details: Training uses 60M interleaved examples and 120M LAION-2B examples with next-token prediction, AdamW, a 1e-4 learning rate, and Flamingo-default loss weights.The loss weights are 1 for MMC4 and 0.2 for LAION-2B.
- 3.3 Training details: All models use 64 GPUs across eight nodes, with FSDP for OpenFlamingo-4B and data parallelism for the other models.The training configurations use either FullyShardedDataParallel or DistributedDataParallel.
- 3.3 Training details: MMC4 loss decreases very slowly after an initial improvement, possibly because long text passages allow token prediction without referencing images.The authors speculate that the loss may instead be dominated by whether the frozen language model fits unrelated paragraphs.
- 3.4 Evaluation method: Evaluation spans seven datasets covering captioning, VQA, and HatefulMemes classification at 0, 4, 8, 16, and 32 in-context examples.Captioning uses CIDEr, VQA uses exact-match accuracy, and HatefulMemes uses AUC ROC; evaluations average three seeds.
4 Results
OpenFlamingo models approach Flamingo performance across seven vision-language datasets, though results vary by model size, instruction tuning, task, and number of in-context examples. Performance generally improves with more examples for 3B and 9B models, but the gap to Flamingo can widen and 4B models may degrade.
- Overall comparison: 51.8% → 52.7% VQA accuracy and 79.4 → 79.5 CIDEr are achieved by OpenFlamingo-9B versus Flamingo-9B in 0-shot VQAv2 and COCO, respectively.OpenFlamingo-9B also approaches Flamingo-9B’s 0-shot performance on Flickr-30K and VizWiz, and its 4-shot performance on COCO, VQAv2, and VizWiz.
- Task variation: 6.9 percentage points on OK-VQA and 7.8 percentage points on TextVQA separate OpenFlamingo-9B from Flamingo-9B in 0-shot evaluations.OpenFlamingo-3B also trails Flamingo-3B by 4.6 percentage points in 0-shot VQAv2 accuracy.
- In-context examples: OpenFlamingo-3B and -9B generally improve with more in-context examples, but their improvement rate is lower than Flamingo’s and OpenFlamingo-4B degrades after 4 or 8 shots.The authors speculate that widening gaps may stem from pre-training sequences containing few images.
- Model size: OpenFlamingo-9B generally outperforms smaller models, while OpenFlamingo-4B often underperforms 3B models on Flickr-30K, HatefulMemes, TextVQA, and VizWiz.The 9B exception is HatefulMemes and large numbers of examples on Flickr-30K and TextVQA.
- Instruction tuning: Instruction-tuned MPT-1B and RedPajama-3B variants generally outperform their corresponding base models, with the difference starkest for RedPajama-3B.The observation comes from Figure 6 across evaluation settings.
5 Discussion
Discussion examines why performance differs across training choices and tasks. It identifies frozen special-token embeddings, language-model backbones, VQA-specific errors, and safety concerns as important considerations.
- Embedding choices: 4.6 CIDEr and 12.1% accuracy are lost in 0-shot COCO and VQAv2, respectively, when <image> and <|endofchunk|> embeddings are frozen and randomly initialized.The experiment uses OPT-125M-based models trained on 20M interleaved samples.
- Training dynamics: COCO validation performance steadily improves throughout training, whereas VQAv2 progress is flatter.Figure 8 reports these validation trends for OpenFlamingo-9B.
- Language model: Nearly 10 percentage points are added in 0-shot VQAv2 validation performance by switching from OPT-1.3B to MPT-1B (Instruct).The report hypothesizes that language-model choice may similarly affect other VQA tasks.
- VQA failure modes: 30.5% versus 70.6% accuracy distinguishes numerical-answer from yes/no questions for OpenFlamingo-9B on VQAv2 validation.Other reported failure modes include verbosity and answering about the central rather than queried non-central object.
- Safety: OpenFlamingo models may produce unexpected, inappropriate, or inaccurate outputs because they use web-scraped data and lack safety-focused fine-tuning.The report identifies safety investigation as future work.
6 Conclusion
The conclusion presents OpenFlamingo as a continuing open-source effort spanning 3B, 4B, and 9B autoregressive vision-language models. The authors aim for the models and project to support further research.
- Conclusion: OpenFlamingo comprises five autoregressive vision-language models across the 3B, 4B, and 9B scales.The project remains active, with continued work on training and releasing high-quality models.
- Conclusion: The authors hope OpenFlamingo enables more researchers to train and study autoregressive vision-language models.This is the report’s stated practical aim.
A Extended results
Extended results provide full evaluations across 0, 4, 8, 16, and 32 in-context examples and express OpenFlamingo performance as a fraction of corresponding Flamingo performance.
- Extended results: 0, 4, 8, 16, and 32 in-context examples are reported in the full evaluation results in Table 11.Figure 11 uses the fraction of corresponding Flamingo performance for ease of comparison.
A.1 Comparison to fine-tuned SoTAs
OpenFlamingo is compared with fine-tuned state-of-the-art methods across datasets and in-context-shot settings, while Table 9 separately compares random and retrieval-selected demonstrations.
- The fine-tuned methods were sourced from PapersWithCode on 06/19/23.
- Figure 9 compares each model’s performance relative to fine-tuned state-of-the-art performance, averaged across datasets.
- Table 9 reports OpenFlamingo-9B scores when RICES selects demonstrations versus when demonstrations are sampled randomly.
A.2 Evaluations using RICES
The appendix evaluates retrieval-based in-context example selection across seven datasets and five shot counts, comparing it with random selection. RICES improves many settings but can substantially hurt Flickr-30K.
- RICES selects the top-k most similar training examples using image cosine similarity from a frozen CLIP ViT-L/14 vision encoder.
- 19.2 ROC AUC is the improvement reported for HatefulMemes with 32 shots when using RICES.
- CIDEr decreases by 20.4 in Flickr-30K 0-shot evaluations and by 13.1 in 4-shot evaluations with RICES.
- In 0-shot evaluations, RICES still selects the two text-only examples used for the prompt.
B Additional notes on filtering MMC4
MMC4 filtering removes image-text pairs below a CLIP similarity threshold, but stricter filtering shortens sequences and discards much more data. The authors identify balancing length, quality, and dataset size as future work.
- Images are removed when CLIP ViT-L/14 cosine similarity with matched sentences falls below 0.24, and sequences with no remaining images are discarded.
- At threshold 0.32, 58% of sampled MMC4 sequences contain only one image, compared with 38% at threshold 0.24.
- At threshold 0.32, an estimated 88.7% of MMC4 sequences are discarded completely, compared with 42.7% at threshold 0.24.
- The authors propose balancing sequence length, data quality, and dataset size to improve OpenFlamingo models.
C Synthetic data prompt
This appendix documents the prompt and image-retrieval procedure used to generate synthetic interleaved image-text data, alongside evaluation and image-credit materials. The prompt asks for specific, self-contained image descriptions within generated sequences.
- C Synthetic data prompt: For each generated caption, the pipeline retrieves 10 candidate LAION-5B images using index, aesthetic-score, and aesthetic-weight settings, then selects the highest-CLIP-similarity image.
- D Image credits: Table 9 compares OpenFlamingo-9B outputs from RICES and random demonstrations, documenting an example where RICES patches together incorrect caption details.
- D Image credits: Figures and tables provide image sources and credits for the synthetic examples and evaluation materials.
- D Image credits: Evaluation tables report results across seven datasets and 0, 4, 8, 16, and 32 in-context examples, including random and RICES settings.
- C Synthetic data prompt: ChatGPT is instructed to generate interleaved sequences of image descriptions and sentences, with each image description specific, one-sentence, and self-contained.
- C Synthetic data prompt: The synthetic-data prompt uses a system prompt, five human-authored interleaved sequences, and a query pattern to guide ChatGPT.
- C Synthetic data prompt: The generated sequences describe how images and subsequent sentences form patterns that lead to conclusions.