Source-linked AI summary
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang
TL;DR
OFA addresses the challenge of unifying diverse multimodal and unimodal tasks without task- or modality-specific customization. It uses an instruction-based sequence-to-sequence framework and, with 20M image-text pairs, achieves strong cross-modal and unimodal results while transferring to unseen tasks and domains. Instruction sensitivity remains an author-identified limitation.
Problem
Existing multimodal models struggle to unify diverse tasks and modalities while preserving downstream performance and support for unseen tasks.
Method
OFA uses a Task-Agnostic and Modality-Agnostic encoder-decoder sequence-to-sequence framework with unified instructions, representations, and no task-specific downstream layers.
Results
OFA achieves new state-of-the-art results across vision-language tasks, comparable unimodal performance, and transfer to unseen tasks and out-of-domain information without finetuning.
Takeaways & Limitations
A single instruction-based model can cover broad cross-modal and unimodal understanding and generation tasks within one unified framework.
Takeaways & Limitations
Model performance is highly sensitive to instruction design, with slight prompt or parameter changes potentially causing drastic performance changes.
Abstract
from arXiv · showhide
In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision & language models that rely on extremely large cross-modal datasets, OFA is pretrained on only 20M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a series of cross-modal tasks while attaining highly competitive performances on uni-modal tasks. Our further analysis indicates that OFA can also effectively transfer to unseen tasks and unseen domains. Our code and models are publicly available at https://github.com/OFA-Sys/OFA.
1 Introduction
OFA targets unified multimodal pretraining by representing tasks and modalities through a shared, instruction-based sequence-to-sequence framework. It combines broad task coverage with competitive cross-modal, unimodal, zero-shot, and transfer performance.
- Motivation: OFA addresses the challenge of satisfying task-agnostic, modality-agnostic, and task-comprehensive design goals while maintaining downstream performance.The framework is motivated by the difficulty of unifying task representation, modality representation, and broad task variety without sacrificing results.
- OFA Framework: OFA formulates pretraining and finetuning as instruction-based sequence-to-sequence tasks without adding task- or modality-specific downstream components.A Transformer serves as the shared computation engine and uses a globally shared multimodal vocabulary.
- Task Coverage: OFA unifies vision-language, vision-only, and language-only understanding and generation tasks, including visual grounding, VQA, image captioning, image classification, and language modeling.The pretraining task set includes visual grounding, grounded captioning, image-text matching, image captioning, VQA, object detection, image infilling, and text infilling.
- Results: 20M publicly available image-text pairs support OFA pretraining, while the model achieves state-of-the-art performance across multiple vision-language downstream tasks.Reported tasks include image captioning, visual question answering, visual entailment, and referring expression comprehension.
- Results: OFA remains comparable to state-of-the-art unimodal pretrained models and demonstrates competitive zero-shot learning, unseen-task transfer, and out-of-domain adaptation without finetuning.The unimodal comparisons span natural language understanding, natural language generation, and image classification.
2 Related Work
Related work established separate advances in language, vision, multimodal pretraining, and unified task representations. OFA builds on these directions by pursuing a unified framework across input-output representations, tasks, and modalities.
- Language Pretraining & Vision Pretraining: Language pretraining advanced through improved pretraining tasks and model architectures, while vision research adopted self-supervised learning and generative pretraining.The discussion cites BERT and GPT as representative language-pretraining models and ViT-based generative pretraining as a recent vision advance.
- Multimodal Pretraining: Multimodal pretraining adapted masking strategies and encoder-decoder architectures for generation tasks while using patch projection to simplify preprocessing.These developments aimed to leverage large-scale weakly supervised data and improve multimodal downstream performance.
- Unified Frameworks: Unified-framework research represented diverse tasks uniformly, extended text-to-text and language-modeling ideas to multimodal settings, and explored both uniform byte sequences and task-specific layers.OFA is positioned as a unified Seq2Seq framework for input-output representations, architectures, tasks, and modalities.
3 OFA
OFA unifies multimodal inputs, outputs, tasks, and architectures through a sequence-to-sequence framework with shared representations and no task- or modality-specific downstream components. It jointly pretrains across cross-modal and unimodal tasks, using instruction-based task formulations and a unified vocabulary.
- I/O: OFA represents linguistic and visual inputs and outputs with one vocabulary containing subwords, image codes, and location tokens.Images are converted into patch features, while objects use discrete labels and bounding-box location tokens.
- Architecture: OFA adopts a unified encoder-decoder Transformer architecture for pretraining, finetuning, and zero-shot tasks across modalities.Both encoder and decoder use Transformer layers, and the framework is intended to support architecture compatibility across downstream tasks.
- Tasks & Modalities: All cross-modal and unimodal pretraining, finetuning, and inference tasks are formulated as sequence-to-sequence generation with shared task schemas.This enables joint pretraining on multimodal and unimodal data within one paradigm.
- Tasks & Modalities: OFA uses five cross-modal representation-learning tasks, including visual grounding, grounded captioning, image-text matching, image captioning, and visual question answering.For example, visual grounding generates location tokens from an image, text, and instruction describing the target region.
- Tasks & Modalities: OFA adds image infilling and object detection for vision representation learning while jointly training across natural language, vision, and cross-modal data.The pretraining corpus combines image-text pairs, raw and object-labeled images, and plain text, using publicly available datasets with downstream validation and test images excluded.
- Training & Inference: For classification, OFA uses Trie-based constrained search to avoid inefficient full-vocabulary optimization and invalid labels during inference.The training objective is cross-entropy over target tokens, while decoding can use beam search or the Trie-based strategy.
4 Experiments
Experiments show that OFA performs strongly across cross-modal and unimodal tasks, while instruction sensitivity and task interactions remain important constraints. Ablations further identify region-prediction tasks as valuable for multimodal understanding.
- Cross-modal tasks: 82.0 VQA test-std and 91.2 SNLI-VE test scores establish OFA’s best performance on both cross-modal understanding tasks.Scaling up OFA also brings significant improvements.
- Cross-modal tasks: 154.9 CIDEr sets a new MSCOCO image-captioning state of the art, with OFA outperforming SimVLMHuge by around 2 CIDEr points under Cross-Entropy optimization.All three OFA sizes outperform the huge-size LEMON under CIDEr optimization.
- Cross-modal tasks: OFA reaches state-of-the-art visual grounding on RefCOCO, RefCOCO+, and RefCOCOg, gaining 3.61, 6.65, and 4.85 points over UNICORN on specified test splits.The gains are reported on RefCOCO testA, RefCOCO+ testA, and RefCOCOg test-u, respectively.
- Cross-modal tasks: OFA outperforms text-to-image baselines on all reported metrics, including FID and IS, despite using sampling size 24 versus 512, 60, and 60 for DALLE, CogView, and NÜWA.The authors attribute this to better learned correspondence among query text, images, and image codes.
- Unimodal tasks: OFA attains competitive unimodal results: comparable natural-language understanding performance, new state-of-the-art Gigaword generation, and image-classification accuracy above EfficientNet-B7 and ViT-L for OFALarge.It outperforms similarly sized contrastive models and matches masked-image-modeling models on image classification.
- Transfer and limitations: Instruction-guided pretraining supports zero-shot and unseen-task transfer, but performance is highly sensitive to instruction templates and model parameters.Both models remain below 60% accuracy on sentence-pair classification in the reported GLUE evaluation.
- Multitask ablations: Image infilling improves image classification by +1.0 Acc. and text-to-image generation by +0.6 CLIPSIM, but reduces image captioning by −0.7 CIDEr and VQA by −0.3 Acc.Text infilling improves image captioning by +0.8 CIDEr and VQA by +0.46 Acc., while language pretraining reduces ImageNet-1K by −1.0 Acc.
- Multitask ablations: Region-prediction tasks improve image captioning by +2.3 CIDEr and +1.4 CIDEr and VQA by +0.6 Acc. and +0.5 Acc., indicating finer-grained vision-language alignment.Detection also improves visual understanding by +0.8 Acc., while region information contributes little to text-to-image generation.
5 Conclusion
OFA unifies architecture, tasks, and modalities for multimodal and unimodal understanding and generation without additional task-specific layers. Experiments report new SOTAs across several tasks, competitive unimodal performance, and transfer across zero-shot, domain, and task settings.
- OFA unifies architecture, tasks, and modalities for multimodal and unimodal understanding and generation without additional layers or task specifications.
- OFA achieves new SOTAs in image captioning, VQA, visual entailment, and referring expression comprehension.
- OFA demonstrates comparable performance with language and vision pretrained models on GLUE, abstractive summarization, and image classification.
- The authors identify building an omni-model that generalizes to complex real-world tasks as future work.
A.1 Pretraining Datasets
The pretraining datasets combine vision-language, vision-only, and language-only data, with public availability and filtering designed to avoid downstream data leakage.
- Pretraining combines image-text pairs, raw and object-labeled images, and plain-text passages across vision-language, vision, and language data.
- The datasets are publicly available and exclude images appearing in downstream validation and test sets to avoid data leakage.
- Vision-language data includes image-caption, image-QA, and image-region pairs from CC12M, CC3M, SBU, COCO, and VG Captions.
- Language pretraining uses around 140GB from Pile, with natural-language preprocessing and truncation to length 512.
A.2 Pretraining Details
OFA preprocessing uses fixed image patches and shared text processing, while initialization, optimization, and mixed-task batching define the pretraining setup.
- Images are resized to 256 × 256, 384 × 384, or 480 × 480 by model size, using a fixed 16 × 16 patch size.
- Each patch is represented with the first three ResNet blocks, jointly trained with the Transformer module.
- Text uses BART’s BPE tokenizer with maximum encoder and decoder lengths of 256, and shared embedding and decoder-softmax parameters.
- Transformer initialization uses most BARTBase or BARTLarge weights, or separately pretrained BART-style language-model weights for other OFA sizes.
- Training uses AdamW, a 2e-4 peak learning rate with linear decay and 0.01 warmup ratio, dropout 0.1, weight decay 0.01, and stochastic depth 0.1.
A.3 Details of Downstream Tasks
Downstream evaluation applies instruction-based transfer across language, vision-language, vision, and image-generation tasks, with task-specific datasets, decoding constraints, and finetuning settings.
- Downstream tasks use task-specific instructions to transfer pretrained knowledge, with 480 × 480 inputs by default when finetuning.
- Image captioning on MSCOCO evaluates BLEU-4, METEOR, CIDEr, and SPICE on the Karpathy test split.
- VQA adapts generative OFA to the benchmark with Trie-based search rather than a candidate-answer classification output layer.
- Visual entailment converts entailment, neutral, and contradiction labels to yes, maybe, and no for decoder generation.
- Referring expression comprehension directly predicts the best matching bounding box without proposals on RefCOCO, RefCOCO+, and RefCOCOg.
- Image generation evaluates 30,000 randomly sampled validation images with FID, IS, and CLIPSIM, while image classification constrains outputs to 1,000 ImageNet-1K labels.
- Natural language understanding covers six GLUE tasks, while Gigaword generation reports ROUGE-1, ROUGE-2, and ROUGE-L.
B Trie-based Search
OFA uses Trie-based search to constrain token generation during classification, limiting predictions to valid candidate-label continuations. Removing this search degrades downstream-task performance.
- Trie-based search constrains classification generation by permitting only tokens consistent with candidate labels.The Trie annotates nodes with candidate-label tokens and masks invalid-token logits to −∞ during training and inference.
- During target-token scoring, OFA considers only valid next tokens, such as “sky” and “ocean” in the illustrated Trie.
- Removing Trie-based search degrades performance on downstream tasks.The OFABase baseline in this ablation was pretrained for 250k steps.
C Qualitative Examples
The qualitative examples cover text-to-image generation, VQA, grounded question answering, and visual grounding, including transfer to unseen tasks and domains. They illustrate OFA outputs across photographic, anime, synthetic, and other unseen settings.
- The qualitative section presents OFA examples for text-to-image generation, open-domain VQA, grounded question answering, and open-domain visual grounding.
- OFA generates text-to-image samples, including examples shown after additional finetuning on a subset of LAION-400M.
- Pretrained OFA answers VQA questions on unseen non-photographic domains without finetuning.The VQA pretraining datasets contain only real-world photographs.
- Pretrained OFA transfers directly to grounded question answering, an unseen task requiring answers about particular image regions.
- Visual grounding examples span unseen anime domains, including Pokemon and One Piece, and synthetic images with attribute combinations.