Source-linked AI summary
Language Is Not All You Need: Aligning Perception with Language Models
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
TL;DR
LLMs lack native multimodal perception, limiting their use with images and other modalities despite the importance of multimodal grounding. The paper introduces KOSMOS-1, trained on web-scale multimodal corpora with a Transformer language model as a unified interface. KOSMOS-1 performs across language, perception-language, and vision tasks, while showing cross-modal transfer and initial zero-shot nonverbal reasoning capability.
Problem
LLMs remain limited in natively using multimodal data, although multimodal perception is important for grounding and broader applications.
Method
KOSMOS-1 is a Transformer-based MLLM trained on text, image-caption pairs, and interleaved image-text corpora with next-token prediction.
Results
KOSMOS-1 supports language, perception-language, and vision tasks, including multimodal dialogue, captioning, visual question answering, OCR, and instruction-based image classification.
Takeaways & Limitations
MLLMs enable zero- and few-shot multimodal learning, multi-turn interaction, cross-modal transfer, and new evaluation of nonverbal reasoning.
Abstract
from arXiv · showhide
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.
1 Introduction: From LLMs to MLLMs
The paper motivates aligning language models with multimodal perception to overcome limitations on non-text inputs and broaden their capabilities. KOSMOS-1 supports multimodal tasks, new reasoning settings, and cross-modal transfer.
- Motivation: Multimodal perception remains difficult for LLMs despite their success as general-purpose interfaces for natural-language tasks.The paper frames multimodal perception as important for knowledge acquisition, grounding, and applications such as robotics and document intelligence.
- KOSMOS-1: KOSMOS-1 aligns perception modules with a Transformer language model and trains from scratch on web-scale multimodal corpora.The corpora include text, image-caption pairs, and arbitrarily interleaved image-text documents.
- Capabilities: KOSMOS-1 natively handles language, perception-language, and vision tasks, including visual dialogue, visual question answering, captioning, OCR, and instruction-based image classification.The reported capabilities span both natural-language and perception-intensive tasks.
- Reasoning and transfer: The authors introduce a Raven IQ benchmark for nonverbal reasoning and report that multimodal models achieve better commonsense reasoning than language-only models.They interpret the comparison as evidence that cross-modal transfer helps knowledge acquisition.
- New capabilities: MLLMs support zero- and few-shot multimodal learning, nonverbal reasoning evaluation, and multi-turn interaction across general modalities.These capabilities are accessed through natural-language instructions and demonstration examples.
2 KOSMOS-1: A Multimodal Large Language Model
KOSMOS-1 uses a Transformer-based causal language model as a unified interface for text and other modalities. Encoded multimodal inputs are processed autoregressively, enabling zero- and few-shot evaluation across language and multimodal tasks.
- Architecture: KOSMOS-1 embeds text and other modalities into a shared sequence processed by a Transformer decoder.The model is trained on monomodal, paired cross-modal, and interleaved multimodal corpora.
- Input Representation: Special tokens mark sequence boundaries and image-embedding spans in interleaved text-image inputs.For example, image embeddings can appear between text segments inside one input sequence.
- Input Representation: A vision encoder produces image representations, while Resampler performs attentive pooling to reduce the number of image embeddings.The encoded representations are then supplied to the language-model decoder.
- Generation: The causal decoder generates each next token from previous context using causal masking and a vocabulary softmax classifier.This preserves the autoregressive generation procedure for multimodal inputs.
- Unified Interface: MLLMs combine language-model instruction following and in-context learning with perception aligned through multimodal training.The framework can handle data types that can be represented as vectors.
- Architecture: The model architecture includes MAGNETO for training stability and XPOS for long-context modeling.These are implementation modifications to the standard Transformer architecture.
- Training Objective: Training uses web-scale text, image-caption, and interleaved image-text data, with next-token prediction maximizing the log-likelihood of discrete tokens.Only discrete tokens, such as text tokens, contribute to the training loss.
3 Model Training
KOSMOS-1 is trained on several web-derived multimodal data sources and then language-only instruction tuning. The training setup combines representation learning, cross-modal data, and instruction-following supervision.
- Multimodal Training Data: The training datasets combine text corpora, image-caption pairs, and interleaved image-text documents.These data types support monomodal representation learning and multimodal modeling.
- Multimodal Training Data: The text data comes from The Pile and Common Crawl after deduplication and filtering against downstream task data.The sources include Common Crawl snapshots, CC-Stories, and RealNews.
- Multimodal Training Data: Image-caption pairs are collected from LAION-2B, LAION-400M, COYO-700M, and Conceptual Captions.Several datasets derive image sources and alt-text from web pages.
- Multimodal Training Data: Interleaved image-text data is extracted from filtered Common Crawl web pages, with each document limited to five images.The collection process selects about 71M web pages from an original 2B-page snapshot.
- Model Configuration: The MLLM component has about 1.3B parameters and uses a pretrained CLIP ViT-L/14 representation for images.The CLIP model is frozen except for its last layer during training.
- Training Setup: KOSMOS-1 is trained for 300k steps with a batch size of 1.2 million tokens, corresponding to about 360 billion tokens.The batch mixes text-corpus, image-caption, and interleaved-data tokens.
- Language-Only Instruction Tuning: Language-only instruction tuning continues training on instruction, input, and output examples, with instructions and inputs excluded from the loss.The instruction data combines Unnatural Instructions and FLANv2.
4 Evaluation
KOSMOS-1 is evaluated across multimodal perception-language, nonverbal reasoning, OCR-free language understanding, web-page comprehension, chain-of-thought, image classification, and language-only tasks. Results show strong zero- and few-shot performance, with gains from in-context descriptions, multimodal reasoning, and language-only instruction tuning.
- Image Captioning: 67.1 CIDEr on Flickr30k exceeds Flamingo-3B’s 60.6 and Flamingo-9B’s 61.5 in zero-shot image captioning.KOSMOS-1 uses 1.6B parameters, smaller than the compared Flamingo models.
- Image Captioning: Few-shot captioning improves as shots increase from two to four and outperforms zero-shot results across COCO and Flickr30k.The reported trends are consistent across both datasets.
- Visual Question Answering: KOSMOS-1 handles VizWiz more accurately and robustly than Flamingo in zero-shot VQA while remaining competitive on VQAv2.In few-shot VQA, it outperforms other models at k = 2 and 4 on VizWiz, and results improve over zero-shot numbers.
- Nonverbal Reasoning: 9.3% improvement over the random baseline is achieved by KOSMOS-1 with language-only instruction tuning on the Raven IQ test.The model perceives abstract patterns and deduces the next element in nonverbal contexts, although it remains below average adult performance.
- OCR-Free Language Understanding: 63.9% ROC AUC on HatefulMemes and 67.1% test accuracy on Rendered SST-2 demonstrate OCR-free language understanding from rendered images.KOSMOS-1 surpasses the cited HatefulMemes AUCs for CLIP ViT-L and Flamingo-9B without external OCR text.
- Multimodal Chain-of-Thought: 72.9 with multimodal chain-of-thought prompting is 5.8 points above standard prompting on Rendered SST-2.The approach generates image content as an intermediate rationale before predicting sentiment.
5 Conclusion
The work introduces KOSMOS-1 as a multimodal large language model with general-modality perception, instruction following, and in-context learning. Trained on web-scale multimodal corpora, it achieves promising results across language and multimodal tasks and supports a unified interface for multimodal learning.
- KOSMOS-1 is a multimodal large language model that perceives general modalities, follows instructions, and performs in-context learning.
- Models trained on web-scale multimodal corpora achieve promising results across a wide range of language and multimodal tasks.
- KOSMOS-1 can serve as a unified interface for multimodal learning, including controlling text-to-image generation with instructions and examples.
- The authors identify scaling model size and integrating speech capability as future directions for KOSMOS-1.
A.1 Training
The training section reports the model and instruction-tuning hyperparameters in dedicated tables.
- Table 17 reports the causal language model hyperparameters of KOSMOS-1.
- Table 18 reports the training hyperparameters of KOSMOS-1.
- Table 19 reports the detailed instruction-tuning hyperparameters.
B Datasets
KOSMOS-1 is trained on large language datasets and image-caption pairs collected from web-scale sources.
- KOSMOS-1 uses The Pile and Common Crawl as language-data sources.
- Table 20 provides a full overview of the language datasets used to train KOSMOS-1.
- The language datasets are organized into Internet and Prose source categories, including web, Wikipedia, books, and stories.
- Image-caption pairs come from English LAION-2B, LAION-400M, COYO-700M, and Conceptual Captions datasets.
B.1.3 Interleaved Data
Interleaved multimodal training data is collected from Common Crawl pages and filtered for English text, images, image quality, and meaningful content. The training data format is documented separately.
- The corpus starts from 2 billion Common Crawl web pages and filters out non-English, image-free, low-resolution, single-colored, and incoherent content.
- The training data format is presented in Table 21.
C Evaluation
The evaluation examines KOSMOS-1 on perception-language tasks in both zero-shot and few-shot settings. Zero-shot evaluation uses language prompts, while few-shot evaluation uses in-context demonstrations.
- Perception-language tasks are evaluated under both zero-shot and few-shot settings.The evaluation setup includes zero-shot image captioning with language prompts and few-shot visual question answering with in-context learning.
C.2 Language Tasks
The language-task evaluation covers four categories spanning completion, Winograd-style reasoning, commonsense reasoning, and selected SuperGLUE tasks.
- Language evaluation includes cloze and completion tasks such as StoryCloze and HellaSwag.
- Winograd-style evaluation uses the Winograd and Winogrande datasets.
- Commonsense reasoning is evaluated with PIQA.
- Three SuperGLUE datasets are included: BoolQ, CB, and COPA.
C.3 WebSRC Task Examples
The WebSRC examples section identifies a question asking what type of drive is shown and points to WebSRC examples as the source.
- The example question asks, “What is the type of this drive?”
- The examples are drawn from WebSRC.