Source-linked AI summary
Kosmos-2: Grounding Multimodal Large Language Models to the World
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei
TL;DR
Multimodal large language models lacked direct, spatially grounded interaction with image regions and grounding in generated text. Kosmos-2 adds this capability using grounded image-text data and evaluates it across grounding, referring, perception-language, and language tasks, achieving strong performance while enabling downstream applications such as grounded captioning and visual question answering.
Problem
Multimodal large language models lacked direct, unambiguous ways to refer to image regions and connect generated text to the visual world.
Method
Kosmos-2 augments KOSMOS-1's multimodal training data with GRIT, linking caption text spans to image regions through bounding-box location tokens.
Results
Kosmos-2 achieves impressive performance across grounding, referring, language, and vision-language tasks; on RefCOCOg referring expression generation, it outperforms finetuned SLR by 1.1 CIDEr scores.
Takeaways & Limitations
Grounding becomes a foundation capability that supports visual answers, direct image-region interaction, and downstream tasks including grounded image captioning and grounded visual question answering.
Takeaways & Limitations
The model is intended for academic and research purposes, and its use to create unsuitable material is strictly forbidden and not endorsed by the work.
Abstract
from arXiv · showhide
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.
1 Introduction
Kosmos-2 adds grounding to multimodal language models by linking text spans with image regions and producing visual answers. It retains competitive multimodal performance while enabling grounding, referring, and downstream grounded applications.
- Motivation and contribution: Kosmos-2 introduces grounding capability so users can directly indicate image regions and receive bounding-box answers linked to generated text.This supports referring expression comprehension and reduces ambiguity compared with text-only responses.
- Method: The model is a Transformer-based causal language model trained with next-word prediction on multimodal corpora augmented by web-scale grounded image-text pairs.The added pairs associate noun phrases and referring expressions with corresponding image regions through location tokens.
- Method: Kosmos-2 represents grounded text spans as hyperlink-like links to bounding-box location tokens, connecting captions with visual objects or regions.This format encodes spatial coordinates after their associated text spans.
- Results: Experimental results show competitive performance on previously evaluated language and vision-language tasks, alongside impressive grounding and referring-task performance.The evaluated grounding tasks include phrase grounding and referring expression comprehension, while referring tasks include referring expression generation.
- Results: Grounding capability enables downstream applications including grounded image captioning and grounded visual question answering.
2 Construction of Web-Scale Grounded Image-Text Pairs (GRIT)
GRIT is constructed from large-scale image-text data by extracting caption phrases, associating them with image regions, and expanding them into referring expressions. The resulting dataset contains approximately 91M images, 115M text spans, and 137M associated bounding boxes.
- Dataset construction: GRIT2 is built from subsets of COYO-700M and LAION-2B using a pipeline that links caption text spans to corresponding image regions.The pipeline has two main stages: generating noun-chunk–bounding-box pairs and producing referring-expression–bounding-box pairs.
- Noun-chunk grounding: The first stage extracts noun chunks with spaCy, removes abstract phrases that are difficult to recognize visually, and associates the remaining phrases with image regions using a pretrained detector.The example extracts “a dog,” “a field,” and “flowers” from a caption before image-region association.
- Referring-expression construction: The second stage expands noun chunks into referring expressions by recursively traversing dependency-tree children and concatenating their tokens.Conjunct expansions are excluded, and expressions without children remain for subsequent processing.
- Referring-expression construction: The pipeline retains non-contained expressions and assigns the noun chunk’s bounding box to the corresponding expanded referring expression.For example, it keeps “a dog in a field of flowers” while dropping contained expressions such as “a field of flowers” and “flowers.”
- Dataset scale: Approximately 91M images, 115M text spans, and 137M associated bounding boxes are obtained in GRIT.
3 KOSMOS-2: A Grounded Multimodal Large Language Model
Kosmos-2 integrates grounded text and image-region representations into a KOSMOS-1-based multimodal language model. It maps bounding boxes to location tokens and trains the model to associate text spans with regions, enabling visual answers and grounded outputs.
- Grounded representations: Kosmos-2 discretizes bounding-box coordinates into location tokens and links those tokens with their corresponding noun phrases or referring expressions.The representation uses discrete tokens for spatial coordinates so text and location information can be encoded together.
- Grounded representations: A bounding box is represented by location tokens for its top-left and bottom-right points, enclosed by special box tokens.Multiple bounding boxes are concatenated with a delimiter token.
- Grounded representations: The model arranges each text span and its location tokens in a Markdown-like hyperlink format that associates the span with the image region inside the box.
- Grounded input representations: The input format includes image embeddings, a grounding token, text spans, and location tokens, with a vision encoder and resampler producing image embeddings.The grounding token signals that text output should be grounded to the visual world.
- Model training: Training adds grounded image-text pairs to KOSMOS-1’s multimodal corpora, allowing the model to locate regions, associate them with text, and output bounding boxes.The training loss considers discrete text and location tokens.
- Capabilities: Kosmos-2 provides bounding-box input and output capabilities and supports downstream tasks including grounded captioning, grounded VQA, and referring expression comprehension and generation.
4 Evaluation
KOSMOS-2 is evaluated across grounding, referring, perception-language, and language tasks. The evaluation tests whether it can generate visual locations from text and retain prior capabilities.
- The evaluation covers phrase grounding, referring expression comprehension, referring expression generation, perception-language tasks, and language understanding and generation.
- Multimodal Grounding: KOSMOS-2 generates location tokens that are converted into bounding boxes for phrase grounding and referring expression comprehension.The grounding prompt uses a special token to elicit location tokens.
- Phrase Grounding: 7.4% R@1 separates KOSMOS-2 from the finetuned VisualBert model on both Flickr30k Entities validation and test splits.KOSMOS-2 also outperforms GRILL by a large margin in zero-shot phrase grounding.
- Phrase Grounding: KOSMOS-2 produces similar R@1, R@5, and R@10 phrase-grounding results without object queries or proposals.The authors interpret this as generating high-quality locations without post-processing redundant locations.
- Referring Expression Comprehension: KOSMOS-2 achieves promising zero-shot referring comprehension and significantly outperforms previous zero-shot models on RefCOCOg.Performance is slightly lower than previous finetuned works on RefCOCO and RefCOCO+; the authors attribute this to shorter expressions in those datasets.
4.2 Multimodal Referring
KOSMOS-2 extends multimodal referring by accepting bounding boxes and generating descriptions, while also being evaluated on captioning, visual question answering, and language tasks. It remains competitive on prior task families and improves referring generation with few-shot demonstrations.
- Multimodal Referring: KOSMOS-2 accepts user-selected bounding boxes as input and can generate unambiguous descriptions for specified objects or regions.The referring expression generation evaluation uses RefCOCOg under zero-shot and few-shot settings.
- Referring Expression Generation: The referring generation prompt combines a text fragment with bounding-box location tokens to elicit an object description.Results are reported with METEOR and CIDEr, using greedy decoding at 224×224 image resolution.
- Referring Expression Generation: 1.1 CIDEr points above finetuned SLR are achieved by KOSMOS-2 in zero-shot referring expression generation on RefCOCOg.Few-shot demonstrations produce further improvements.
- Perception-Language Tasks: KOSMOS-2 shows comparable overall performance to KOSMOS-1 on zero-shot Flickr30k captioning and VQAv2 visual question answering.It improves slightly on Flickr30k while decreasing marginally on VQA.
- Language Tasks: KOSMOS-2 has similar performance to KOSMOS-1 on most evaluated language tasks, with decreases on CB and improvements on BoolQ and COPA.The evaluated set includes eight zero-shot language tasks spanning completion, Winograd-style, commonsense, and SuperGLUE benchmarks.
5 Conclusion
KOSMOS-2 grounds language in the visual world by associating text expressions with image objects or regions. It supports grounding, referring, language, and vision-language tasks.
- KOSMOS-2 grounds noun phrases and referring expressions to objects or regions in images using the GRIT dataset.GRIT is created by extracting and associating expressions in captions with corresponding scene locations.
- KOSMOS-2 enables perception of image regions and grounding of generated text in the visual world.
- KOSMOS-2 achieves impressive results on language, vision-language, grounding, and referring tasks.The reported grounding tasks include phrase grounding and referring expression comprehension, while referring tasks include referring expression generation.
Ethics Statement
The model is intended for academic and research purposes, with unsuitable uses forbidden and responsibility for improper applications assigned to users.
- KOSMOS-2 is intended for academic and research purposes.
- The paper forbids and does not endorse using the model to create unsuitable material.
- Individuals who generate improper or unacceptable content are held accountable for its application.
- The model development process incorporates Microsoft AI Principles.
A Hyperparameters
The appendix directs readers to separate tables for KOSMOS-2’s training and instruction-tuning hyperparameters.
- Training hyperparameters for KOSMOS-2 are listed in Table 7.
- Instruction-tuning hyperparameters for KOSMOS-2 are listed in Table 8.
B Templates for Grounded Instruction Data
The appendix provides instruction templates for grounded expression generation based on associated bounding boxes.
- Table 9 presents instruction templates for expression generation during instruction tuning.
- The templates generate expressions based on associated bounding boxes.
C Examples of GRIT
The GRIT corpus contains grounded image-text pairs spanning varied domains and numbers of objects, illustrated through food, architecture, and people-centered scenes.
- GRIT grounded image-text pairs span various domains and contain different numbers of objects.
- One GRIT example depicts kale and roasted vegetable salad served with avocado Caesar dressing.
- Other examples depict a keto chicken nugget being dipped in mustard, solar cells against the Sydney skyline, and a woman with a child in a city landscape.
D More Examples of KOSMOS-2
Additional examples show KOSMOS-2 applying multimodal referring across visual dialogue, object detection, visual question answering, image captioning, and grounded image captioning.
- Visual dialogue examples illustrate multimodal referring as a potential human-AI interaction capability.
- An object-detection example demonstrates fine-grained detection using both text and image descriptions through in-context learning.
- Further examples include grounded visual question answering, grounded image captioning, and multimodal referring.