Source-linked AI summary
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
TL;DR
Prior subject-driven generation methods include per-image-set fine-tuning and approaches with limits on joint text-image guidance or multi-entity inputs. KOSMOS-G aligns MLLM outputs with CLIP using language as an anchor, then applies compositional instruction tuning; it extends zero-shot subject-driven generation to multi-entity scenarios and can replace CLIP without modifying the image decoder.
Problem
Prior approaches fine-tune on each reference-image set, separate text and image guidance, or have input-template and training-data limits on multiple entities.
Source: §1 · p. 2Method
KOSMOS-G aligns MLLM output space with CLIP using language as an anchor, then applies instruction tuning on curated data.
Source: §1 · p. 2Results
KOSMOS-G extends zero-shot subject-driven generation to multi-entity scenarios and demonstrates competitive single-entity and text-to-image capability.
Source: §5 · p. 10Takeaways & Limitations
Score distillation instruction tuning permits CLIP substitution without modifying the image decoder, supporting integration with U-Net techniques such as ControlNet and LoRA.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Recent advancements in subject-driven image generation have made significant strides. However, current methods still fall short in diverse application scenarios, as they require test-time tuning and cannot accept interleaved multi-image and text input. These limitations keep them far from the ultimate goal of "image as a foreign language in image generation." This paper presents Kosmos-G, a model that leverages the advanced multimodal perception capabilities of Multimodal Large Language Models (MLLMs) to tackle the aforementioned challenge. Our approach aligns the output space of MLLM with CLIP using the textual modality as an anchor and performs compositional instruction tuning on curated data. Kosmos-G demonstrates an impressive capability of zero-shot subject-driven generation with interleaved multi-image and text input. Notably, the score distillation instruction tuning requires no modifications to the image decoder. This allows for a seamless substitution of CLIP and effortless integration with a myriad of U-Net techniques ranging from fine-grained controls to personalized image decoder variants. We posit Kosmos-G as an initial attempt towards the goal of "image as a foreign language in image generation." The code can be found at https://aka.ms/Kosmos-G
1 Introduction
KOSMOS-G uses an MLLM to support zero-shot subject-driven generation from interleaved image and text input. Its align-before-instruct approach pairs compositional instruction tuning with a frozen image decoder, enabling CLIP substitution and integration with U-Net techniques.
1 Introduction
- Earlier methods either fine-tuned on each reference-image set, separated text and image guidance, or used input templates and data that limited multi-entity scalability.The cited approaches also struggled to generalize or extend to scenarios involving multiple entities.Source: §1 · p. 2
- KOSMOS-G leverages MLLM perception, aligns its output space with CLIP using language as an anchor, then performs instruction tuning on curated data.Its training uses captions with each entity followed by its segmented image, and a frozen diffusion decoder supplies the score metric for distillation.Source: §1 · p. 2
- KOSMOS-G generates images that faithfully reproduce input contents across varied contexts, including interleaved multi-image and text settings.The paper describes it as the first model to master zero-shot subject-driven generation with interleaved multi-image and text input.Source: §1 · p. 3
- Score distillation instruction tuning leaves the diffusion U-Net and VAEs unchanged, allowing KOSMOS-G to replace CLIP in systems using varied U-Net techniques.The paper names fine-grained controls such as ControlNet and personalized or stylized decoder variants such as LoRA checkpoints.Source: §1 · p. 3
- The approach uses compositional instruction tuning to support zero-shot multi-entity subject-driven generation from interleaved multi-image and text input.The paper presents MLLMs as suitable because of their vision-language alignment, support for interleaved inputs, and ability to model multimodal input in context.Source: §1 · pp. 2–3
2 KOSMOS-G: Image as a Foreign Language in Image Generation
KOSMOS-G processes interleaved image and text inputs, aligns its output space with a diffusion decoder through text, and then uses compositional instruction tuning to improve context-sensitive image generation.
2 KOSMOS-G: Image as a Foreign Language in Image Generation
- KOSMOS-G training follows an “align before instruct” pipeline divided into three stages: only the MLLM is trained in Stage 1, AlignerNet with MLLM frozen in Stage 2, then both jointly in Stage 3, while the image decoder remains frozen throughout.Source: §2 · pp. 3–4
2.1 Multimodal Language Modeling
- The multimodal language model represents text and embedded images as one tokenized sequence and processes it autoregressively with a Transformer decoder.Its inputs use start/end sequence and image delimiters; the image embeddings come from a vision Transformer with Resampler pooling.Source: §2 · p. 3, §2.1 · p. 4
2.2 Image Decoder Aligning
- KOSMOS-G aligns its output space with the Stable Diffusion decoder by replacing only the CLIP text encoder, leaving the U-Net architecture and weights unchanged.The decoder alignment stage trains AlignerNet on text-only data with CLIP supervision, using language as the anchor for image inputs too.Source: §2.2 · p. 4, §2 · p. 4
CLIP-T Space
- Text-anchored alignment maps KOSMOS-G’s source space to CLIP’s text space and thereby semantically aligns image embeddings with the decoder input space.AlignerNet uses an encoder to map source embeddings toward CLIP text embeddings and a decoder to reconstruct the source representation.Source: CLIP-T Space · pp. 5–6
2.3 Instruction Tuning
- Compositional instruction tuning uses interleaved captions and entity images with diffusion loss to train KOSMOS-G to generate images from the input instruction.The frozen U-Net supplies the differentiable gradient; the tuning targets richer image features and faithful reproduction across contexts.Source: §2.3 · pp. 6–7
3 Model Training
Training combines web-scale multimodal pretraining with curated compositional-generation and editing data. The constructed data uses captions, extracted entities, and image segmentation, alongside a mixed training set.
3.1 Multimodal Training Data
- Multimodal pretraining uses text corpora, image-caption pairs, and interleaved image-text data; later stages use captions, constructed data, and InstructPix2Pix editing data.Caption sources include LAION-2B, LAION-400M, COYO-700M, and Conceptual Captions.Source: §3.1 · p. 7
- Approximately 9M Open Images V7 images are used to construct compositional instruction-tuning data through captioning, entity extraction, and entity segmentation.The pipeline uses BLIP-2-OPT-6.7b for captions, MPT-7B-Instruct for entity extraction, and CLIPSeg for segmentation.Source: §3.1 · p. 7
3.2 Training Setup
- Instruction tuning blends constructed data, InstructPix2Pix data, and caption data at a 2:2:1 ratio, with entity-text and background details randomly dropped or retained at 0.5 probability.Training took around four days on 256 NVIDIA V100 GPUs: one day for decoder alignment and three days for instruction tuning.Source: §3.2 · p. 7
4 Evaluation
Evaluation reports coherent zero-shot examples, competitive benchmark comparisons, ablations of decoder alignment and instruction tuning, and compatibility with ControlNet and LoRA.
4.1 Main Qualitative Results
- Figure 5 shows coherent zero-shot outputs for customized subjects, including re-contextualization, stylization, modification, accessory incorporation, and multi-entity generation.The paper describes multi-entity generation as a zero-shot capability enabled by compositional generation instruction tuning.Source: §4.1 · pp. 7–8, Figure 5 · p. 8
4.2 Quantitative Results
- On DreamBench, KOSMOS-G outperforms Textual Inversion and Re-Imagen, is marginally better than DreamBooth and BLIP-Diffusion, and is comparable with SuTI; on MS-COCO, it surpasses other CLIP-aligned VL2I models in alignment.Table 1 covers DreamBench comparisons and zero-shot FID comparisons on MS-COCO.Source: Table 1 · p. 8, §4.2 · p. 9
4.3 Ablation Studies
- Direct end-to-end fine-tuning and decoder-only architectures fail to generate meaningful images; AlignerNet with CLIP supervision approaches SD v1.5, while equal-GPU-day end-to-end training performs worse.Figure 6 contrasts pre-instruction KOSMOS-G and the SD baseline, which remain semantically aligned but do not faithfully reproduce entities.Source: §4.3 · p. 9, Table 2 · p. 9, Figure 6 · p. 9
4.4 Applications
- KOSMOS-G replaces CLIP and works with ControlNet and LoRA variants; Figure 7b contrasts standard and LoRA-tuned U-Net outputs.The authors describe these integrations as applications of KOSMOS-G.Source: §4.4 · pp. 9–10, Figure 7 · p. 9
5 Conclusion
The conclusion presents KOSMOS-G as a preliminary effort toward image-as-language generation, supporting zero-shot subject-driven generation from interleaved multi-image and text input.
5 Conclusion
- KOSMOS-G combines competitive single-entity generation with zero-shot multi-entity generation and can replace CLIP alongside U-Net techniques such as ControlNet and LoRA.The authors frame the model as a preliminary effort toward “image as a foreign language in image generation.”Source: §5 · p. 10
Ethics Statement
The ethics statement describes the project’s current access plans and data-filtering measures.
Ethics Statement
- KOSMOS-G is a research project, with no current plans to productize it or expand public access.The authors state that they will apply Microsoft AI principles in further model development.Source: Ethics Statement · p. 10
- Training data underwent filtering to remove inappropriate content, including explicit imagery and offensive language, to minimize the likelihood of inappropriate outputs.Source: Ethics Statement · p. 10
A Detail Training Setup
Training proceeds through multimodal language modeling, image-decoder alignment, and instruction tuning, with separate configurations for each stage.
Multimodal Language Modeling
- Multimodal language modeling used 1.2 million tokens per batch and 300,000 steps, totaling about 360 billion tokens.Each batch comprised 0.5 million text tokens, 0.5 million image-caption tokens, and 0.2 million interleaved-data tokens; AdamW used β = (0.9, 0.98), weight decay 0.01, and dropout 0.1.Source: A Detail Training Setup · p. 15
Image Decoder Aligning
- AlignerNet training used batches of 3,584 sentences for 300,000 steps, with a maximum learning rate of 1e-3.This stage processed approximately 1 billion sentences.Source: A Detail Training Setup · p. 15
Instruction Tuning
- Instruction tuning jointly trained the MLLM and AlignerNet on batches of 1,024 images for 200,000 steps, with a peak learning rate of 1e-3.The training totaled approximately 200 million images.Source: A Detail Training Setup · p. 15
B Images and Prompts for DreamBench Evaluation
The DreamBench evaluation selects reference images per entity and modifies prompts to better align with the training data. The authors note that the prefix “a” can affect editing and customized generation.
B Images and Prompts for DreamBench Evaluation
- Removing the prefix “a” from the remaining prompt slightly affects image editing or customized-generation performance, possibly because training captions often start with “a photo of” and editing data for prompts starting with “a” is limited.Source: B Images and Prompts for DreamBench… · p. 16
- The authors associate the prefix’s effect on editing and customized generation with training captions frequently beginning “a photo of.”They suggest changing the data paradigm or fine-tuning the alignment as possible refinements.Source: B Images and Prompts for DreamBench… · p. 16
- When prompts begin with “a,” the model often refrains from altering the input image, which the authors relate to limited editing data for that prompt form.Source: B Images and Prompts for DreamBench… · p. 16
C Additional Details about Score Distillation Instruction Tuning
The instruction-tuning objective distills the Stable Diffusion U-Net’s learned score function into KOSMOS-G while keeping the U-Net frozen. The section relates diffusion-loss optimization to score distillation and SDS.
C Additional Details about Score Distillation Instruction Tuning
- The authors treat diffusion-loss training as effectively equivalent to optimizing the Score Distillation Sampling loss during instruction tuning.They cite Dreamfusion [PJBM22] for the equivalence when the U-Net is frozen.Source: C Additional Details about Score… · p. 16
- KOSMOS-G is optimized with diffusion loss to encode image features into embeddings understood by Stable Diffusion, while the diffusion U-Net remains frozen.The objective is described as distilling the U-Net’s learned score function into KOSMOS-G.Source: C Additional Details about Score… · p. 16
- Following Dreamfusion [PJBM22], the SDS formulation is obtained by omitting certain terms from the diffusion-loss gradient.The equation labels identify a U-Net Jacobian and a noise residual.Source: C Additional Details about Score… · p. 16
- With the U-Net frozen, optimizing SDS is described as equivalent to optimizing diffusion loss; the authors also frame this as minimizing a KL divergence between conditions and the pretrained score function.The KL-divergence discussion cites [DHP+23] [Luo22].Source: C Additional Details about Score… · p. 16
- Additional Figure 9 examples use three to four interleaved images and text to prompt Stable Diffusion [RBL+22].Source: Figure 9 · p. 17
D Additional Examples
Additional examples show KOSMOS-G handling complex prompts that interleave three to four images with text.
D Additional Examples
- Figure 9 presents three-to-four-image and text interleavings as challenging cases, with the authors reporting robust performance on them.Source: D Additional Examples · p. 17