M$^{2}$Chat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generation

1. Introduction

• Previous approaches apply the same knowledge across tasks, overlooking task-specific characteristics of vision-language models..Source: §1 · p. 2
• The paper identifies difficulty aligning complex multimodal features into coherent text-image pairs and tailoring VLMs for diverse, contextually consistent dialogue.Source: §1 · p. 2
• M2Chat integrates low-level visual information with high-level semantic features through a learnable gate that balances modality contributions and creativity with consistency.It combines Stable Diffusion XL with LLaMA-AdapterV2.Source: §1 · p. 2
• M3FT uses distinct parameter groups for image-text alignment and visual-instruction tuning.Its first stage tunes alignment parameters; its second adds a specific token and trains M3Adapter components on instruction data from different fields.Source: §1 · p. 2
• M2Chat is reported to outperform current models in fine-tuning efficiency and generation quality across image editing, storytelling, and multimodal dialogue.The paper also describes its outputs as imaginary yet coherent images and text.Source: §1 · p. 2

2.1. Multimodal Large Language Model

• Prior multimodal language models focus on captioning, visual understanding, instruction tuning, or adding input modalities, while few extend VLM output to text-image generation.Examples include KOSMOS-1 [8], FROMAGe [10], BLIP-2 [12], Llava, Llava1.5, and MiniGPT4 [49].Source: §2.1 · p. 2

2.2. VLM Downstream Tasks

• Image-generation work spans stable diffusion, prompt-based editing, and ControlNet, while interleaved methods vary in conditioning, resource needs, and alignment scope.DALLE-3 uses prompts without image conditions; Emu [38] requires substantial computation; NextGPT [41] aligns output text embeddings.Source: §2.2 · p. 2
• The paper identifies efficient alignment and VLM generalization in interleaved text-image generation as areas that remain unexplored.Figure 2 depicts M2Chat’s image-and-text pipeline and names T2I generation, storytelling, and multimodal dialogue as its three key areas.Source: §2.2 · p. 3, Figure 2 · p. 3

3. Proposed Method

• M²Chat pairs LLaMA-AdapterV2 with SDXL: the VLM encodes multimodal context, the adapter aligns features, and SDXL synthesizes images.The framework targets cohesive text-image conversations.Source: §3.1 · p. 3, §3 · p. 3

3.1. Preliminary

• Visual-instruction contexts use <|img|> for padding and <|IC|> to mark an image caption; the target second-round caption should meet task demands and guide image generation.The context includes task prompts, an initial caption, preceding dialogue, and response text.Source: §3.1 · p. 3

3.2. Framework Architecture

• M3Adapter maps VLM features toward SDXL’s text-feature space, then fuses low-level visual features with high-level semantics using a gate modulated by answer-caption similarity.The fusion is intended to balance creativity in multimodal dialogue with coherence in other interleaved tasks.Source: §3.2 · p. 4

3.3. Training Strategy

• In M3FT’s alignment stage, an alignment loss complements DDPM loss, and only M3Adapter is updated; the paper reports alignment success across diverse multimodal generation tasks.Aligned features condition the denoising phase, and the multimodal loss weights alignment with a hyperparameter.Source: §3.3 · pp. 4–5
• The second M3FT stage tunes answer and image-token generation on text-image patterns, updating M3Adapter components and using text-conditioning and DDPM losses.The paper describes this stage as targeting consistency between generated answers and images.Source: §3.3 · p. 5

4.1. Downstream Tasks

• The evaluated tasks cover chat-based image generation, instruction-based editing, illustrated storytelling, and multimodal dialogue with topic-specific visuals.Source: §4.1 · p. 5

4.2. Experiment Setup

• Evaluation uses MS-COCO, CC3M, and MMDialog; metrics include CLIP and FID for visual quality, BLEU and ROUGE for text, and InterRel for conversational cross-modal alignment.Training uses CC3M and LAION-Aesthetics; the reported implementation trains the M3Adapter on eight H800 GPUs.Source: §4.2 · pp. 5–6

4.3. Quantitative Results

• On MS-COCO, M2Chat scored 29.87, surpassing other multimodality generation models by 0.56 and slightly exceeding Emu2.The paper reports this as a state-of-the-art score.Source: Table 1 · p. 7, §4.3 · p. 7
• On CC3M, M2Chat improved FID by 2.56 and CLIP by 1.51 over MiniGPT5, which has a similar-sized LLM.Source: Table 1 · p. 7, §4.3 · p. 7

4.5. Ablation Study

• With M3FT, MS-COCO CLIP improved by 1.39; on CC3M, FID improved by 2.56 and CLIP by 1.64 versus M2Chat without M3FT.Source: Table 1 · p. 7, §4.5 · p. 8

4.3. Quantitative Results

• On MMDialog, M2Chat improved InterRel by 5.52, BLEU-1 by 2.77, BLEU-2 by 2.16, and ROUGE-L by 4.62 over VLM+SD finetune.Both models used the same pretraining and finetuning settings.Source: Table 2 · p. 7, §4.3 · p. 7

4.4. Qualitative Comparisons

• Storytelling pairs text with four storyline-linked images, while qualitative comparisons show consistent instruction-following edits, including pose changes and character replacement.Against SDXL-generated pictures using M2Chat prompts, M2Chat shows clearer story progress, especially in the last two pictures; hidden features also transform from dog to cat under instruction.Source: §4.4 · pp. 7–8

4.6. Efficiency Comparison

• M2Chat uses fewer training resources than compared models: Emu and SEED exceed 10,000 GPU hours, while DreamLLM exceeds 2,240 GPU hours.M2Chat uses around 2.5 million data points, close to MiniGPT5, and has fewer additional parameters than billion-parameter-scale works.Source: §4.6 · p. 9, Table 3 · p. 9

• M2Chat combines LLaMA-AdapterV2 and SDXL with M3Adapter feature alignment and gating, plus two-stage M3FT to enhance semantic consistency across multimodal interleaved tasks.Source: §5 · p. 9