MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer

1 Introduction

• Fixed visual-token budgets can overlook critical image details, while increasing tokens per image strains limited LLM context, especially for multi-image inputs.The paper identifies context-insensitive Perceiver Resamplers, which compress images without using LLM context, as the central issue.Source: §1 · p. 2
• MM-Interleaved uses MMFS in an end-to-end model that generates textual and visual responses to interleaved image-text inputs.During generation, tokens can access multi-scale high-resolution image features while maintaining the causal relation between image and text tokens.Source: Figure 1 · p. 2, §1 · p. 3
• The pipeline predicts text words from LLM text-token features and invokes a diffusion-based image decoder when it predicts a <BoI> token.The image decoder receives prior LLM output features as conditions and can extract details from previous images.Source: §1 · p. 3
• Pre-training uses paired and interleaved image-text data, followed by supervised fine-tuning on multimodal tasks including visual question answering and visual storytelling.The paper reports competitive results and higher token efficiency than previous image-to-text and text-to-image methods, and new state-of-the-art results against joint image-text generation models.Source: §1 · p. 3
• MMFS dynamically extracts relevant details from multi-scale feature maps of multiple images using intermediate LLM context features.This is intended to preserve fine-grained visual information with fewer visual tokens per image.Source: §1 · pp. 2–4
• The model achieves state-of-the-art results on varied multimodal comprehension and generation benchmarks without using in-house data.It generates accurate text descriptions and visually consistent images from interleaved image-text inputs.Source: §1 · pp. 3–4

2 Related Work

• Paired image-text research spans contrastive understanding, captioning, text-to-image generation, and LLM-based multimodal models.Examples include LLaVA [54], which connects an LLM with pretrained visual encoders through linear projections.Source: §2 · p. 4
• Interleaved-data work progressed from understanding systems to models generating images and text, but prior generative approaches retain distinct limitations.Some systems generate only text; others use two-stage generation, focus mainly on image generation, or add an image tokenizer-detokenizer training stage.Source: §2 · p. 4
• Fixed-token Perceiver Resamplers can overlook image details, while raising visual-token counts increases computational and memory demands, particularly with multiple images.Prior work ranges from 32 or 64 tokens per image to 2,890 in SPHINX.Source: §2 · p. 5

3.1 Task Formulation

• The task treats content as text tokens and whole images in original order, training likelihoods for next text or image conditioned on preceding embeddings.Text and images are embedded before being fed to the model, and the objective sums likelihoods over the text-token and image positions.Source: §3.1 · p. 5

Text Generation with Multi-modal Condition. log p(xL

• Text generation uses an LLM-conditioned linear classifier to predict the next text token against the ground-truth token.The ground-truth text is represented as a one-hot vector.Source: Text Generation with Multi-modal… · p. 6

Image Generation with Multi-modal Condition. Maximizing log p(xV

• The image branch trains a diffusion denoiser to predict noise in a noisy target image conditioned on previous LLM context, including interleaved inputs.The framework can combine language models, image encoders, and image decoders and be optimized end-to-end.Source: Image Generation with Multi-modal… · p. 6

3.2 Architecture

• The architecture integrates a visual foundation model, LLM, and diffusion model; MMFS lets intermediate LLM layers retrieve multi-scale image features on demand and supplies detailed visual conditions to the image decoder.Training sums next-text-token and next-image prediction losses, and the framework is optimized end-to-end.Source: §3.2 · pp. 7, 9

4.2 Evaluation

• Zero-shot MM-Interleaved reports strong comprehension across benchmarks and state-of-the-art results on SEED-Bench-2 interleaved comprehension and generation.The SEED-Bench-2 table reports average task accuracy across image/text comprehension, interleaved comprehension, and image/text generation.Source: §4.2 · pp. 10–11, Table 2 · p. 11
• Fine-tuned MM-Interleaved reports state-of-the-art comprehension, matches LLaVA-1.5 on visual question answering, and uses 64 rather than 576 visual tokens.It also generates images as well as text, while LLaVA-1.5 generates text only.Source: Table 1 · p. 10, §4.2 · p. 11
• For referring expression comprehension, MM-Interleaved matches Qwen-VL using public REC training data and 224-pixel resolution, versus Qwen-VL’s extra 22M in-house grounding examples and 448-pixel resolution.The comparison is reported for the referring-expression-comprehension benchmarks.Source: §4.2 · p. 12, Table 4 · p. 12
• MM-Interleaved reports state-of-the-art visual storytelling results and outperforms baselines, including ControlNet, on segmentation-to-image translation.The storytelling evaluation includes last-image generation on VIST and autoregressive multi-image generation on Pororo and Flintstones.Source: §4.2 · p. 12, Table 5 · p. 12

4.3 Ablation Study

• MMFS lets 32 visual tokens per image outperform 256 tokens without MMFS, with a larger improvement when input resolution rises from 224 to 448.The ablation evaluates captioning, text-to-image generation, and visual question answering.Source: §4.3 · p. 13, Table 6 · p. 13
• Without MMFS, segmentation-to-image translation fails to preserve spatial information and achieves extremely low mIoU.This task requires pixel-level alignment between the segmentation condition and generated image.Source: §4.3 · p. 13, Table 6 · p. 13
• Deformable attention performs significantly better than other tested attention mechanisms on TextVQA, while replacing MMFS with dense cross-attention lowers overall performance.The authors suggest slower convergence as a possible reason for the dense-attention drop.Source: Table 7 · p. 13, §4.3 · p. 14
• Multi-image and multi-scale MMFS improves performance; adding it to 32-token inputs costs about 2% more FLOPs, parameters, 6% runtime, and 3% memory.Against 256 visual tokens without MMFS, the 32-token MMFS setup achieves better performance with 2.8× fewer FLOPs and 1.3× fewer runtime.Source: §4.3 · pp. 14–15, Table 8 · p. 14, Figure 6 · p. 14

5 Conclusion

• MM-Interleaved is an end-to-end trained model for interleaved image-text comprehension and generation, built on MMFS to reduce visual-token requirements and extract visual details.The paper reports state-of-the-art performance on various multi-modal benchmarks without in-house data.Source: §5 · p. 15
• The authors identify the relatively low quality and quantity of public interleaved image-text data as limiting the ability to exploit the full potential of interleaved generative modeling.They aim to improve this data while ensuring safety and reliability.Source: §5 · p. 15
• The authors note that noisy training data may contribute to hallucinations and biased generated content.They frame these as potential negative impacts, not established outcomes in every use.Source: §5 · p. 15

A.1 Pre-training

• Pre-training uses MMC4, LAION-2B, LAION-COCO, CC-12M, and Objects365, with BLIP-2 captions substituted for original CC-12M and Objects365 annotations.The LAION-2B subset is further filtered, and MMC4 is filtered by CLIP similarity while limiting each document to six images.Source: §A.1 · p. 22
• Image-text pairs are sampled and concatenated to the 2048-token context limit, while interleaved MMC4 documents are split and concatenated into training samples.The stated purpose is to use the LLM context window and improve data efficiency.Source: §A.1 · p. 22

A.2 Supervised Fine-tuning

• Supervised fine-tuning uses public VQA, captioning, and referring-expression datasets, with REC boxes counted correct when their IoU with ground truth exceeds 0.5.The appendix also specifies public training data for segmentation-to-image translation and visual storytelling.Source: §A.2 · pp. 22–23
• Storytelling training predicts images from sequential text-image samples; inference generates VIST’s last image or the final four Pororo and Flintstones images autoregressively.VIST uses FID and CLIP similarity for evaluation; Pororo and Flintstones use FID.Source: §A.2 · p. 23

A.3 Evaluation

• The benchmark set spans image captioning, visual question answering, and text-to-image generation, among other tasks.Image captioning includes short-caption datasets COCO Caption, Flickr30k, and NoCaps, plus long-caption Image2Paragraph.Source: §A.3 · p. 24
• Image-generation evaluations use classifier-free guidance scale 3.5 and 250 inference steps by default.Source: §A.3 · p. 24
• The appendix provides prompt templates for each text-generation task.Source: §A.3 · p. 24

• These additional ablations use the main-text settings by default, and the compared models are not fine-tuned on downstream tasks.Source: B Additional Ablation Studies · p. 24

Pre-training with Different Loss Terms

• Joint LNT P and LNIP training performs better on comprehension and generation, with λ = 10 empirically balancing the two losses.The comparison is summarized in Table 16; no metric values are provided here.Source: B Additional Ablation Studies · pp. 24–25

The Relationship between MMFS and Resampler

• Tab. 17 results validate mutual benefits between the proposed MMFS module and the Resampler in the image tokenizer, as removing either leads to overall performance degradation.Source: B Additional Ablation Studies · p. 25

• Examples show emergent multimodal in-context learning and text generation in complex robotics, gaming, and GUI scenarios.The few-shot examples use interleaved images and text as context.Source: §C.1 · p. 25, Figure 7 · p. 26
• The model generates images described as appropriate to contexts specifying desired styles or concepts.Figure 9 presents zero-shot image-generation examples with interleaved images and text.Source: §C.1 · p. 25
• The zero-shot examples also show the model generating images and text interleavedly.Source: §C.1 · p. 26

• MMFS yields more accurate answers to TextVQA queries requiring fine-grained visual details and more accurate coordinates for referring expressions.Source: §C.2 · pp. 26–27, Figure 11 · p. 32, Figure 12 · p. 32
• For segmentation-to-image translation, MMFS produces images whose spatial layout is significantly closer to the ground truth than results without MMFS.The comparison conditions generation on a text prompt and segmentation map.Source: §C.2 · p. 27, Figure 13 · p. 33
• Sequential images generated with MMFS show better spatial consistency and closer semantic alignment with the interleaved image-text context than outputs without it.The comparison includes background, viewpoint, and character-position relationships.Source: §C.2 · p. 27, Figure 14 · p. 34
• Given the first frame and caption as context, the fine-tuned model generates following interleaved images and texts coherently, balancing diversity with spatial-semantic consistency.Source: §C.2 · pp. 27–28, Figure 15 · p. 35