Source-linked AI summary
Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation
Yabo Zhang, Kunchang Li, Dewei Zhou, Xinyu Huang, Xun Wang
TL;DR
Complex interleaved image-text instructions remain challenging because existing methods struggle with multi-image constraints and limited training data. INSET embeds images as native vocabulary within instructions and introduces a scalable data engine, achieving superior consistency and text alignment as complexity increases.
Problem
Existing interleaved generation methods handle simple references but struggle with complex multi-image constraints, while available datasets lack scale, complexity, and long-horizon interactions.
Method
INSET embeds images at corresponding semantic slots as native instruction vocabulary and uses a scalable engine to construct 15M interleaved samples.
Results
INSET achieves superior image and text consistency on complex generation tasks, with advantages amplifying as input complexity increases.
Takeaways & Limitations
Treating visual content as dense language tokens provides a unified approach that also generalizes to multimodal image editing.
Abstract
from arXiv · showhide
While recent advancements in multimodal language models have enabled image generation from expressive multi-image instructions, existing methods struggle to maintain performance under complex interleaved instructions. This limitation stems from the structural separation of images and text in current paradigms, which forces models to bridge difficult long-range dependencies to match descriptions with visual targets. To address these challenges, we propose \texttt{I}mages i\texttt{N} \texttt{SE}n\texttt{T}ences (\textit{a.k.a}, INSET), a unified generation model that seamlessly embeds images as native vocabulary within textual instructions. By positioning visual features directly at their corresponding semantic slots, INSET leverages the contextual locality of transformers for precise object binding, effectively treating images as dense, expressive language tokens. Furthermore, we introduce a scalable data engine that synthesizes 15M high-quality interleaved samples from standard image and video datasets, utilizing VLMs and LLMs to construct rich, long-horizon sequences. Evaluation results on InterleaveBench demonstrate that INSET significantly outperforms state-of-the-art methods in multi-image consistency and text alignment, with performance gaps widening as input complexity increases. Beyond standard generation, our approach inherently extends to multimodal image editing, integrating visual content as part of the instruction to facilitate highly expressive and creative visual manipulations.
1 Introduction
INSET addresses the difficulty of complex interleaved image-text instructions by embedding images as native vocabulary at their corresponding semantic slots. It combines this unified representation with scalable data generation and evaluation, achieving stronger consistency and extending naturally to multimodal image editing.
- Motivation: Existing methods degrade sharply on complex interleaved instructions because indirect image references create difficult long-range alignment between indices and visual features.They commonly retrieve visual content through explicit references such as “the dog in Image 1,” while complex datasets also remain limited in scale and interaction richness.
- Method: INSET embeds images as native vocabulary directly at their corresponding semantic slots, using transformer contextual locality for precise object binding.This treats input images as a detailed form of language rather than external references requiring retrieval.
- Data and Evaluation: InterleaveBench evaluates complex multi-image compositions with intricate interleaved instructions, while INSET is implemented on top of BAGEL.The benchmark is designed to assess capabilities on complex interleaved tasks.
- Data and Evaluation: 15M high-quality interleaved samples are constructed from image and video datasets by a scalable data engine, addressing the lack of rich long-horizon training examples.The resulting data targets intricate compositional reasoning involving dense visual contexts.
- Results and Extensions: INSET achieves superior image and text consistency, with advantages amplifying as complexity increases, and naturally generalizes to multimodal image editing.Both textual instructions and visual reference tokens guide the editing process.
2 Related Works
Prior interleaved image-text generation methods commonly rely on pretrained visual encoders, while existing datasets remain inadequate for complex instruction-following because of alignment noise or visual redundancy.
- Interleaved Image-Text Generation: Early interleaved generation methods use pretrained image encoders such as CLIP to extract visual features but can produce rigid copy-paste artifacts and conflate multiple references.These limitations arise when processing multiple reference images.
- Interleaved Dataset Construction: Existing interleaved datasets poorly support complex instruction-following: web-crawled corpora have loose semantic alignment and noisy correlations, whereas video-derived data contains high visual redundancy.Web corpora are suboptimal for precise generation, while video datasets primarily target multi-turn editing.
3 Method
INSET unifies interleaved image generation by embedding visual features directly into their corresponding textual semantic slots, enabling precise multimodal binding. Its method combines a scalable data engine for 15M interleaved samples with specialized architecture, guidance, and evaluation for complex multi-image scenarios.
- Interleaved Data Engine: 15M high-quality interleaved samples are curated from real-world image and video corpora through a scalable data engine.For images, the pipeline combines VLM global captioning, fine-grained object processing with masks and captions, and LLM-driven insertion into precise semantic contexts.
- Native Interleaved Formulation: INSET treats input images as native vocabulary, placing visual features at their corresponding semantic slots to exploit transformer contextual locality.This expands prompts from text-only inputs to expressive interleaved instructions and avoids the long-range dependencies of query-based formulations.
- Model Architecture: INSET uses a Mixture-of-Transformer architecture with an understanding branch for interleaved instructions and a generation branch for image synthesis.It uses semantic ViT embeddings alone, discarding pixel-level VAE latents to mitigate image-pasting and encourage semantic composition.
- Inference Strategy: s1 = 4.0 boosts textual adherence before global classifier-free guidance with s2 = 1.5.The two-stage strategy first calibrates text relative to visual conditioning, then applies guidance using the null embedding ∅.
- Benchmark: InterleaveBench is introduced to evaluate complex multi-image scenarios that existing benchmarks inadequately cover.The benchmark addresses limited reference images and simple spatial relationships in DreamBench++ and OmniContext.
4 Experiments
Experiments show that Inset outperforms open-source baselines in consistency, instruction following, and attribute binding, with stronger advantages as input complexity increases. Ablations further show that native image placement, video-based data, and removing the VAE improve complex interleaved understanding and editing.
- Qualitative Comparisons: Inset avoids common baseline failures, including object misalignment, ignored visual inputs, and inaccurate binding of attributes such as a cream-colored sweater.DreamOmni 2 and Flux-Kontext are specifically described as weaker in precise attribute binding.
- Overall Results: Inset significantly outperforms open-source methods in visual consistency and instruction following, while achieving performance comparable to powerful closed-source models despite having the fewest parameters.Its advantage over state-of-the-art open-source baselines becomes increasingly pronounced as the number of input images rises.
- Multimodal Image Editing: Inset integrates interleaved instruction understanding with text-guided image editing, enabling edits driven by instructions that combine text with specific visual references.Figure 6 demonstrates this capability by comparing editing with and without input images.
- Effect of Image Placement: Explicitly positioning visual tokens improves precise object binding over the standard Image First formulation by exploiting contextual locality instead of long-range dependencies.The Image First baseline often misbinds attributes to targets or neglects specific image inputs.
- Effect of Video-based Data: Adding video-based interleaved data enables more natural and semantically accurate object interactions beyond the satisfactory consistency and simple spatial alignment achieved with image-only training.Image-only training falters when altering object attributes or synthesizing interactive dynamics from text.
- Effect of VAE Vision Encoder: Removing the pixel-level VAE encoder avoids image-pasting, object omission, and inferior consistency caused by rigid pixel copying and excessive token overhead.The comparison is against the standard BAGEL architecture with VAE, favoring the ViT-only design.
5 Conclusion
INSET is a unified generation model that embeds images as native vocabulary within textual instructions for precise object binding. The paper also introduces a scalable data engine that constructs 15M high-quality interleaved samples and validates the model on InterleaveBench.
- 5 Conclusion: INSET embeds images as native vocabulary within textual instructions, positioning visual features at corresponding semantic slots for precise object binding.This design addresses limitations of indirect referencing mechanisms.
- 5 Conclusion: 15M high-quality interleaved samples are constructed using a scalable data engine developed to enable the proposed paradigm.
- 5 Conclusion: The model is validated on the proposed InterleaveBench.