Source-linked AI summary
ContextGen: Contextual Layout Anchoring for Identity-Consistent Multi-Instance Generation
Ruihang Xu, Dewei Zhou, Fan Ma, Yi Yang
TL;DR
Multi-instance generation must jointly control object layout and preserve distinct subject identities, while available data lacks large-scale, precisely aligned reference-layout pairs. ContextGen unifies layout and reference images through CLA and ICA, and reports state-of-the-art results across three benchmarks, especially in layout and identity measures.
Problem
Existing methods have inadequate spatial precision and weak identity preservation across multiple instances, and datasets lack large-scale, precisely aligned reference images and layout annotations.
Source: §1 · p. 2Method
ContextGen combines a composite layout image and reference images in a DiT framework, using CLA for spatial anchoring, ICA for instance identity transfer, and IMIG-100K for annotated training data.
Source: §1 · p. 2Results
ContextGen reports state-of-the-art performance across three benchmarks, including +1.3% average score over open-source models on LAMICBench++ and +13.3% identity retention over GPT-4o.
Source: §1 · p. 2Takeaways & Limitations
The reported gains are especially strong in layout control and identity preservation across multi-instance generation benchmarks.
Source: §1 · p. 2When references conflict with each other or the prompt, the model tends to prioritize reference identity, which can reduce flexibility in lighting, color, or pose and compromise visual harmony or text-image consistency.
Source: E LIMITATIONS AND FUTURE WORK · p. 30
Abstract
from arXiv · showhide
Multi-instance image generation (MIG) remains a significant challenge for modern diffusion models due to key limitations in achieving precise control over object layout and preserving the identity of multiple distinct subjects. To address these limitations, we introduce ContextGen, a novel Diffusion Transformer framework for multi-instance generation that is guided by both layout and reference images. Our approach integrates two key technical contributions: a Contextual Layout Anchoring (CLA) mechanism that incorporates the composite layout image into the generation context to robustly anchor the objects in their desired positions, and Identity Consistency Attention (ICA), an innovative attention mechanism that leverages contextual reference images to ensure the identity consistency of multiple instances. To address the absence of a large-scale, high-quality dataset for this task, we introduce IMIG-100K, the first dataset to provide detailed layout and identity annotations specifically designed for Multi-Instance Generation. Extensive experiments demonstrate that ContextGen sets a new state-of-the-art, outperforming existing methods especially in layout control and identity fidelity.
1 INTRODUCTION
The introduction identifies joint layout precision, multi-instance identity preservation, and suitable training data as unresolved needs. ContextGen combines layout and reference images with dedicated attention mechanisms, introduces IMIG-100K, and reports benchmark-leading results.
1 INTRODUCTION
- Existing methods are limited in spatial precision, preservation of fine identity details across multiple instances, and availability of large-scale, precisely aligned reference-layout training pairs.The introduction says these deficiencies hinder simultaneous compositional accuracy and identity fidelity.Source: §1 · p. 2
- ContextGen unifies a composite layout image and reference images, using CLA to anchor positions and ICA to transfer fine-grained identity information to desired locations.The layout can be user-provided or automatically synthesized; references also address instance information loss from overlaps in layout-only generation.Source: §1 · p. 2
- IMIG-100K is presented as a large-scale, hierarchically structured dataset with annotated bounding boxes and identity-matched references for image-guided multi-instance generation.The introduction identifies it as the first such dataset and says it addresses current data scarcity.Source: §1 · p. 2
- On LAMICBench++, ContextGen exceeds open-source models by +1.3% average score and GPT-4o in identity retention (+13.3%).On COCO-MIG, it improves instance-level success rate by +3.3% and spatial accuracy (mIoU) by +5.9% over prior art; on LayoutSAM-Eval it attains the highest texture and color fidelity scores.Source: §1 · p. 2
2 RELATED WORK
The related work describes diffusion models progressing toward transformer-based multimodal generation, alongside methods for controlling multiple instances. It situates ContextGen's architecture and attention mechanisms within that landscape.
2.1 DIFFUSION MODELS
- Diffusion models have moved from UNet architectures to transformer approaches such as DiT, while FLUX unifies visual and textual inputs through multimodal attention.The passage also associates DiT with scalable multimodal generation and names Stable Diffusion 3 as an example.Source: §2.1 · p. 2
- Figure 2 presents layout-compositing options, a FLUX DiT-Block generation core, and MM-Attention components including position indexing, CLA, and ICA.Source: Figure 2 · p. 3
2.2 INSTANCE-LEVEL CONTROLLABLE IMAGE GENERATION
- Multi-instance methods include UNet- and DiT-based approaches, while OmniGen2 and DreamO use integrated token sequences but face identity degradation with many subjects.MS-Diffusion and LAMIC combine reference-driven generation with layout control, yet challenges in layout precision and identity consistency remain.Source: §2.2 · p. 3
3 METHOD
ContextGen uses unified multimodal attention with layout and reference images, combining global layout anchoring with instance-level identity preservation. IMIG-100K supplies annotated multi-instance training data through three specialized subsets.
3.1 PRELIMINARIES
- MM-Attention concatenates text embeddings and image tokens into one sequence for joint self-attention across modalities.RoPE is applied using position indices, and FLUX self-attention derives queries, keys, and values from the unified sequence with an all-True mask.Source: §3.1 · pp. 3–4
3.2 CONTEXTUAL ATTENTION WITH LAYOUT ANCHORING AND IDENTITY PRESERVATION
- The composite layout image is the primary input to CLA and may be user-composed or automatically sorted by instance occlusion ratio.The method also supplies high-fidelity reference images because compositing can lose or degrade instance details in overlap regions.Source: §3.2 · p. 4
- CLA processes text, image, and layout tokens in front and back layers for global spatial anchoring, while ICA operates in middle layers for instance-level identity injection.For tokens inside a bounding box, ICA forces connections to the corresponding reference tokens; background tokens use the CLA mask.Source: §3.2 · p. 4
- Distinct position indices for auxiliary layout and reference images let attention distinguish conditioning images from the target latent and from one another.The target latent retains its original indexing; the paper reports validating the refined strategy in an ablation study.Source: §3.2 · p. 5
3.3 IMIG-100K: AN IMAGE-GUIDED MULTI-INSTANCE-GENERATION DATASET
- IMIG-100K was introduced to address the lack of high-resolution, high-fidelity multi-instance data with precise layout and reference images.Its three subsets are basic composition (50K), complex interaction (50K, up to eight instances), and flexible composition with references (10K).Source: §3.3 · pp. 5–6
- The basic and complex subsets use FLUX.1-Dev ground-truth images and derived references, with the complex subset's references edited to simulate occlusion, viewpoint rotation, and pose changes.The flexible-composition subset builds scenes from individual references using subject-driven models and filters for identity consistency.Source: §3.3 · pp. 5–6
4 EXPERIMENTS
Across three benchmarks, ContextGen leads in overall LAMICBench++ score and layout fidelity; ablations examine CLA, ICA placement, and DPO.
4.1 EXPERIMENTAL SETTING
- Evaluation spans LAMICBench++, COCO-MIG, and LayoutSAM-Eval; training fine-tunes FLUX.1-Kontext with LoRA rank 512 for 5K steps and DPO.The benchmarks assess identity preservation, spatial and attribute accuracy, and open-set layout-to-image generation.Source: §4.1 · p. 6
4.2 BASELINES
- Baselines cover layout-to-image, subject-driven, and proprietary and open-source models, with manual annotations and composited inputs for differing requirements.The comparisons include layout-aware methods, single-image editing systems, and models accepting multiple reference images.Source: §4.2 · pp. 6–7, Table 1 · p. 7
4.3 COMPARISON
- 64.66 vs 63.71/64.11: ContextGen achieves the superior overall LAMICBench++ score, outperforming closed-source models on object preservation and facial identity retention.It slightly trails GPT-4o and Nano Banana in ITC and AES.Source: Table 1 · p. 7, §4.3 · p. 7
- ContextGen achieves the highest correctness on COCO-MIG and leads image-guided methods in layout fidelity and LayoutSAM-Eval color and texture accuracy.The authors note that direct comparison with text-guided layout-to-image methods is infeasible because their input modalities differ.Source: Table 2 · p. 8, §4.3 · p. 8
- Qualitative comparisons show preserved object relationships and details, robust handling of overlaps, and accurate fine-grained prompt details with layout control.The model also adapts subjects’ postures and attributes to text prompts rather than rigidly transferring references.Source: §4.3 · pp. 7–9, Figure 4 · p. 7, Figure 5 · p. 8
4.4 ABLATION STUDY
- Without CLA, all metrics decline; selectively applying ICA to MID-19 yields AVG 64.66 and IDS 32.72.The study also reports ICA’s importance for fine-grained identity details in scenes with instance overlaps.Source: §4.4 · p. 9, Table 3 · p. 9
- β = 1000 yields the highest overall average, 62.67, while composition metrics improve and identity metrics decrease slightly as regularization strengthens.This setting outperforms the non-DPO baseline at 62.55 and the other tested β settings.Source: Table 4 · p. 9, §4.4 · p. 9
5 CONCLUSION
The conclusion presents ContextGen as a controllable multi-instance generation framework, paired with a richly annotated dataset and evaluations against state-of-the-art models.
5 CONCLUSION
- ContextGen combines a unified text, layout, and reference-image token sequence with CLA for spatial structure and ICA for instance-specific attribute preservation.The framework also uses a hierarchical attention architecture informed by exploration of attention specialization across DiT layers.Source: §5 · p. 10
- IMIG-100K is a large-scale, hierarchically structured dataset with detailed layout and identity annotations, contributed to accelerate research in this domain.Source: §5 · p. 10
- The authors report that quantitative and qualitative evaluations show ContextGen consistently outperforming state-of-the-art models.They describe the framework as a foundation for customizable image-generation systems.Source: §5 · p. 10
A MORE IMPLEMENTATION DETAILS
Implementation details cover backbone selection, layout compositing and sorting, evaluation inputs, and position-indexing ablations.
A.1 BASE MODEL SELECTION
- FLUX.1-Kontext was selected because it supports attention masks and showed stronger identity preservation than FLUX.1-Fill; FLUX.1-Dev failed with masks without fine-tuning.Source: §A.1 · p. 15
A.2 DETAILS OF COMPOSITING LAYOUT IMAGE
- CLA composes instances using containment-first ordering and a hybrid priority score based on instance area and overlap; at inference, user layouts provide more direct control.The random factor in the score is intended to diversify training data and improve robustness, while overlap relationships may still affect generated images.Source: §A.2 · p. 15
B.1 EXPERIMENTAL DETAILS ON LAMICBENCH++
- For fair LAMICBench++ comparisons, single-image editors receive manually composited layouts, while commercial multimodal models receive multiple references and a dedicated prompt.Source: §B.1 · pp. 15–16
B.2 ABLATION ON POSITION INDEXING
- Removing either x or y offsets significantly degrades all metrics; removing the editing token changes final metrics little but causes some early-training gradient instability.The offsets provide explicit spatial separation in the non-overlapping position-indexing strategy.Source: §B.2 · p. 16, Table 5 · p. 16
B.3.1 QUALITATIVE RESULTS OF DPO FINE-TUNING PROCESS
DPO fine-tuning improves generated scene quality while retaining layout and subject identity when its preference strength is calibrated. The section also examines metric limitations and reports qualitative improvements on the final high-rank model.
B.3.1 QUALITATIVE RESULTS OF DPO FINE-TUNING PROCESS
- With calibrated β, DPO enriched backgrounds while preserving layout, whereas β = 50 risked loss of subject identity during convergence.The illustrated progression adds a realistic shadow and then a coherent background; excessive β may limit meaningful tuning, while too little can overfit preferences.Source: §B.3.1 · pp. 16–17, Figure 7 · p. 17
B.3.2 IMAGE QUALITY ASSESSMENT ON DPO FINE-TUNING
- LayoutSAM-Eval FID may poorly reflect generated-image quality because it misses fine detail and requires resizing structurally diverse ground-truth images.The FID-optimal setting looked less satisfactory in background richness, lighting and shadows, and fine details than a smaller-β setting.Source: §B.3.2 · p. 17, Figure 8 · p. 18
- DPO fine-tuning enhanced image quality in user comparisons, and a smaller β generally led to better quality across LAMICBench++ and LayoutSAM-Eval.The study compared six DPO β settings with a model without DPO fine-tuning, using 10 reviewers.Source: §B.3.2 · p. 17, Table 6 · p. 17
B.3.3 QUALITATIVE VALIDATION ON FINAL MODEL
- On the final ContextGen model with LoRA Rank 512, DPO at β = 1000 improved background detail, composition flexibility, and natural light and shadow rendering.The before-and-after images used the same random seed.Source: §B.3.3 · pp. 17–18, Figure 9 · p. 18
B.4 RESULTS AND ANALYSIS ON MODEL’S GENERATION FLEXIBILITY
Prompt complexity shifts ContextGen’s balance between fidelity to references and layout and flexibility to follow described interactions. ICA’s contribution is clearest in overlapping layouts, where it helps retain identity details that are otherwise lost.
B.4 RESULTS AND ANALYSIS ON MODEL’S GENERATION FLEXIBILITY
- More detailed interaction prompts increase generation flexibility, while simple prompts favor fidelity to reference-image details and posture.With a fixed layout, prompts can change contextual details or posture to better match the described action.Source: §B.4 · p. 18, Figure 10 · p. 19
B.5 ICA: AN INDISPENSABLE COMPONENT HANDLING IDENTITY IN OVERLAPPED REGIONS
- In overlapping layouts, ICA’s reference tokens provide identity information for occluded regions, helping mitigate identity loss.Without ICA, examples lose the headphones’ red logo and fine facial features such as eye color and the nasolabial fold.Source: §B.5 · pp. 18–19, Figure 11 · p. 19
- Overlapping cases are relatively rare in the benchmark, so their image-level differences have limited impact on macro-level evaluation indicators.The qualitative comparisons nevertheless show identity and visual-detail losses without ICA in these cases.Source: §B.5 · p. 19
B.6 ANALYSIS ON LAYER-WISE ICA ABLATION
The layer-wise ablation finds that ICA placement matters: Mid-19 performs strongly, while some broader or late-only placements score lower. A controlled MS-FLUX comparison reports poor identity preservation and degraded layout control on FLUX-DiT.
B.6 ANALYSIS ON LAYER-WISE ICA ABLATION
- F + M + B, F + M, and B yielded lower scores, while F, without CLA, M + B, and M clustered at higher scores, indicating sensitivity to ICA placement.The reported score groups suggest different functional behavior across FLUX-DiT layers.Source: §B.6 · p. 19
- F, M, M + B, and no ICA had relatively close performance scores, which the authors relate to avoiding further disruption of Front-19 blocks.The layer-placement discussion contrasts these strategies with broader ICA application, which can have side effects.Source: §B.6 · p. 20
- Applying ICA only to Back-19 was detrimental, with the authors associating late-stage masking with artifacts and blurred identity.The text describes Back-19 as sensitive post-processing layers and reports that adding ICA to more blocks can also cause identity distortions.Source: §B.6 · p. 20
- Applying ICA to Mid-19 blocks had an advantage on LAMICBench++ average score over other strategies and was selected as a relatively efficient configuration.The ablation identifies Mid-19 as the best-performing placement for this task.Source: §B.6 · p. 20
B.7 MS-FLUX: A CONTROLLED CASE STUDY OF STRATEGY FAILURE ON STRONG BACKBONE
- After 50K training steps on IMIG-100K, MS-FLUX showed severely poor identity preservation and complete degradation of layout control on COCO-MIG and LAMICBench++.The comparison adapts a Grounding Resampler to FLUX-DiT with a LoRA adapter of rank 512.Source: §B.7 · p. 20, Table 7 · p. 21
C.1 BRIEF SURVEY AND DISCUSSION ON EXISTING MULTI-INSTANCE DATASETS
Existing multi-instance datasets have gaps in detailed layout annotations, examples with more than three subjects, and variation between references and targets. The paper describes IMIG-100K’s synthetic construction, spatial annotations, and quality and diversity measures, then motivates visual comparisons across FLUX-based methods.
C.1 BRIEF SURVEY AND DISCUSSION ON EXISTING MULTI-INSTANCE DATASETS
- Existing datasets offer limited data with more than three subjects and lack detailed spatial or layout annotations; MS-Diffusion’s cross-verified data can also restrict reference-to-target variation.Low identity matching across video frames can require using parts of the ground-truth image directly as reference.Source: §C.1 · p. 23
- The authors cite real-world collection and matching as less flexible and harder to scale in diversity and quantity when motivating synthetic IMIG-100K construction.The proposed dataset pipeline is described as addressing limitations of existing training resources.Source: §C.1 · p. 23
C.2 MORE DETAILS OF IMIG-100K DATASET
- IMIG-100K varies prompts and styles, includes fantasy and anime alongside realistic images, and keeps natural scenes at approximately 85% to 90%.Multiple prompt templates are combined and fed to LLMs for diversity; different LLMs contribute lexical and stylistic variations.Source: §C.2 · pp. 23–24, Figure 12 · p. 24
- IMIG-100K annotates instance phrases, valid-area boxes and masks, and positions; its flexible-composite subset also aligns and filters face pairs.The face-position annotations support positioning and sizing reference instances more precisely than whole-instance boxes.Source: §C.2 · p. 24
- The dataset assesses reference and ground-truth image quality and uses MLLMs to check semantic consistency between references and generated captions.The paper provides additional samples illustrating quality and annotation detail.Source: §C.2 · p. 24
D.1 QUALITATIVE RESULTS COMPARING ACROSS FLUX-BASED METHODS
- The authors provide qualitative comparisons of FLUX-based methods on LAMICBench++ because the methods use highly similar or identical foundational backbones.They describe the visual comparison as direct and fair.Source: §D.1 · p. 24
D.2 MORE RESULTS ON COCO-MIG
COCO-MIG results report leading average success rate and mIoU, with the strongest success-rate advantages in complex scenes. Qualitative results also highlight layout and detail preservation on COCO-MIG and LayoutSAM-Eval.
D.2 MORE RESULTS ON COCO-MIG
- 33.12% average Success Rate and 65.12% mIoU are the highest among compared methods on COCO-MIG, with Success Rates of 28.12%, 23.12%, and 24.38% at L4, L5, and L6.The largest performance advantage is reported in complex, high-instance-count scenarios.Source: §D.2 · pp. 24, 28, Table 8 · p. 28
- COCO-MIG examples show more accurate vase placement and green-pot attribute binding, while other methods sometimes merge instances or produce less coherent compositions.The authors also report stronger layout control than MS-Diffusion and more robust subject fidelity than text-guided methods in intricate scenes.Source: §D.2 · p. 28, Figure 17 · p. 29
D.3 MORE QUALITATIVE RESULTS ON LAYOUTSAM-EVAL
- LayoutSAM-Eval examples show stronger visual quality and realism, with more faithful preservation of details such as building text and shorts color.The authors contrast this with text-guided layout-to-image methods that struggle with such fine-grained attributes.Source: §D.3 · p. 28, Figure 18 · p. 29
E LIMITATIONS AND FUTURE WORK
ContextGen prioritizes reference-subject identity when references conflict with one another or the text prompt, which can reduce flexibility in adjusting visual attributes and compromise harmony or text-image consistency.
E LIMITATIONS AND FUTURE WORK
- When references conflict with one another or with the prompt, the model tends to prioritize reference identity, potentially reducing flexibility in lighting, color, or pose.The authors identify balancing identity fidelity with contextual flexibility as future work.Source: E LIMITATIONS AND FUTURE WORK · p. 30
- This trade-off can compromise overall visual harmony and text-image consistency in the generated output.The proposed future direction is more dynamic attention that better balances identity fidelity with flexibility for style and attribute transfer.Source: E LIMITATIONS AND FUTURE WORK · p. 30
- The current scope is 2D image synthesis; extending identity-consistent control to 3D assets remains a future direction.The authors also plan to draw on zero-shot consistency priors to improve multi-instance scene harmony.Source: E LIMITATIONS AND FUTURE WORK · p. 30