PaperlayerView exact revision on arXiv

Source-linked AI summary

TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation

Junhao Cheng, Baiqiao Yin, Kaixin Cai, Minbin Huang, Hanhui Li, Yuxin He, Xi Lu, Yue Li, Yifei Li, Yuhao Cheng, Yiqiang Yan, Xiaodan Liang

arXiv:2404.18919v2cs.CV

TL;DR

Multi-turn image generation must preserve both semantic alignment with user instructions and the identity of subjects across interactive turns. TheaterGen combines an LLM-managed character prompt book with reference images and guidance during T2I generation, and it is evaluated on a new benchmark spanning story generation and editing. The reported results show substantial improvements over prior methods, while remaining limited by randomness in image adapters and the generation capabilities of T2I models.

  • Problem

    Multi-turn image generation struggles with semantic consistency between images and text and contextual consistency of the same subject across turns.

    Source: §1 · pp. 2–3

  • Method

    TheaterGen is a training-free framework in which an LLM maintains character prompts and layouts, while reference images and guidance steer T2I generation.

    Source: §1 · p. 3

  • Results

    TheaterGen outperforms existing methods across story generation and multi-turn editing, including 27% and 21% gains in aFID and aCCS and a 19% gain in aTIS over Mini DALLE·3 with diffusion version XL.

    Source: §1 · p. 3, §4.2 · p. 11

  • Takeaways & Limitations

    TheaterGen provides a unified approach for evaluating and improving semantic and contextual consistency in multi-turn image generation.

    Source: §1 · pp. 3–4

    Randomness in existing pre-trained image adapters and limited T2I generation capability can still cause contextual and semantic inconsistency.

    Source: §5 · p. 15

Abstract

from arXiv · show

Recent advances in diffusion models can generate high-quality and stunning images from text. However, multi-turn image generation, which is of high demand in real-world scenarios, still faces challenges in maintaining semantic consistency between images and texts, as well as contextual consistency of the same subject across multiple interactive turns. To address this issue, we introduce TheaterGen, a training-free framework that integrates large language models (LLMs) and text-to-image (T2I) models to provide the capability of multi-turn image generation. Within this framework, LLMs, acting as a "Screenwriter", engage in multi-turn interaction, generating and managing a standardized prompt book that encompasses prompts and layout designs for each character in the target image. Based on these, Theatergen generate a list of character images and extract guidance information, akin to the "Rehearsal". Subsequently, through incorporating the prompt book and guidance information into the reverse denoising process of T2I diffusion models, Theatergen generate the final image, as conducting the "Final Performance". With the effective management of prompt books and character images, TheaterGen significantly improves semantic and contextual consistency in synthesized images. Furthermore, we introduce a dedicated benchmark, CMIGBench (Consistent Multi-turn Image Generation Benchmark) with 8000 multi-turn instructions. Different from previous multi-turn benchmarks, CMIGBench does not define characters in advance. Both the tasks of story generation and multi-turn editing are included on CMIGBench for comprehensive evaluation. Extensive experimental results show that TheaterGen outperforms state-of-the-art methods significantly. It raises the performance bar of the cutting-edge Mini DALLE 3 model by 21% in average character-character similarity and 19% in average text-image similarity.

1 Introduction

TheaterGen addresses semantic and contextual consistency challenges in multi-turn image generation with an LLM-managed character representation and reference-guided synthesis. It introduces CMIGBench to evaluate story generation and multi-turn editing, with results showing substantial gains over existing methods.

Motivation

  • The framework targets two persistent failures in multi-turn generation: semantic mismatch in complex references and spatial relations, and changing or forgotten entities across turns.These issues include difficulty handling quantities, spatial relations, and expressions such as “it” and “they.”Source: §1 · pp. 2–3

Problem and proposed framework

  • TheaterGen uses an LLM to manage standardized character prompts and bounding boxes, simplifying multi-turn intent interpretation and supporting semantic consistency.The framework represents each foreground character with structured prompt and layout information.Source: §1 · p. 3
  • Reference images preserve character context across turns, while lineart and latent guidance help generate consistent multi-character scenes without specialized training.Character references are combined with current-turn prompts before final image generation.Source: §1 · p. 3

Evaluation benchmark

  • CMIGBench contains 8000 four-turn scripted dialogues covering story generation and multi-turn editing without pre-defined characters.The benchmark evaluates both semantic and contextual consistency in a setting where characters may emerge during interaction.Source: §1 · p. 3

2 Related Works

Prior work spans diffusion-based text-to-image generation, LLM-mediated multi-turn interaction, and iterative image editing. TheaterGen extends these directions with structured character prompts and layouts for multi-turn generation.

2.1 Text to Image Generation

  • Diffusion models generate images from text through multimodal text encoding and cross-modal attention, forming the foundation for modern text-to-image systems.The related work section situates TheaterGen within diffusion-based image generation.Source: §2.1 · p. 4

2.2 Multi-turn Image Generation with LLMs

  • LLM-integrated systems improve natural-language interaction by interpreting dialogue, images, or multimodal inputs to guide subsequent image generation.Examples include Mini DALL·E 3, Diffusion GPT, Next GPT, and MiniGPT-5.Source: §2.2 · p. 4
  • TheaterGen maintains foreground-character prompts and layouts during multi-turn interaction to support consistent image generation.Its framework is organized around an LLM-based character designer, character image management, and character-guided generation.Source: §2.2 · p. 4, Figure 2 · p. 5

2.3 Compositional Image Generation Task for Multi-turn Image Editing

  • Compositional multi-turn editing methods use recurrent interpretation, attention, or loss guidance to incorporate previous conditions and current instructions.GeNeVA-GAN, SSCR, and LatteGAN represent distinct approaches to iterative editing without LLMs.Source: §2.3 · p. 5

3 Method

TheaterGen is a training-free framework that coordinates LLM-generated character prompt books with T2I models to support consistent multi-turn image generation. Its pipeline manages character identities, reference images, layouts, and guidance throughout generation.

3 Method

  • The overall framework uses an LLM-based character designer, character image manager, and character-guided generator to produce each turn's final image.The three stages generate prompt books, on-stage character images with guidance, and the final synthesized image without training.Source: §3 · pp. 5–6

3.1 LLM-based Character Designer

  • Unique character IDs, prompts, and bounding boxes encode user intent, track characters across turns, and organize their spatial relationships.The prompt book also includes background and negative prompts, while layouts provide spatial priors for multi-character synthesis.Source: §3.1 · p. 6

3.2 Character Image Manager

  • Reference images preserve a character’s contextual appearance, while on-stage images represent the character in the current interaction turn.Existing reference images are injected through an adapter; first-time characters are generated directly and then retained as references.Source: §3.2 · pp. 6–7
  • The character image manager segments on-stage characters, rearranges them according to the prompt-book layouts, and forms a blank-background middle-state image.This middle-state image consolidates the characters before guidance extraction.Source: §3.2 · p. 7, Figure 3 · p. 7
  • Latent guidance is extracted through forward diffusion, while lineart guidance supplies stronger positional and structural constraints for subsequent generation.Latent guidance preserves on-stage characteristics through a shared latent space; lineart guidance helps retain edges and details.Source: §3.2 · pp. 7–8

3.3 Character-Guided Generator

  • The final generator injects global prompts and lineart guidance into denoising, then applies latent guidance within character masks during selected steps.The ratio r controls how many denoising steps receive latent guidance, balancing character integration with the background.Source: §3.3 · p. 8, Lineart guidance · pp. 8–9

4 Experiments

The experiments evaluate TheaterGen on CMIGBench across consistency, alignment, human ratings, ablations, and visual examples. TheaterGen outperforms comparative methods while its prompt-book and guidance components contribute to consistency and editing performance.

4.1 Setup

  • CMIGBench contains 8,000 four-turn dialogues split evenly between story generation and multi-turn editing, enabling zero-shot evaluation without predefined characters.The benchmark is automatically generated through GPT-4 and manually curated; editing dialogues fix spatial, attribute, negative-prompt, and numeracy edits.Source: §4.1 · p. 9

4.2 Comparison with SOTA methods

  • Across story generation and multi-turn editing, TheaterGen surpasses comparative models on all reported metrics, improving aFID by 27%, aCCS by 21%, and aTIS by 19% over Mini DALLE·3 with diffusion XL.The results cover semantic and contextual consistency on CMIGBench.Source: §4.2 · pp. 10–11, Table 2 · p. 10, Table 3 · p. 11
  • TheaterGen outperforms baselines on completion for every editing type, with particularly strong gains for spatial relationships and quantities attributed mainly to the LLM’s layout capability.Alignment is evaluated for spatial, attribute, negative, and numeracy edits.Source: Evaluation metrics · p. 10, §4.2 · p. 11, Table 3 · p. 11

4.3 Human evaluation

  • Human ratings are higher for TheaterGen than for other methods, and machine metrics show strong correlation with human evaluations.The user study sampled 50 dialogues and used ratings from 10 volunteers; aFID is expected to correlate negatively, while other metrics are expected to correlate positively.Source: §4.3 · p. 12, Table 4 · p. 12

4.4 Ablation experiment

  • Removing latent guidance makes characters inconsistent with their reference images, while removing lineart guidance significantly decreases all metrics and harms character positions, quantities, and character-background interactions.The two guidance types therefore complement each other in maintaining contextual consistency and controlling layout and detail.Source: Latent guidance · pp. 12–13, Table 5 · p. 12, Table 6 · p. 12
  • Among evaluated LLMs, GPT-4 achieves the best average prompt-book performance, while the tested models generally assign character IDs and generate bounding boxes consistent with captions.The evaluation measures character-ID accuracy and bounding-box accuracy.Source: LLM Character Designer · p. 13, Table 7 · p. 13, Table 8 · p. 13

4.5 Visualization

  • Visualizations show that TheaterGen maintains semantic and contextual consistency while handling complex references and preserving discernible main-character features.Additional qualitative results are reported in the appendix.Source: §4.5 · p. 13

5 Discussion and Conclusion

TheaterGen combines prompt-book management, character-specific image generation, and guidance information to address semantic and contextual consistency in multi-turn image generation, while acknowledging remaining limitations.

5 Discussion and Conclusion

  • TheaterGen manages character prompts and layouts, then uses character images and lineart guidance to generate the final multi-turn image.The framework organizes generation as a Screenwriter-managed prompt book, character-image Rehearsal, and Final Performance.Source: §5 · pp. 13–14
  • The framework’s visualizations cover both story generation and multi-turn editing.Source: Figure 5 · p. 14, Figure 6 · p. 15
  • Randomness in pretrained image adapters can cause contextual inconsistency, while limited T2I generation capability can cause semantic inconsistency.The authors identify more powerful T2I models as a future direction for addressing these issues.Source: §5 · p. 15

A Prompt book Generation Details

The prompt-book generation procedure asks an LLM to convert human instructions and dialogue history into structured character layouts, background prompts, and negative prompts.

A Prompt book Generation Details

B.1 Models Input

The experiments use prompt-book-based inputs for TheaterGen and adapted dialogue inputs for the baselines, with TheaterGen requiring more inference time because it generates character images at each turn.

B.1 Models Input

  • TheaterGen directly uses the prompt book, while MiniGPT-5, SEED-LLaMA, and Mini DALLE·3 receive inputs adapted to their respective designs.Source: §B.1 · p. 17

B.2 Inference Time

  • TheaterGen requires about 35–40 seconds per four-turn dialogue with SD1.5, compared with 10–15 seconds for the baselines.With SDXL, Mini DALLE·3 takes approximately 36 seconds and TheaterGen around 86 seconds.Source: §B.2 · p. 17

C Dataset Construction Details

The dataset construction samples diverse characters and scenes, then generates four-turn story or editing dialogues together with ground-truth prompt books and layouts.

C.1 Character Pools

  • Character pools contain 50 fruits, 100 objects, 7 humans, and 50 scenes to provide diversity.Source: §C.1 · p. 17

C.2 Prompt for Story Generation

  • Story-generation dialogues are created from sampled human or animal protagonists and backgrounds, with four natural-language sentences plus bounding-box layouts.Source: §C.2 · pp. 18–19

C.3 Prompt for Multi-turn Editing

  • Multi-turn editing dialogues sample objects or fruits and require spatial, attribute, negative, and numeracy edit rounds.The prompts enforce consistent object identifiers and specify sampled quantities and spatial relations.Source: §C.3 · pp. 19–20

C.4 Automated & Human Corrections

CMIGBench dialogues undergo automated format, character-placement, and layout-overlap corrections, followed by manual visualization and adjustment. These steps refine generated content while preserving correct dialogue content.

  • GPT-4-generated dialogues are validated and corrected to produce the final CMIGBench.Source: §C.4 · p. 20
  • Regular expressions repair formatting discrepancies, including text-versus-list distinctions, multilingual punctuation, and unenclosed brackets.Source: CMIGBench · p. 20
  • Characters mistakenly placed in the background are detected and regenerated by GPT-4.Source: CMIGBench · p. 20
  • Layouts exceeding the 25% overlap threshold are dispersed iteratively, with GPT-4 regeneration used if overlap remains excessive.The procedure computes a collective bounding rectangle and moves elements outward using random distances.Source: CMIGBench · p. 21
  • A manual visualization and drag-and-drop toolkit enables dialogue verification and layout refinement for improved dataset accuracy and presentation quality.Source: CMIGBench · p. 21

C.5 Benchmark Examples

CMIGBench examples encode multi-turn story generation and editing as structured dialogues with captions, objects, layouts, backgrounds, and negatives. The examples show both evolving narratives and cumulative object modifications.

Story Generation

  • Story-generation dialogues specify characters, scenes, captions, object bounding boxes, backgrounds, and negative prompts across turns.Source: §C.5 · pp. 21–22
  • The story example tracks a sparrow, lion, and eagle through successive library scenes while preserving their identities and changing descriptions and positions.Source: §C.5 · pp. 21–22

Multi-Turn Editing

  • Multi-turn editing examples represent transformations such as changing a pen to blue, removing it, and replicating the remaining spatula four times.Source: §C.5 · pp. 22–23

D More Visualization Results

Additional visualizations cover multi-turn editing, story generation, and comparisons between TheaterGen’s on-stage character images and final images. The reported comparisons emphasize character-feature preservation alongside foreground-background integration.

Multi-Turn Editing

Story Generation

Loading 2404.18919v2…