PaperlayerView exact revision on arXiv

Source-linked AI summary

One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt

Tao Liu, Kai Wang, Senmao Li, Joost van de Weijer, Fahad Shahbaz Khan, Shiqi Yang, Yaxing Wang, Jian Yang, Ming-Ming Cheng

arXiv:2501.13554v3cs.CVcs.AIcs.LG

TL;DR

Consistent identity-preserving T2I generation for storytelling remains difficult, while existing solutions often require extensive training or architectural modifications. 1Prompt1Story concatenates frame descriptions into one prompt and refines generation with Singular-Value Reweighting and Identity-Preserving Cross-Attention. Experiments on ConsiStory+ report effective identity preservation across lengthy narratives, with stated limitations around its singular-value assumption and current-method constraints.

  • Problem

    Existing T2I models struggle to maintain subject identity across diverse scenes, while many approaches require extensive training or architectural changes.

  • Method

    1Prompt1Story is a training-free method that consolidates identity and frame prompts, then applies Singular-Value Reweighting and Identity-Preserving Cross-Attention.

  • Results

    1Prompt1Story demonstrates effective identity consistency throughout lengthy narratives on the extended ConsiStory+ benchmark.

  • Takeaways & Limitations

    The method offers a single-prompt route to consistent character generation across diverse story scenes and different T2I model backbones.

  • Takeaways & Limitations

    The method assumes the main singular values of X exp correspond to the fundamental information of Pexp, while the paper notes limitations in current methods.

Abstract

from arXiv · show

Text-to-image generation models can create high-quality images from input prompts. However, they struggle to support the consistent generation of identity-preserving requirements for storytelling. Existing approaches to this problem typically require extensive training in large datasets or additional modifications to the original model architectures. This limits their applicability across different domains and diverse diffusion model configurations. In this paper, we first observe the inherent capability of language models, coined context consistency, to comprehend identity through context with a single prompt. Drawing inspiration from the inherent context consistency, we propose a novel training-free method for consistent text-to-image (T2I) generation, termed "One-Prompt-One-Story" (1Prompt1Story). Our approach 1Prompt1Story concatenates all prompts into a single input for T2I diffusion models, initially preserving character identities. We then refine the generation process using two novel techniques: Singular-Value Reweighting and Identity-Preserving Cross-Attention, ensuring better alignment with the input description for each frame. In our experiments, we compare our method against various existing consistent T2I generation approaches to demonstrate its effectiveness through quantitative metrics and qualitative assessments. Code is available at https://github.com/byliutao/1Prompt1Story.

1 INTRODUCTION

Consistent text-to-image generation is important for narrative applications but remains difficult because identity can drift across scenes. 1Prompt1Story uses context consistency in language models, a single consolidated prompt, and two refinement techniques to preserve identity and prompt alignment.

  • Consistent T2I generation is important for animation, storytelling, video generation, and other narrative-driven visual applications.
  • Existing methods often require large-dataset training, mapping encoders, fine-tuning, extensive memory, or complex modules.
  • Language models implicitly maintain identity across multiple descriptions connected by context, a property termed context consistency.
  • 1Prompt1Story consolidates an identity prompt and frame prompts into one longer prompt for training-free consistent T2I generation.
  • Singular-Value Reweighting and Identity-Preserving Cross-Attention improve text-image alignment and subject consistency while allowing frame prompts to remain individually expressed.
  • Extensive comparisons on the extended ConsiStory+ benchmark report consistent identity throughout lengthy narratives.

2 RELATED WORK

Prior work addresses consistent T2I generation through personalization, storytelling, and training-free approaches, but often sacrifices efficiency, compatibility, fidelity, or subject breadth. 1Prompt1Story instead exploits context consistency without fine-tuning large models or designing complex modules.

  • T2I personalized generation: T2I personalization adapts a model to a new concept using a few images and a unique token, typically through extensive training.
  • Consistent T2I generation: Many consistent T2I methods train model parameters, reducing compatibility with pretrained community models or face fidelity.
  • Storytelling: Face-focused systems can be limited on non-human subjects, while storytelling requires support for varied or multiple subjects.
  • Storytelling: Storytelling approaches commonly require intensive training on storytelling datasets and differ from consistent T2I generation in their subject requirements.
  • Consistent T2I generation: 1Prompt1Story uses language-model context consistency instead of fine-tuning large models or designing complex modules, supporting compatibility with different T2I backbones.

3 METHOD

The method consolidates identity and frame prompts into one prompt, exploiting tighter contextual relationships in text embeddings to preserve identity across generated frames. It then reweights prompt semantics and cross-attention features to improve frame-specific alignment while retaining consistency.

  • Prompt Consolidation: 1Prompt1Story concatenates the identity prompt and all frame prompts into a single input, unlike multi-prompt generation that processes each frame separately.The single-prompt configuration is [P0; P1; ...; PN], whereas multi-prompt generation uses [P0; Pi] for each frame.
  • Context Consistency: 46.42 versus 71.25 average L2 distance shows more compact frame-prompt embeddings for single-prompt than multi-prompt generation.The distances are measured after extracting frame-prompt embeddings and applying t-SNE visualization.
  • Naive Prompt Reweighting: Naive Prompt Reweighting magnifies the desired frame embedding and reduces the others, producing shared subject identity but blending backgrounds across frames.The reweighted consolidated embedding is supplied to the T2I model for each target frame.
  • Evaluation: Across qualitative and quantitative analyses, 1Prompt1Story produces more consistent identities and better frame-specific alignment than SDXL and Naive Prompt Reweighting.The complete method also outperforms the comparison methods in identity consistency on the extended ConsiStory+ benchmark.
  • Singular-Value Reweighting: Singular-Value Reweighting enhances the target frame and suppresses remaining frame semantics, using SVD on express and suppress token embeddings.The express set contains the target frame and [EOT], while the suppress set contains the other frames and [EOT].
  • Identity-Preserving Cross-Attention: Identity-Preserving Cross-Attention concatenates filtered identity keys and values with current features to strengthen identity similarity after semantic reweighting.The method addresses the observed reduction in context consistency caused by Singular-Value Reweighting.

4 EXPERIMENTS

Experiments evaluate 1Prompt1Story against consistent T2I methods on the ConsiStory+ benchmark using prompt-alignment, identity-consistency, qualitative, user-study, and ablation analyses. The method achieves strong balance between prompt alignment and identity consistency, while both components contribute to its performance.

  • Experimental Setup: ConsiStory+ extends the original benchmark with broader subjects, descriptions, and styles, evaluating prompt alignment and subject consistency across up to 1,500 images from 200 prompt sets.
  • Experimental Setup: Prompt alignment is measured with CLIP-T, while identity consistency uses CLIP-I and DreamSim after background removal to focus similarity measurements on subject identities.
  • Qualitative Comparison: Qualitatively, 1Prompt1Story balances identity preservation, frame-description accuracy, and pose diversity, whereas competing methods show identity inconsistency, repetitive poses, or duplicated backgrounds.
  • Quantitative Comparison: 1Prompt1Story ranks first among training-free methods across evaluation metrics and second when training-required methods are included.
  • User Study: A user study found that 1Prompt1Story aligned best with human preferences for identity consistency, prompt alignment, and image diversity.
  • Ablation Study: Combining SVR+ and SVR- reduces frame-prompt blending, while adding IPCA further improves consistency and yields the best ablation performance.
  • Additional Applications: 1Prompt1Story also integrates with ControlNet for human-pose control and with PhotoMaker to improve resemblance to real identities.

5 CONCLUSION

The conclusion presents 1Prompt1Story as a single-prompt approach for maintaining identity across diverse T2I scenes. It attributes the method’s performance to context consistency together with Singular-Value Reweighting and Identity-Preserving Cross-Attention.

  • 1Prompt1Story uses natural-language context consistency and a single extended prompt to maintain consistent identity across diverse scenes.
  • Singular-Value Reweighting refines frame descriptions, while Identity-Preserving Cross-Attention strengthens consistency at the attention level.
  • Experiments on ConsiStory+ demonstrate superiority over state-of-the-art techniques for coherent, narrative-consistent visual output.

A BOARDER IMPACTS AND LIMITATIONS

The paper positions 1Prompt1Story as a time- and resource-saving intermediary for customizing consistent characters across story scenarios. Its use requires advance knowledge of all prompts and is constrained by text-encoder capacity, with sliding-window generation potentially causing identity drift.

  • Boarder Impacts: 1Prompt1Story can help users customize desired characters across different story scenarios while saving time and resources.
  • Limitations: The method requires all prompts to be known in advance and limits input length to the maximum capacity of the text encoder.
  • Limitations: Although sliding windows facilitate infinite-length story generation, generated identities may gradually diverge and become less consistent.

B IMPLEMENTATION DETAILS

The method performs subject-consistent generation at inference time by modifying text embeddings and cross-attention without training or optimization. Its implementation uses SDXL’s two text encoders and reweights frame-prompt embeddings.

  • The primary base model is pretrained Stable Diffusion XL, which uses CLIP L/14 and OpenCLIP bigG/14 text encoders.The method separately updates the text embeddings produced by each encoder.
  • Naive Prompt Reweighting multiplies the expressed frame prompt embedding by 2 and suppressed frame-prompt embeddings by 0.5, while keeping cEOT unchanged.

B.2 BENCHMARK DETAILS

The paper evaluates 1Prompt1Story on the extended ConsiStory+ benchmark and describes its inference pipeline from text embeddings to subject-consistent frame images. The algorithm applies Singular-Value Reweighting and Identity-Preserving Cross-Attention before decoding outputs.

  • Benchmark details: ConsiStory+ expands the original benchmark from 100 prompt sets across 4 superclasses to 200 sets across 8 superclasses.Each expanded set contains between 5 and 10 prompts covering humans, animals, fantasy, inanimate, fairy tales, nature, technology, and foods.
  • Extensions: 1Prompt1Story integrates with ControlNet for spatial control and with PhotoMaker for real-image personalization with improved identity consistency.
  • Algorithm: Algorithm 1 takes text embeddings C and a latent vector zt, then returns subject-consistent images I1 through IN.
  • Algorithm: The pipeline first applies Singular-Value Reweighting to the consolidated embeddings.
  • Algorithm: It then applies Identity-Preserving Cross-Attention during reverse diffusion before decoding each final latent into an image.

B.3 COMPARISON METHOD IMPLEMENTATIONS

The comparison evaluates methods primarily on Stable Diffusion XL using their available implementations and default DDIM settings, while also testing prompt-order robustness. The method maintains identity consistency across reordered frame prompts.

  • Comparison setup: Comparisons use Stable Diffusion XL for all methods except BLIP-Diffusion, which uses Stable Diffusion v1.5.DDIM steps follow each method’s default open-source setting.
  • Compared methods: The evaluation includes unofficial implementations of Textual Inversion and The Chosen One, plus official implementations of IP-Adapter and PhotoMaker.
  • Additional evaluation: Prompt sets are expanded to random lengths between 5 and 10, and new prompt sets are generated within the fairy-tales superclass.
  • Compared methods: It also includes official implementations of BLIP-Diffusion and StoryDiffusion, while ConsiStory is reimplemented because it is not open-source.
  • Prompt-order robustness: Across six orderings of three frame prompts, 1Prompt1Story generates images with identity consistency using the same fox identity prompt and seed.

C.2 Singular-Value Reweighting ANALYSIS

The analysis studies Singular-Value Reweighting’s ordering, suppression strategy, and identity enhancement. Iterative suppression better removes irrelevant frame prompts, while stronger identity enhancement trades pose and background diversity for consistency.

  • SVR components: SVR+ enhances the expressed frame prompt, whereas SVR- iteratively weakens prompts intended for suppression.
  • Ordering: Applying SVR- before SVR+ produces results similar to applying SVR+ before SVR-.
  • Suppression strategy: Iterative Suppress more effectively minimizes other frame prompts’ influence than Joint Suppress.Joint Suppress can produce images with similar backgrounds because it does not adequately eliminate all suppressed-frame descriptions.
  • Identity enhancement: Enhancing the identity prompt alongside the current frame prompt improves identity consistency but makes object poses and backgrounds more similar across images.

C.3 Naive Prompt Reweighting ABLATION STUDY

The ablation shows that naive prompt reweighting allows semantic interference across frames, whereas the proposed method suppresses irrelevant information and improves frame-specific alignment. Dynamic cEOT updating also prevents background blending, while seed changes preserve identity across diverse outputs.

  • C.3 Naive Prompt Reweighting ABLATION STUDY: The ablation evaluates Naive Prompt Reweighting against the proposed method for identity preservation and prompt alignment.NPR+ scales the current frame embedding by 2, while NPR- scales other frame embeddings by 0.5.
  • C.3 Naive Prompt Reweighting ABLATION STUDY: NPR+, NPR-, and NPR allow semantic interference from other frame prompts, unlike the proposed method.The puppy appears in earlier frames despite being specified only for the third frame under naive reweighting.
  • C.3 Naive Prompt Reweighting ABLATION STUDY: Keeping cEOT fixed causes background blending across images generated from different frame prompts.The ablation identifies dynamic cEOT updating as important for avoiding this blending.
  • C.3 Naive Prompt Reweighting ABLATION STUDY: The proposed method removes irrelevant semantic information from other frame descriptions, producing images more aligned with their corresponding frame prompts.
  • C.4 SEED VARIETY: Changing the seed produces diverse characters and backgrounds while maintaining strong identity consistency and prompt alignment.The method preserves the diffusion model’s ability to generate varied identities and backgrounds because it does not modify the model’s original parameters.
  • C.4 SEED VARIETY: The seed-variation results indicate that identity consistency can be retained across outputs generated with different initial noise.

D ADDITIONAL RESULTS OF OUR METHOD 1Prompt1Story

Additional experiments examine multi-subject and long-story generation, model portability, evaluation criteria, and comparisons with existing methods. The results show broader applicability while identifying a constraint in multi-subject generation.

  • D.1 CONSISTENT STORY GENERATION WITH MULTIPLE SUBJECTS.: Specifying multiple subjects in the identity prompt generates stories whose characters maintain consistent identities.All generated images include every character referenced in the identity prompt, limiting flexibility.
  • D.2 LONG STORY GENERATION: A sliding-window technique overcomes diffusion-model input-length limits to support stories of any length.The method uses a window of frame prompts together with the identity prompt.
  • D ADDITIONAL RESULTS OF OUR METHOD 1Prompt1Story: The additional evaluation compares prompt alignment using DSG and VQAScore, which assess image-text consistency through questions and answers.
  • D ADDITIONAL RESULTS OF OUR METHOD 1Prompt1Story: 1Prompt1Story achieves the second-best FID result in the additional comparison, behind Naive Prompt Reweighting.The FID setup treats base-model images as real data and method-generated images as fake data.
  • D ADDITIONAL RESULTS OF OUR METHOD 1Prompt1Story: The user study asks participants to select the best image set across identity consistency, prompt alignment, and image diversity.It compares IP-Adapter, Consistory, Story Diffusion, and the proposed method on 30 prompt sets with four frames each.
Loading 2501.13554v3…