PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Consistent Text-to-Image Generation

Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, Yuval Atzmon

arXiv:2402.03286v3cs.CVcs.AIcs.GRcs.LG

TL;DR

Consistent subject generation across diverse prompts is challenging because existing methods require per-subject optimization or image-conditioning infrastructure and can struggle with prompt alignment and multiple subjects. ConsiStory shares internal diffusion activations through subject-driven attention and correspondence-based feature injection, while preserving layout diversity. It achieves state-of-the-art subject consistency and text alignment without optimization, with extensions to multi-subject scenes and training-free common-object personalization.

  • Problem

    Existing consistent text-to-image methods require per-subject training or substantial pretraining and can struggle with prompt alignment and multiple consistent subjects.

  • Method

    ConsiStory shares pretrained diffusion activations through subject-driven attention, correspondence-based feature injection, and layout-diversity strategies.

  • Results

    ConsiStory achieves state-of-the-art subject consistency and text alignment without a single optimization step, while operating roughly 20× faster than the current state of the art.

  • Takeaways & Limitations

    The approach extends naturally to multi-subject scenes and enables training-free personalization for common objects without tuning the pretrained model.

  • Takeaways & Limitations

    The method can fail to localize objects, especially in unusual styles, and struggles to separate appearance from style, limiting consistent generations to images sharing the same style.

Abstract

from arXiv · show

Text-to-image models offer a new level of creative flexibility by allowing users to guide the image generation process through natural language. However, using these models to consistently portray the same subject across diverse prompts remains challenging. Existing approaches fine-tune the model to teach it new words that describe specific user-provided subjects or add image conditioning to the model. These methods require lengthy per-subject optimization or large-scale pre-training. Moreover, they struggle to align generated images with text prompts and face difficulties in portraying multiple subjects. Here, we present ConsiStory, a training-free approach that enables consistent subject generation by sharing the internal activations of the pretrained model. We introduce a subject-driven shared attention block and correspondence-based feature injection to promote subject consistency between images. Additionally, we develop strategies to encourage layout diversity while maintaining subject consistency. We compare ConsiStory to a range of baselines, and demonstrate state-of-the-art performance on subject consistency and text alignment, without requiring a single optimization step. Finally, ConsiStory can naturally extend to multi-subject scenarios, and even enable training-free personalization for common objects.

1 INTRODUCTION

Consistent subject generation across diverse prompts is difficult because existing personalization and image-conditioned methods require substantial optimization or pretraining and can compromise prompt alignment or multi-subject portrayal. ConsiStory addresses this with training-free activation sharing, layout-diversity strategies, and extensions to multi-subject generation and common-object personalization.

  • Existing personalization methods require per-subject training and can struggle with multiple subjects and prompt alignment, while image-conditioned alternatives require significant computation.
  • ConsiStory shares pretrained diffusion-model activations to align generated images and uses attention and feature-alignment mechanisms for subject consistency.Its process localizes subjects, shares subject patches through extended self-attention, and aligns output features across corresponding subject pixels.
  • The method mitigates reduced layout diversity caused by cross-image attention through vanilla-sampling features and inference-time dropout on shared keys and values.
  • ConsiStory extends to multi-subject scenes, common-object personalization, and compatibility with editing tools such as ControlNet.

2 RELATED WORK

Prior consistent text-to-image methods commonly use fine-tuning, personalized models, image conditioning, or extended attention. ConsiStory instead generates consistent images from text alone by applying attention sharing and diffusion-feature correspondences at the subject level.

  • Consistent text-to-image generation synthesizes image sets whose subjects remain visually consistent across images.
  • Earlier approaches rely on fine-tuning, personalized LoRA models, storyboard data, image conditioning, or iterative identity extraction.
  • ConsiStory does not tune or personalize the pretrained text-to-image model and generates consistent images from text prompts alone.
  • The method applies attention-sharing ideas to subject-level consistency across novel images rather than using existing images or aligning entire frames.
  • ConsiStory uses diffusion-based DIFT maps to share features during denoising instead of performing appearance transfer after generation.

3 PRELIMINARIES: SELF-ATTENTION IN T2I MODELS

Self-attention transforms image-patch features into queries, keys, and values, computes patch-to-patch relevance, and combines weighted value features into output activations. ConsiStory modifies this mechanism so images in a generated batch can attend to one another.

  • A self-attention layer projects image-patch tokens into Keys, Values, and Queries using separate matrices.
  • For image i, input features x_i are projected as K_i = x_i·W_K, V_i = x_i·W_V, and Q_i = x_i·W_Q before forming the self-attention map.
  • The attention map scores patch relevance and weights Value features to produce hidden features, which are output-projected and added to the input for the next layer.
  • ConsiStory allows generated images in the same batch to attend to one another and be influenced by each other’s output activations.

4 METHOD

ConsiStory aligns subject representations across independently generated images during inference, while using feature injection and diversity controls to preserve identity, prompt alignment, and varied layouts.

  • 4.1 Subject-driven self-attention: ConsiStory replaces standard decoder self-attention with Subject-Driven Self-Attention, allowing each image to attend to its own patches and subject patches from other images.Subject masks are derived from cross-attention maps and restrict cross-image sharing to subject regions.
  • 4.2 Enriching layout diversity: SDSA can make image layouts overly similar, so ConsiStory blends vanilla query features and applies attention dropout to recover pose and layout diversity.The two mechanisms weaken shared attention or import structural variation from non-consistent sampling while preserving consistency.
  • 4.2 Enriching layout diversity: The architecture localizes subjects, shares masked cross-image attention, injects corresponding features, and uses dropout plus vanilla-query blending to balance consistency and diversity.Figure 2 summarizes the generation pipeline and its two diversity controls.
  • 4.3 Feature injection: The method combines attention-based subject sharing with correspondence-based Feature Injection to refine fine visual details and subject identity across images.DIFT patch correspondences identify matching regions, whose self-attention features are then blended across the batch.
  • 4.4 Experimental results: Qualitative comparisons show that ConsiStory follows prompts while maintaining subject consistency, unlike baselines that sacrifice one of these properties.The comparison includes IP-Adapter, TI, and DB-LORA.
  • 4.4 Anchor images and reusable subjects: Anchor images can be reused with changed non-anchor prompts, enabling new scenes and unlimited consistent image generation while reducing extended-attention cost.Restricting attention to a small number of anchors also reduces VRAM requirements and can improve quality in large batches.

5 EXPERIMENTS

ConsiStory is evaluated against personalization and encoder-based baselines for subject consistency, prompt alignment, diversity, multi-subject generation, runtime, and extended applications. Across these evaluations, it maintains strong consistency and text alignment without optimization, while supporting varied layouts and multiple subjects.

  • 5.2 Qualitative Results: ConsiStory preserves subject identity while varying seeds, facial attributes, styles, settings, and other prompt-controlled content.Qualitative examples include different consistent image sets from different noise inputs and identity preservation across expression, hair color, and tattoos.
  • 5.2 Qualitative Results: ConsiStory generates multiple consistent subjects, whereas LORA-DB may neglect one or both subjects when combining independently personalized models.The method instead builds on the diffusion model’s compositional ability.
  • 5.3 Quantitative evaluation: ConsiStory achieves a Pareto-front trade-off between subject consistency and prompt alignment, matching SDXL in text-alignment scores.The evaluation uses DreamSim for consistency and CLIP for prompt alignment; error bars represent standard error of the mean.
  • 5.3 Quantitative evaluation: User studies show that participants typically prefer ConsiStory for both subject consistency and textual alignment over the compared baselines.The study collected 3,000 two-alternative forced-choice responses across the two question types.
  • 5.3 Quantitative evaluation: ConsiStory reaches a 32-second time-to-consistent-subject, reported as 25× faster than Avrahami et al. and 8–14× faster than LORA-DB and TI.The measurement uses an H100 GPU and includes generating two anchors plus an image for a new prompt.
  • 5.4 Ablation study: Removing SDSA harms shape and texture consistency, removing feature injection weakens identity accuracy, and removing query blending or dropout reduces layout diversity.The component ablation attributes different consistency and diversity effects to the method’s principal components.
  • 5.4 Ablation study: Feature injection generally improves consistency and can handle occluded objects, although noisy DIFT correspondences arise when a subject has substantially different semantic forms.The method’s thresholding mechanism drops patches with low similarity scores.
  • 5.5 Extended Applications: ConsiStory supports ControlNet pose guidance and training-free personalization of common subject classes without tuning or an encoder.The personalization application uses inverted images as anchors, but struggles with complex objects and style-changing prompts.

6 LIMITATIONS

The method’s limitations concern localization reliability, appearance–style entanglement, and biases inherited from the underlying SDXL model. These constraints limit some unusual-style, cross-style, and demographic-generation settings.

  • 6 LIMITATIONS: Object localization through cross-attention can fail occasionally, especially for unusual styles, although the authors report fewer than 5% failures that can be resolved by changing the seed.The method is therefore not uniformly reliable across unusual visual styles.
  • 6 LIMITATIONS: Appearance and style remain entangled, limiting consistent generations to images that share the same style.The authors identify separation of appearance from style as a central scope boundary.
  • 6 LIMITATIONS: The underlying SDXL model may exhibit biases toward certain groups, though specifying gender or ethnicity modifiers can significantly reduce them.This limitation is inherited from the base model rather than attributed specifically to ConsiStory’s consistency mechanism.

7 CONCLUSIONS

ConsiStory is a training-free method that improves prompt alignment while creating visually consistent subjects, and extends to multi-subject scenarios and common-object personalization. Its limitations include difficulty with mixed styles, dependence on cross-attention localization, and inherited biases from the underlying model.

  • 7 CONCLUSIONS: ConsiStory is 20× faster than state-of-the-art methods and better preserves alignment with the given prompts.The method creates visually consistent subjects using a pretrained text-to-image diffusion model without per-subject optimization.
  • 7 CONCLUSIONS: ConsiStory often struggles with different styles within the same image set and depends on cross-attention quality for subject localization.
  • 7 CONCLUSIONS: The method inherits biases of the underlying SDXL model toward certain ethnic groups, although prompting diverse groups can produce consistent subjects.
  • 7 CONCLUSIONS: The method extends to multi-subject scenarios and training-free personalization for common objects.
  • 7 CONCLUSIONS: The authors hope the results support consistent generation for creative applications and motivate further training-free alternatives to personalization.

A ADDITIONAL RESULTS

The appendix adds qualitative comparisons, quantitative ablations, and measurements spanning text similarity, subject consistency, and layout diversity.

  • A ADDITIONAL RESULTS: Additional experiments compare single- and multi-subject outputs with existing baselines and ablate text similarity, subject consistency, and layout diversity.

B ADDITIONAL IMPLEMENTATION DETAILS

The implementation extracts subject masks from cross-attention, injects correspondence-based features during selected denoising steps, and applies attention-based controls to preserve subjects and vary poses. Additional settings specify diffusion sampling, correspondence construction, and feature-selection thresholds.

  • B ADDITIONAL IMPLEMENTATION DETAILS: Feature injection runs for timesteps 680–900 with α=0.8 and retains only patches whose similarity exceeds an automatically selected Otsu threshold.
  • B ADDITIONAL IMPLEMENTATION DETAILS: DIFT features are computed after denoising from t=1000 to t=261, then guide a second denoising pass from t=1000 to t=0.
  • B ADDITIONAL IMPLEMENTATION DETAILS: Pose variation injects Vanilla Query Features during the first five denoising steps while ν_t decays from 0.9 to 0.8 and self-attention dropout is 0.5.
  • B ADDITIONAL IMPLEMENTATION DETAILS: Images use 50 DDIM steps with guidance scale 5, while ControlNet pose conditioning raises self-attention dropout to 0.7.
  • B ADDITIONAL IMPLEMENTATION DETAILS: Subject masks average cross-attention maps for each subject token across prior diffusion steps and 32 × 32 layers, then threshold them with Otsu’s method.The mask identifies noisy latent patches likely to contain each subject.
  • B ADDITIONAL IMPLEMENTATION DETAILS: Patch correspondences use cosine similarity between source and target DIFT feature vectors to identify matching patches.For target patch p_t, the correspondence map gives the index of the most similar source patch.

C DIVERSITY EVALUATIONS

The paper evaluates layout diversity using Vanilla Query Injection and Self-Attention Dropout, with a geometric metric based on displacements between corresponding points. Lower displacement indicates more aligned layouts and therefore less diversity across images.

  • C DIVERSITY EVALUATIONS: Layout diversity is assessed through Vanilla Query Injection and Self-Attention Dropout, alongside an automatic evaluation metric.
  • C DIVERSITY EVALUATIONS: The metric computes average displacement between corresponding points across image pairs using DIFT features and dense correspondence maps.
  • C DIVERSITY EVALUATIONS: Low displacement indicates aligned subject layouts across images, and persistent low displacement signals limited diversity.

D PROMPTS DATASET DETAILS

The evaluation uses a 100-set, 500-image prompt dataset spanning subject categories, description levels, and styles, alongside qualitative comparisons and component ablations.

  • D PROMPTS DATASET DETAILS: 100 prompt sets produced 500 images across humans, animals, fantasy, and inanimate subjects, with varied description levels and styles.Each set contains five prompts sharing a recurring subject and includes metadata such as subject token and description.
  • D PROMPTS DATASET DETAILS: Additional comparisons show ConsiStory jointly follows prompts and preserves consistency, unlike baselines that sacrifice one objective or the other.The comparisons include IP-Adapter, TI, ELITE, DB-LoRA, and a concurrent multi-subject method.
  • D PROMPTS DATASET DETAILS: The prompt dataset is released as a YAML file in the supplemental materials.
  • D PROMPTS DATASET DETAILS: Removing SDSA or feature injection reduces subject consistency, whereas removing dropout or query injection decreases textual similarity.These ablations isolate consistency and textual-alignment effects of the method’s components.
  • D PROMPTS DATASET DETAILS: ConsiStory achieves the highest Diversity Score when both dropout and query injection are retained.The layout-diversity ablation compares variants that omit either component or both.

E TRAINING-FREE PERSONALIZATION DETAILS

The personalization extension adapts the method to inverted real-image anchors and evaluates consistency and textual alignment through two human preference studies. Its supported personalization scope is limited to simple objects.

  • E TRAINING-FREE PERSONALIZATION DETAILS: Training-free personalization uses two inverted real images as anchors, with anchors sharing attention maps only with generated images.This modification avoids attention sharing among inverted anchors, which is unsuitable for the anchoring setup.
  • E TRAINING-FREE PERSONALIZATION DETAILS: The personalization process uses DDPM scheduling for 100 generation steps and a guidance scale of 5.0 for generated images.The inversion procedure uses DDPM-based scheduling, while the generated images use the default guidance scale.
  • E TRAINING-FREE PERSONALIZATION DETAILS: Training-free personalization works only for simple objects and has documented failure cases for non-common subjects.
  • E TRAINING-FREE PERSONALIZATION DETAILS: Two Amazon Mechanical Turk studies separately measure visual consistency and textual alignment using two-alternative forced choice judgments.Raters compare image sets for identity consistency or individual images for match to a text description.
  • E TRAINING-FREE PERSONALIZATION DETAILS: The visual-consistency study used 100 trials, 100 unique subjects, five prompt settings per subject, and five repeated ratings per trial.Raters judged subject identity while disregarding background, clothing, and pose.
  • E TRAINING-FREE PERSONALIZATION DETAILS: The textual-alignment study used 500 trials across 100 unique subjects, with five distinct prompt settings for each subject.Raters selected which image better matched the supplied textual description.
Loading 2402.03286v3…