Source-linked AI summary
StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, Qibin Hou
TL;DR
StoryDiffusion addresses the difficulty of maintaining subject consistency across images and videos that narrate a text-based story. It introduces zero-shot Consistent Self-Attention for image generation and Semantic Motion Predictor for semantic-space transitions, producing consistent story images and smoother, more stable videos. The authors report strong consistency and text controllability, while noting limitations for minor clothing details and very long videos.
Problem
Maintaining consistent subjects and complex details across generated story images and videos remains challenging for diffusion models.
Method
StoryDiffusion combines zero-shot Consistent Self-Attention for subject-consistent images with Semantic Motion Predictor for semantic-space transitions between images.
Results
StoryDiffusion generates subject-consistent images with strong text controllability and smoother, more stable transition videos, including transitions with large motion.
Takeaways & Limitations
The framework supports text-based storytelling with consistent image sequences and videos by combining training-free attention with semantic-space motion prediction.
Takeaways & Limitations
Minor clothing details may remain inconsistent, and sliding-window generation lacks global information exchange for very long videos.
Abstract
from arXiv · showhide
For recent diffusion-based generative models, maintaining consistent content across a series of generated images, especially those containing subjects and complex details, presents a significant challenge. In this paper, we propose a new way of self-attention calculation, termed Consistent Self-Attention, that significantly boosts the consistency between the generated images and augments prevalent pretrained diffusion-based text-to-image models in a zero-shot manner. To extend our method to long-range video generation, we further introduce a novel semantic space temporal motion prediction module, named Semantic Motion Predictor. It is trained to estimate the motion conditions between two provided images in the semantic spaces. This module converts the generated sequence of images into videos with smooth transitions and consistent subjects that are significantly more stable than the modules based on latent spaces only, especially in the context of long video generation. By merging these two novel components, our framework, referred to as StoryDiffusion, can describe a text-based story with consistent images or videos encompassing a rich variety of contents. The proposed StoryDiffusion encompasses pioneering explorations in visual story generation with the presentation of images and videos, which we hope could inspire more research from the aspect of architectural modifications. Our code is made publicly available at https://github.com/HVision-NKU/StoryDiffusion.
1 INTRODUCTION
StoryDiffusion addresses the challenge of generating subject-consistent images and videos for text-based stories. It combines a zero-shot Consistent Self-Attention module with a semantic-space motion predictor for longer, smoother sequences.
- Existing diffusion models generate high-quality content, but maintaining characters’ identity and attire across story images and videos remains challenging.
- Temporal consistency modules can preserve relationships across images or video frames, but typically require extensive data and computation.
- Consistent Self-Attention guides self-attention with reference-image tokens, enabling zero-shot consistency without model training or fine-tuning.
- The method connects features across images in a batch, promoting consistency in characters, faces, and attire while retaining text-based storytelling control.
- StoryDiffusion combines training-free image consistency with Semantic Motion Predictor transitions to generate long image sequences or videos from text-based stories.
2 RELATED WORK
Related work spans diffusion-based image generation, controllable identity preservation, text-to-video synthesis, and condition-guided transition videos. StoryDiffusion builds on these areas while targeting consistent story images and transitions between frames.
- Diffusion-based image generation includes latent diffusion, DiT, Stable XL, and methods adding structural controls such as depth, pose, sketches, or layout.
- Identity-preservation methods either fine-tune models using a given image or directly use pretrained models such as IP-Adapter and PhotoMaker.
- Video diffusion research includes 3D U-Nets, temporal attention, cascaded sampling, and multi-stage pipelines designed to balance video quality and computational cost.
- Condition-guided video generation uses auxiliary signals including depth maps, pose maps, RGB images, and guided motion videos.
- Transition-video methods such as SEINE and SparseCtrl generate videos conditioned on sparse or endpoint information, but StoryDiffusion focuses on joining given start and end frames.
3 METHOD
StoryDiffusion has two stages: training-free Consistent Self-Attention generates subject-consistent story images, and Semantic Motion Predictor converts adjacent images into transition videos using semantic-space motion conditions.
- Training-Free Consistent Images Generation: The first stage inserts Consistent Self-Attention into a pretrained text-to-image diffusion model and generates images from story prompts in a batch.
- Training-Free Consistent Images Generation: Consistent Self-Attention replaces original U-Net self-attention while reusing its weights, making the module training-free and pluggable.
- Training-Free Consistent Images Generation: For image features I ∈ R^B×N×C, standard self-attention independently processes each image’s projected queries, keys, and values.
- Training-Free Consistent Images Generation: Consistent Self-Attention samples tokens from other batch images, pairs them with the current image, and uses the original query with new keys and values for cross-image attention.
- Training-Free Consistent Images Generation: A sliding window supports longer story generation by limiting peak memory dependence on text length, while cross-image interactions promote consistent characters, faces, and attire.
- Semantic Motion Predictor for Video Generation: The Semantic Motion Predictor encodes start and end images into semantic space, interpolates their embeddings, and predicts intermediate frames with transformer blocks.
- Semantic Motion Predictor for Video Generation: Predicted semantic embeddings condition a video diffusion decoder through cross-attention with text embeddings, and training uses MSE between predicted and ground-truth frames.
- Semantic Motion Predictor for Video Generation: Semantic-space encoding integrates spatial relationships and enables smoother transition videos with large motion.
4 EXPERIMENTS
The experiments evaluate StoryDiffusion for subject-consistent image generation and transition video generation against recent methods, using qualitative, quantitative, ablation, and user-study comparisons. Results show stronger consistency, text controllability, and video continuity, including without training for consistent image generation.
- Experimental Setup: StoryDiffusion is evaluated on Stable Diffusion XL and Stable Diffusion 1.5, with separate image and video-generation configurations.Image comparisons use Stable-XL weights, while video generation uses Stable Diffusion 1.5 with a pretrained temporal module.
- Consistent Image Generation: StoryDiffusion produces more consistent faces and attire while preserving prompt-specific details than IP-Adapter and PhotoMaker.Qualitative examples report better retention of objects such as telescopes, dogs, and cards alongside consistent identity and clothing.
- Consistent Image Generation: StoryDiffusion achieves the best performance on both text-image similarity and character similarity metrics.These metrics assess prompt-image alignment and similarity of generated characters, respectively.
- Transition Video Generation: StoryDiffusion generates smoother, more physically plausible transition videos than SEINE and SparseCtrl, without corrupted intermediate frames in the reported examples.The comparisons also describe better appearance consistency and preservation of physical spatial relationships.
- Transition Video Generation: StoryDiffusion outperforms SEINE and SparseCtrl across LPIPS-first, LPIPS-frames, CLIPSIM-first, and CLIPSIM-frames.The first-frame metrics reflect overall video continuity, while consecutive-frame metrics reflect continuity between adjacent frames.
- Ablation and User-Specified ID Generation: A sampling rate of 0.3 fails to maintain subject consistency, whereas higher rates preserve it; the method uses 0.5 in practice.The study also shows that Consistent Self-Attention can generate images conforming to user-specified ID images when combined with PhotoMaker.
- User Study: In a blinded user study with 30 participants, StoryDiffusion shows an overwhelming advantage for both subject-consistent images and transition videos.Users evaluated 50 questions each, with randomized result ordering and comparisons against IP-Adapter, PhotoMaker, SparseCtrl, and SEINE.
5 CONCLUSIONS
StoryDiffusion generates consistent images for storytelling without training and transitions those images into videos. Its two components connect images to preserve visual consistency and create video transitions.
- StoryDiffusion generates consistent images in a training-free manner for storytelling.
- Consistent Self-Attention builds connections among multiple images to generate images with consistent faces and clothing.
- Semantic Motion Predictor transitions the consistent images into videos to better narrate the story.
A.1 TRANSITION VIDEOS GENERATION
Additional StoryDiffusion results demonstrate its transition-video generation capabilities. The reported videos are high-quality and smooth, with Fig. 7 presenting further examples.
- StoryDiffusion produces high-quality transition videos with smooth transitions.
- Fig. 7 presents additional videos created by StoryDiffusion to showcase its transition-video generation capabilities.
- The figure includes videos that can be played in Acrobat Reader by clicking the image.
A.2 CONSISTENT IMAGES GENERATION WITH CONTROLNET
StoryDiffusion’s training-free, pluggable Consistent Self-Attention can be integrated with ControlNet for pose-controlled generation. The combined approach produces subject-consistent images under ControlNet guidance.
- Consistent Self-Attention integrates with ControlNet to introduce pose control into subject-consistent image generation.
- The combined approach generates subject-consistent images under ControlNet guidance.
- The ControlNet experiment is displayed in Fig. 8.
B IMPLEMENTATION DETAILS
The Semantic Motion Predictor is trained from AnimateDiff V2 initialization using semantic conditioning from OpenCLIP. Its architecture includes transformer layers with a 1024-dimensional hidden representation.
- Training uses AnimateDiff V2 motion-module weights, a learning rate of 1e-4, 100k iterations, and 8 A100 GPUs.
- OpenCLIP ViT-H-14 encodes the conditional images into the image semantic space.
- The Semantic Motion Predictor incorporates 8 transformer layers with a hidden dimension of 1024.
C LIMITATIONS
StoryDiffusion has limitations in both subject-consistent image generation and transition video generation. Minor clothing details may remain inconsistent, while very long videos lack global information exchange.
- Minor clothing details, such as ties, may remain inconsistent across subject-consistent generated images.
- More detailed prompts may be required to maintain consistency when clothing details are difficult to preserve.
- Very long video generation remains imperfect because the method lacks global information exchange.