Source-linked AI summary
AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort
Wen Wang, Canyu Zhao, Hao Chen, Zhekai Chen, Kecheng Zheng, Chunhua Shen
TL;DR
Story visualization must generate images that are high-quality, text-aligned, and identity-consistent, but existing methods often restrict their scope or require per-image user controls. AutoStory combines LLM-based layout planning with text-to-image generation and automatically produced dense controls, alongside a method for generating multi-view consistent character images. The paper reports improvements over existing approaches in story quality and subject-characteristic preservation.
Problem
Story visualization must balance image quality, text alignment, and character identity consistency, while existing approaches often restrict characters and scenarios or require per-image controls.
Source: §1 · p. 2Method
AutoStory uses LLM planning for layouts, dense conditions for image generation, and a module that converts sparse layouts into sketch or keypoint controls, with multi-view character generation.
Source: §1 · p. 2Results
The authors report that AutoStory outperforms existing approaches in story quality and subject-characteristic preservation, while generating text-aligned, identity-consistent, high-quality images.
Source: §4.2 · p. 8, §5 · p. 12Takeaways & Limitations
The approach reduces labor-intensive character data collection and achieves its reported results without time-consuming, computationally expensive large-scale training.
Source: §5 · p. 12
Abstract
from arXiv · showhide
Story visualization aims to generate a series of images that match the story described in texts, and it requires the generated images to satisfy high quality, alignment with the text description, and consistency in character identities. Given the complexity of story visualization, existing methods drastically simplify the problem by considering only a few specific characters and scenarios, or requiring the users to provide per-image control conditions such as sketches. However, these simplifications render these methods incompetent for real applications. To this end, we propose an automated story visualization system that can effectively generate diverse, high-quality, and consistent sets of story images, with minimal human interactions. Specifically, we utilize the comprehension and planning capabilities of large language models for layout planning, and then leverage large-scale text-to-image models to generate sophisticated story images based on the layout. We empirically find that sparse control conditions, such as bounding boxes, are suitable for layout planning, while dense control conditions, e.g., sketches and keypoints, are suitable for generating high-quality image content. To obtain the best of both worlds, we devise a dense condition generation module to transform simple bounding box layouts into sketch or keypoint control conditions for final image generation, which not only improves the image quality but also allows easy and intuitive user interactions. In addition, we propose a simple yet effective method to generate multi-view consistent character images, eliminating the reliance on human labor to collect or draw character images.
1 INTRODUCTION
Story visualization must balance image quality, alignment with text, identity consistency, and versatility, while prior approaches often restrict characters, scenes, styles, or require per-image input. AutoStory combines LLM-based layout planning, dense image controls, and automatically generated multi-view character data to reduce user effort.
1 INTRODUCTION
- Story visualization must produce appealing, well-laid-out images aligned with text while keeping character and scene identities consistent across images and supporting varied styles, characters, and scenes.These requirements define the task's quality, consistency, and versatility goals.Source: §1 · p. 2
- Earlier methods simplified the task around specific characters, scenes, and styles, while some newer methods remained comic-book-focused or required user sketches.AutoStory is framed against these restrictions and the labor of collecting or drawing character images.Source: §1 · p. 2
- The system accepts simple story descriptions and also supports optional user controls such as character images, layout adjustments, pose changes, or sketches.The authors describe the pipeline as fully automated with minimal user input while retaining a flexible interface.Source: §1 · p. 2
- AutoStory uses LLM comprehension and planning for layouts, sparse controls for planning, and dense controls such as sketches or keypoints for high-quality image generation.Its dense-condition module automatically transforms bounding-box layouts into sketch or keypoint controls; users can also edit generated layouts and conditions.Source: §1 · p. 2
- AutoStory combines few-shot parameter-efficient fine-tuning and customized generation to preserve character identity while generalizing across characters, scenes, and styles.This approach trains on only a few images per character rather than time-consuming domain-specific data.Source: §1 · p. 2
- A training-free multi-view method treats character views as a video and uses temporal-aware attention, with 3D priors increasing image diversity without compromising identity consistency.The method addresses the need for users to provide multiple diverse, identity-consistent character images.Source: §1 · p. 2
2 RELATED WORK
Earlier story-visualization work often relied on fixed datasets, while newer generative approaches broadened the available controls and customization techniques. AutoStory is positioned amid these approaches as a pipeline using layouts and intuitive dense controls, alongside a method for generating multi-view character data.
2.1 Story Visualization
- Earlier story-visualization methods used GANs, VQ-VAE, or diffusion models but commonly targeted specific characters, scenes, styles, and datasets, limiting generalization.StoryDALL-E also introduced story continuation from an initial image.Source: §2.1 · p. 3
- Recent work pursued diverse characters, but Jeong et al. [Jeong et al. 2023] had identity-preservation and single-human-character limits, while Intelligent Grimm [Liu et al. 2023c] focused on children’s-book style and required retraining for other styles or contents.TaleCraft [Gong et al. 2023] required user sketches for each character in each image, and existing methods relied on multiple character images.Source: §2.1 · p. 3
2.2 Controllable Image Generation
- Text-to-image systems can struggle with multiple characters and complex scenes, prompting spatial controls such as ControlNet, T2I-Adapter, and GLIGEN.LLM-based layout methods include LayoutGPT and LLM-grounded Diffusion; the latter requires hyperparameter tuning and offers limited detailed object control, while AutoStory uses editable sketches or keypoints.Source: §2.2 · pp. 3–4
2.3 Customized Image Generation
- Customized-generation methods support single- and multi-subject personalization, but existing story-visualization methods still require multiple diverse, identity-consistent images for each character.AutoStory proposes training-free consistency modeling and uses 3D priors to generate multi-view consistent character images, reducing reliance on human collection or drawing.Source: §2.3 · p. 4
3 OUR METHOD
AutoStory combines LLM-based story layout planning with text-to-image generation, preparing sparse layouts and dense controls before generating story images. It also generates multi-view character images to reduce the need for user-collected character data.
3 OUR METHOD
- AutoStory uses LLMs to turn story text into panel descriptions and scene layouts, then prepares sparse and dense controls for conditional story-image generation.The layouts use global and local prompts with bounding boxes; the dense controls are sketches or keypoints.Source: §3 · p. 4, §3.1 · pp. 4–5
3.2 Dense Condition Generation
- Dense sketches or keypoints supplement bounding boxes because sparse layouts alone can produce images with poor quality or imperfect scene-layout alignment.The method generates individual objects, extracts sketches or human keypoints, and composes these controls into a full-image condition.Source: §3.2 · pp. 5–6
3.3 Controllable Storytelling Image Generation
- Story images combine sparse box-based layout control, dense T2I-Adapter guidance, and fused character-specific ED-LoRA weights for identity preservation.The dense conditions are generated automatically rather than drawn by users for each character and image.Source: §3.3 · p. 6
3.4 Eliminating Character-wise Data Collection
- A training-free consistency method treats a character’s images as video frames, using temporal-aware attention across frames to improve identity consistency.The method also addresses distortion and color or texture differences that can arise when using viewpoint-translated images directly as customization data.Source: §3.4 · pp. 7–8
- For each character, AutoStory generates one image, creates viewpoint variants, and extracts sketches or keypoints as diverse customization conditions.The viewpoint-conditioned images supply varied controls; generated training data is filtered with CLIP score before customization.Source: §3.4 · pp. 7–8
4 EXPERIMENTS
AutoStory combines language-model layout planning with dense image controls to produce story images across varied inputs. Comparisons and ablations report gains in image quality, text alignment, and character consistency, while showing that layout and dense controls address different failure modes.
4.1 Implementation Details
- The pipeline uses GPT-4 for story-to-layout generation, Stable Diffusion for image generation, and T2I-Adapter sketches or keypoints for dense control.Multi-subject customization is the only trained component; other pipeline parts are training-free.Source: §4.1 · p. 8
4.2 Main Results
- With user-provided characters and with text-only input, AutoStory generates high-quality, text-aligned story images with consistent character identities.Text-only input uses automatically generated character training data; the authors also report varied poses and effective rendering of small story objects.Source: §4.2 · p. 8
4.3 Comparison with Existing Methods
- Qualitative comparisons report stronger identity preservation, text alignment, and generation quality; on FlintstonesSV, AutoStory better aligns images with text than TaleCrafter.TaleCrafter has higher image quality but misses a suitcase; Make-A-Story produces lower-quality images in this comparison.Source: §4.3 · pp. 8, 10
- AutoStory outperforms existing methods on both CLIP text-to-image and image-to-image similarity, and users favor it across alignment, identity preservation, and image quality.The quantitative comparison covers 10 stories and 71 prompts; the user study covers 10 stories and 32 participants.Source: Table 1 · p. 11, §4.3 · p. 11, Table 2 · p. 11
4.4 Ablation Studies
- Layout control alleviates conflicts between object concepts, dense control reduces omissions, and combining them avoids both omissions and conflicts.Without controls, subjects may be omitted or blended; layout control alone can still omit subjects, while dense control alone can leave conceptual conflicts.Source: §4.4 · p. 11, Figure 7 · p. 12
- AutoStory’s multi-view character generation balances diversity and identity consistency, unlike comparison variants that lose consistency or pose diversity or produce artifacts.The method guides generation with One-2-3-45 sketches and uses extended self-attention for consistency.Source: §4.4 · pp. 11–12
5 CONCLUSION
AutoStory combines language models and diffusion models to generate diverse, text-aligned, identity-consistent, high-quality story images with minimal human effort. Experiments report stronger quality and subject preservation than existing approaches without large-scale, time-consuming training.
5 CONCLUSION
- AutoStory combines language models, diffusion models, and a character data generation module to streamline story visualization and reduce user effort.The approach eliminates the need for users to perform labor-intensive character data collection.Source: §5 · p. 12
- The authors report better story quality and subject preservation than existing approaches, while generalizing to varying characters, scenes, and styles without large-scale training.Accelerating multi-concept customization and enabling real-time operation remain future work.Source: §5 · p. 12
A.1 More Implementation Details
The appendix details how the language model prepares story panels and how character inputs differ between user-supplied and text-only stories. It reports high-quality, consistent generated character data, with less pose diversity for human characters than animals.
A.1 More Implementation Details
- The language model generates stories, divides them into panels, converts panel text into image prompts, and parses prompts into layouts and local prompts.The appendix presents detailed prompts and sampled outputs in Fig. 11.Source: §A.1 · p. 14
- For text-only stories, AutoStory generates character training images; animal characters show diverse views, while human characters have less facial-orientation variation.The authors attribute the human-character limitation to diffusion models being trained primarily on frontal faces; generated data supports customization without overfitting.Source: §A.1 · p. 14
A.2 Intermediate Results Visualization
The intermediate visualization shows how local prompts, perception models, and the LLM-generated layout produce dense controls for story-image generation.
A.2 Intermediate Results Visualization
- Local prompts first generate single-character images, then perception models extract human keypoints or non-human sketches for the dense condition.The LLM-generated layout composes the individual subjects’ controls before the final story image is generated from the dense condition, prompts, and layout.Source: §A.2 · pp. 14–15
A.3 More Story Visualization Results
AutoStory produces high-quality, text-aligned, identity-consistent images even for long stories. The section also shows story and layout prompts and intermediate results using sketches for non-human subjects and keypoints for human characters.
A.3 More Story Visualization Results
- AutoStory produces high-quality, text-aligned, identity-consistent story images, including for long stories.Figures 14 and 15 show additional story visualization results.Source: §A.3 · p. 15, Figure 14 · p. 18, Figure 15 · p. 19
- Users can provide story requirements, such as a request for a short story about two named characters, as prompts for story and layout generation.Figure 11 illustrates the prompt format.Source: Figure 11 · p. 16
- Intermediate results use sketch conditions for non-human characters and subjects, and keypoint conditions for human characters.Figures 12 and 13 visualize these respective conditions.Source: Figure 12 · p. 17, Figure 13 · p. 17