PaperlayerView exact revision on arXiv

Source-linked AI summary

Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation

Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, Qifeng Chen

arXiv:2307.06940v1cs.CV

TL;DR

Creating storytelling videos is complex, while text-to-video generation has limited effectiveness and retrieval-guided generation can vary character appearance across clips. The paper retrieves existing videos for structural guidance, combines this with structure-guided video synthesis and character personalization, and reports advantages over existing baselines.

  • Problem

    Storytelling video production is complex, and text-to-video generation remains limited; retrieval-guided generation also struggles with consistent, user-controlled character appearance across clips.

    Source: §1 · p. 2

  • Method

    The framework retrieves videos from text queries as structure references for text-to-video synthesis and uses timestep-variable textual inversion to personalize characters across clips.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    The authors report that depth-guided synthesis surpasses pure-text synthesis and T2V-Zero combined with ControlNet, while TimeInv outperforms existing personalization competitors.

    Source: §1 · p. 2, §4.3 · p. 7

  • Takeaways & Limitations

    The framework is presented as a convenient video-making tool for storytelling with control over layout, motion, and character personalization.

    Source: §1 · p. 2, §5 · p. 8

    The authors identify general character control without finetuning as an area for future improvement.

    Source: §5 · p. 8

Abstract

from arXiv · show

Generating videos for visual storytelling can be a tedious and complex process that typically requires either live-action filming or graphics animation rendering. To bypass these challenges, our key idea is to utilize the abundance of existing video clips and synthesize a coherent storytelling video by customizing their appearances. We achieve this by developing a framework comprised of two functional modules: (i) Motion Structure Retrieval, which provides video candidates with desired scene or motion context described by query texts, and (ii) Structure-Guided Text-to-Video Synthesis, which generates plot-aligned videos under the guidance of motion structure and text prompts. For the first module, we leverage an off-the-shelf video retrieval system and extract video depths as motion structure. For the second module, we propose a controllable video generation model that offers flexible controls over structure and characters. The videos are synthesized by following the structural guidance and appearance instruction. To ensure visual consistency across clips, we propose an effective concept personalization approach, which allows the specification of the desired character identities through text prompts. Extensive experiments demonstrate that our approach exhibits significant advantages over various existing baselines.

1 INTRODUCTION

Storytelling video production is laborious, while text-to-video models struggle to control film-like motion and layout. The paper proposes retrieval-guided video synthesis and character personalization to address these challenges.

1 INTRODUCTION

  • Conventional storytelling video production requires laborious filming or CG animation, and text-to-video models struggle to produce film-suitable motion and layouts.The introduction notes that layout and composition are difficult to control through text, despite their role in visual storytelling.Source: §1 · p. 2
  • The framework retrieves existing videos as structure references for text-to-video synthesis, while prompts specify scene and object appearances.The authors describe retrieval-augmented generation as a way to control layout and composition and use information in retrieved videos to improve realism and coherence.Source: §1 · p. 2
  • TimeInv targets inconsistent character appearance across clips by personalizing the generation model to render user-specified identities.The authors identify stochastic, text-controlled appearance as a source of limited user control.Source: §1 · p. 2
  • The authors evaluate retrieval-enhanced text-to-video generation, personalization, and the overall storytelling synthesis framework as separate perspectives.They report notable superiority for the retrieval-enhanced model and personalization method, and suggest potential for practical applications.Source: §1 · p. 2
  • The paper presents an adjustable structure-guided text-to-video model and TimeInv alongside its retrieval-augmented storytelling paradigm.The stated contributions include resolving conflict between structure guidance and character generation, and outperforming existing personalization competitors.Source: §1 · p. 2

2 RELATED WORK

Related work covers video generation approaches, methods for guiding video structure, and techniques for customizing concepts. These lines of work include unconditional generation, text-conditioned generation, and personalization of specified objects.

2.1 Video Generation

2.2 Structure-guided Video Generation

  • Structure-guided methods use depth, pose, or multiple image conditions; Make-Your-Video adds depth alongside text [Xing et al. 2023].VideoComposer combines conditions such as RGB images, sketches, and depths [Wang et al. 2023b].Source: §2.2 · p. 3

2.3 Concept Customization

3 METHOD

The method turns story scripts into plot-aligned clips by retrieving existing videos for motion structure and synthesizing new appearances. Its controllable video model combines depth guidance with text prompts, while character rerendering targets consistent identities across clips.

3 METHOD

  • The framework extracts structures from retrieved videos to guide text-to-video generation, then rerenders characters for consistency across clips.It aims to generate storytelling videos from scripts or with minimal interaction.Source: §3 · p. 3

3.1 Retrieval-augmented Text-to-Video Generation

  • Each story plot is treated as a single event, converted into a query and prompt, and processed using retrieved video depth as motion structure.Text processing can be manual or assisted by an LLM; retrieval uses an off-the-shelf engine and a database of about 10M open-world videos.Source: §3.1 · pp. 3–4

3.2 Structure-Guided Text-to-Video Synthesis

  • The latent video diffusion model conditions denoising on text prompts and frame-wise depth, adding multiscale structure features to the U-Net while cross-attention handles semantic control.Depth is estimated from the retrieved video, whose appearance need not match the target plot.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 Video Character Rerendering

  • Character rerendering addresses inconsistent appearances across clips and the difficulty of collecting character-specific video data for personalization.It also targets the tradeoff between concept compositionality and character fidelity.Source: §3.3 · p. 4
  • TimeInv replaces one fixed concept token with timestep-dependent embeddings, motivated by denoising steps controlling different image attributes.The method optimizes an embedding table across randomly sampled diffusion timesteps and selects the corresponding embedding during inference.Source: §3.3 · p. 5
  • For image-based personalization, a concept image is repeated into a pseudo-video and depth-guided to avoid binding the learned concept to static motion.At inference, the concept can be combined with other motion guidance to produce diverse actions.Source: §3.3 · p. 5
  • Low-rank trainable matrices added to attention layers capture character appearance details without directly optimizing the pretrained model parameters.The matrices are inserted in cross- and spatial self-attention modules.Source: §3.3 · p. 5
  • Timestep clamping relaxes depth guidance after timestep τ, allowing the character shape to diverge from the source depth while retaining coarse layout and action guidance.The paper reports feature rescaling as a less effective depth-adjustment alternative.Source: §3.3 · p. 5, Figure 5 · p. 5

4 EXPERIMENT

Because storyboard splitting is manual or LLM-assisted, the experiments focus mainly on retrieval-augmented video generation, assessing storytelling synthesis, video quality, and personalization.

4 EXPERIMENT

  • The evaluation covers storytelling synthesis, text-to-video quality, and concept customization, with storyboard splitting treated as manual or LLM-assisted.The authors identify retrieval-augmented generation as the major technical innovation under evaluation.Source: §4 · pp. 5–6

4.1 Implementation Details

  • Training proceeds from a WebVid-10M base text-to-video model to a structure encoder trained with the base frozen, then task-specific depth-guided finetuning and token optimization.Although trained at fixed resolution, the model supports other resolutions at inference.Source: §4.1 · p. 6

4.2 Evaluation on Storytelling Video Synthesis

  • Ablations associate TimeInv with character quality and concept composition, personalization with cross-clip consistency, and retrieval with control of character position and layout.Without retrieval, generation performance is also reported as inferior to retrieval-augmented generation in Table 1.Source: §4.2 · p. 7

4.3 Evaluation on Text-to-Video Synthesis

  • Depth-guided synthesis outperforms pure-text video generation and T2V-Zero combined with ControlNet on UCF-101 zero-shot evaluation.The paper reports these comparisons using FVD and KVD, but the supplied passages provide no metric values.Source: Table 1 · p. 7, §4.3 · p. 7

4.4 Evaluation on Personalization

  • The personalization comparison evaluates semantic alignment and concept fidelity across 200 generated videos using 20 prompt-video pairs and shared random seeds.The compared approaches are DreamBooth, Textual Inversion, Custom Diffusion, and the proposed method.Source: §4.4 · pp. 7–8
  • DreamBooth tends toward overfitting and repetitive backgrounds, Textual Inversion captures fewer character details, and Custom Diffusion often produces character artifacts.The qualitative comparison uses two characters and four approaches with the same random seed.Source: Figure 7 · p. 7, §4.4 · p. 8
  • In image personalization, TimeInv with Custom Diffusion shows better background diversity and concept compositionality, while TimeInv alone improves character similarity over Textual Inversion.The character-similarity comparison notes the cat’s distinctive texture.Source: §4.4 · p. 8, Figure 8 · p. 8
  • TimeInv can replace Textual Inversion or combine with Custom Diffusion, achieving better semantic alignment in the reported quantitative comparison.The supplied passage gives no numerical scores.Source: Table 2 · p. 8, §4.4 · p. 8

5 CONCLUSION

The conclusion presents a retrieval-based pipeline for customized, character-consistent storytelling videos and identifies finetuning-free character control and improved coordination of character and structure controls as future directions.

5 CONCLUSION

  • The pipeline combines structure-guided video generation with a personalization method to improve character control and address character-depth conflict.The conclusion describes the system as supporting video synthesis quality, layout, motion control, and character personalization.Source: §5 · p. 8
  • The authors identify general character control without finetuning and better cooperation between character and structure control as potential future directions.Source: §5 · p. 8
  • The paper concludes that its retrieval-based pipeline can produce customized storytelling videos with character consistency across clips.Source: §5 · p. 8
Loading 2307.06940v1…