PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality

Mohamed Elmoghany, Ryan Rossi, Seunghyun Yoon, Subhojyoti Mukherjee, Eslam Bakr, Puneet Mathur, Gang Wu, Viet Dac Lai, Nedim Lipka, Ruiyi Zhang, Varun Manjunatha, Chien Nguyen, Daksh Dangi, Abel Salinas, Mohammad Taesiri, Hongjie Chen, Xiaolei Huang, Joe Barrow, Nesreen Ahmed, Hoda Eldardiry, Namyong Park, Yu Wang, Jaemin Cho, Anh Totti Nguyen, Zhengzhong Tu, Thien Nguyen, Dinesh Manocha, Mohamed Elhoseiny, Franck Dernoncourt

arXiv:2507.07202v1cs.CV

TL;DR

Long-video generation faces challenges in temporal consistency and dataset availability, while methods beyond 16 seconds can accumulate quality degradation. This survey synthesizes architectural components and training strategies and organizes methods into a taxonomy. It concludes that several backbone, training, encoder, VAE, positional-encoding, and prompt-rewriting patterns are effective in modern video generation.

  • Problem

    Video generation faces temporal-consistency challenges beyond 16 seconds, alongside limited open-source long-form datasets and missing metadata for detailed scene annotation.

    Source: §1 · p. 1

  • Method

    The survey analyzes architectural elements and training methodologies, introduces a taxonomy, compares state-of-the-art components, and catalogs datasets and evaluation metrics.

    Source: §1 · pp. 2–3

  • Results

    The survey identifies MM-DiT and Flux-MM-DiT as effective backbones, flow matching as the preferred objective for realism, and MLLMs as outperforming T5 text encoders, among other patterns.

    Source: §4 · p. 8

  • Takeaways & Limitations

    The survey’s taxonomy and component comparisons provide architectural guidance for next-generation foundational video-diffusion models.

    Source: §1 · p. 3

    Current limitations include substantial memory and GPU requirements, inconsistent temporal coherence in longer sequences, and generated videos that are very short, between 5–16 seconds.

    Source: §4 · p. 8

Abstract

from arXiv · show

Despite the significant progress that has been made in video generative models, existing state-of-the-art methods can only produce videos lasting 5-16 seconds, often labeled "long-form videos". Furthermore, videos exceeding 16 seconds struggle to maintain consistent character appearances and scene layouts throughout the narrative. In particular, multi-subject long videos still fail to preserve character consistency and motion coherence. While some methods can generate videos up to 150 seconds long, they often suffer from frame redundancy and low temporal diversity. Recent work has attempted to produce long-form videos featuring multiple characters, narrative coherence, and high-fidelity detail. We comprehensively studied 32 papers on video generation to identify key architectural components and training strategies that consistently yield these qualities. We also construct a comprehensive novel taxonomy of existing methods and present comparative tables that categorize papers by their architectural designs and performance characteristics.

1. Introduction

Long-video generation must manage temporal and spatial consistency while overcoming limits in long-range coherence, multi-subject continuity, and access to richly annotated datasets. The survey organizes architectural and training approaches intended to support longer, coherent, visually compelling videos.

1. Introduction

  • Beyond 16 seconds, autoregressive methods accumulate quality degradation, including inconsistencies and visual artifacts, while spatiotemporal attention requires substantial memory and compute.Character appearance and temporal scene consistency become especially challenging with movement and multiple characters, while abrupt transitions and physical interactions add further difficulties.Source: §1 · pp. 1–2
  • Video generation combines temporal and spatial complexity, making frame consistency a key challenge beyond text or image generation.Diffusion models and advances in large language models are described as foundations for cinematic long-form video generation.Source: §1 · p. 1
  • Open-source datasets typically contain only a few seconds of footage and rarely include metadata such as shot type, camera motion, character emotions, background context, and action labels.Commercially usable video datasets are also very limited.Source: §1 · p. 1
  • The survey highlights architectural elements and training methodologies addressing video-generation obstacles and proposes a framework of components for longer, coherent, visually compelling videos.It also introduces a taxonomy of methods, compares state-of-the-art architectural components, surveys architectural advances, and catalogs datasets and evaluation metrics.Source: §1 · pp. 2–3

2. Long Video Generation Architectural Styles

The survey groups long-video generation into architectural styles with different trade-offs in temporal decomposition, compression, consistency, and control. It also distinguishes personalization and multi-shot planning strategies, alongside token-stream and proprietary approaches.

2. Long Video Generation Architectural Styles

  • The survey organizes video-generation research into six taxonomies and provides analysis and guidelines for selecting an architectural paradigm.The categories include keyframes-to-video, discrete temporal chunks, high compression, flattened 3D one-shot, token-stream autoregressive generation, and closed-source systems.Source: §2 · p. 3

2.1. Keyframes-to-Video

  • Keyframes-to-video generates keyframe images first, then uses motion pipelines and stitched segments to scale to arbitrarily long videos.The sequential stages can add latency and may require separate text-to-image and text-to-video models; StoryDiffusion generates keyframes from sub-prompts, while KeyVID segments audio latents.Source: §2.1 · p. 3

2.2. Discrete Temporal Chunks

  • Discrete temporal chunks generate and concatenate disjoint N-frame segments, capping memory at the chunk level and supporting parallel processing, but potentially introducing boundary artifacts.MAGI-1 uses 24-frame segments, with each denoised segment conditioning the next; CA2-VDM shows chunked training schedules can require additional epochs.Source: §2.2 · p. 3

2.3. High Compression

  • High compression reduces inference costs but can sacrifice fine details and motion diversity: LTXVideo compresses spatiotemporal dimensions 192×, while FramePack weights recent frames most heavily.LTXVideo may introduce artifacts in rapid movement; FramePack supports arbitrary-duration extension but can yield limited background and motion diversity.Source: §2.3 · p. 3

2.4.1. Foundational One-Shot

  • Flattened 3D one-shot models synthesize a full spatiotemporal tensor in one pass, with foundational models learning joint priors and using 3D VAEs, but GPU demands limit clip length and resolution.The one-shot approach captures cross-frame dependencies and semantic coherence; foundational backbones include 3D UNet Diffusion, DiT, and MM-DiT, with WAN2.1 using DiT plus cross-attention and HunyuanVideo using dual-stream MM-DiT.Source: §2.4 · p. 3, §2.4.1 · p. 4

2.4.2. Single-Subject Personalization

  • Single-subject personalization adapts a base generator using identity adapters, textual inversion, or LoRA; multi-subject personalization separately encodes and fuses entities through four modular strategies.The multi-subject strategies are cross-attention fusion, element embedding fusion, decoupled concept embeddings, and multimodal adapter fusion.Source: §2.4.2 · p. 4, §2.4.3 · p. 4

2.4.4. Multi-Shot Narrative Planning

  • Multi-shot planning segments long videos into shots or scenes for coherent transitions, using native diffusion planning, a planner layered over a backbone, or LLM-directed autoregressive generation.SkyReels-V2 renders shots sequentially and supports hard cuts, soft dissolves, and infinite-length film generation.Source: §2.4.4 · p. 4

2.5. Token-Stream Autoregressive-Token

  • Token-stream autoregressive methods predict unified text-video tokens but face detail loss, compression artifacts, high memory and compute costs, slow inference, and accumulating long-range errors.Proprietary systems are described as having a substantial performance gap over open-source approaches; Veo3 and Seedance 1.0 are reported to produce high-fidelity videos with physical plausibility, complex multi-subject scenes, intricate motion, and character consistency.Source: §2.5 · p. 4, §2.6 · p. 5

3. Long Video Generation Architectural Modules Recommendations

The survey recommends architectural components while noting that the architecture can vary by taxonomy. For text-visual encoding, it discusses CLIP and T5 combinations and reports HunyuanVideo’s replacement of T5 with an MLLM for better alignment.

3. Long Video Generation Architectural Modules Recommendations

  • The survey recommends MLLMs as text encoders, MeanFlow as a diffusion training objective, and MM-DiT or Flux-MM-DiT backbones, while allowing architecture to vary by taxonomy.Table 1 compares key architectural components across surveyed literature.Source: §3 · p. 6

3.1. Text-Visual Encoder

  • CLIP text-visual encoding is commonly paired with T5 variants for semantically rich embeddings, while HunyuanVideo replaces T5 with an MLLM and reports better text-visual alignment.The passage describes CLIP, T5, T5-XXL, and umT5 as common encoder components.Source: §3.1 · p. 6

3.2. Diffusion Training Objective

The section traces diffusion and latent autoencoders as foundations for video generation, then contrasts flow matching and MeanFlow with denoising diffusion. It reports MeanFlow’s quality metrics and faster inference relative to flow matching.

3.2. Diffusion Training Objective

  • Diffusion models such as DDPM [27] and DDIM [65] laid groundwork for video generation, while VAE latent representations reduced computational cost and enabled more efficient sampling [60].Source: §3.2 · p. 6
  • MeanFlow reaches an FVD of 128 versus flow-matching’s 142 on Kinetics-400, an SSIM of 0.85 versus 0.82 on UCF-101, and an LPIPS of 0.12 versus 0.15.It also reduces inference time by 4× compared with flow matching and narrows the quality gap with multi-step diffusion models.Source: §3.2 · p. 7
  • Flow matching directly regresses continuous-time vector fields that transport noise to data, reducing reliance on multi-step noise schedules.The survey describes it as a more robust and efficient alternative to DDPM and DDIM denoising samplers.Source: §3.2 · p. 7

3.3. Variational Auto Encoder (VAE)

The survey covers compressed video representations, separate appearance and motion encoders, and attention designs for modeling spatial and temporal dependencies. It notes both reported efficiencies and remaining limits in long-term consistency.

3.3. Variational Auto Encoder (VAE)

  • 3D VAEs capture spatio-temporal dependencies, and the survey identifies them as the most common choice in top-performing literature.Other designs use VQ-VAE [58] [70] for compression and reconstruction, or modify the decoder to perform a final denoising step [24].Source: §3.3 · p. 7

3.4. Dual Variational Auto Encoder

  • Separate appearance and motion encoders specialize feature learning and reduce modality interference; Open-Sora 2.0 [54] reports 5–10× lower training costs while maintaining state-of-the-art video quality.VideoAlchemist [10] uses separate foreground and background streams for built-in multi-subject personalization and improved identity consistency without test-time fine-tuning.Source: §3.4 · p. 7

3.5. Attention Mechanism

  • Video diffusion architectures evolved from 3D UNets to 2D spatial backbones with temporal attention, but early systems still produced 2–5 second clips with artifacts and limited long-term consistency.Recent designs include Seedance’s windowed temporal attention, reported as 10× faster on 1080p benchmarks, and MAGI-1’s block-causal attention and shared query projections.Source: §3.5 · p. 7

3.6. Positional Encoding

The survey reviews positional encodings, diffusion-transformer backbones, prompt rewriting, and story agents as components used in video generation. It describes reported design trade-offs and how these approaches structure conditioning and narrative generation.

3.6. Positional Encoding

  • 3D RoPE is used to enhance motion coherence and length extrapolation, while Seedance’s MM-RoPE appends caption tokens to 3D RoPE to tighten text–video alignment in multi-shot generation.WAN2.1 [71] retains sinusoidal encodings, and LTX-Video [24] uses one-dimensional RoPE over flattened tokens for real-time inference.Source: §3.6 · pp. 7–8

3.7. Diffusion-based Transformers

  • DiT backbones generate better images and videos than UNet backbones; MM-DiT links text and visual embeddings for stronger alignment and lower FID at increased parameter and inference cost.Flux-MM-DiT adds rectified-flow residual modules for one-step sampling and faster convergence, with added architectural complexity.Source: §3.7 · p. 8

3.8. Prompt Enhancement

  • LLM-powered prompt rewriting aligns short user prompts with richly detailed training captions, and the survey reports sharper detail, less flicker, and more unified style across frames.Examples include Seedance [20], HunyuanVideo-Avatar [12], and StepVideo [32].Source: §3.8 · p. 8

3.9. Story Agent

  • A story agent segments plots into scenes and shots, aligns characters, locations, and camera cues, then generates scene-specific prompts for keyframe-based clips that are concatenated into a long-form video.The described architecture decouples prompt-level frame-quality refinements from agent-governed plot flow.Source: §3.9 · p. 8

4. Conclusions and Future Work

The conclusion synthesizes architectural findings, identifies persistent limitations, and proposes research directions for long-video generation. It highlights that closed-source systems still lead overall quality despite progress in open-source models.

4. Conclusions and Future Work

  • Closed-source systems still lead in overall quality, although open-source HunyuanVideo [42] and Wan2.1 [71] can generate realistic, high-fidelity videos.Source: §4 · p. 8
  • The survey favors MM-DiT and Flux-MM-DiT backbones, flow matching for realism, MeanFlow as a promising successor, MLLMs over T5, convolutional VAEs with discriminator loss, dual VAEs, 3D RoPE variants, and LLM prompt rewriting.Source: §4 · p. 8
  • Named limitations include high memory and GPU requirements, few open-source long-form datasets, missing annotations, weak temporal coherence, limited interaction control, multi-subject identity problems, short 5–16-second videos, and incoherent stories.Source: §4 · p. 8
  • Proposed directions include collecting open-source long-video data, adding hierarchical metadata, quantizing and pruning models, distilling models, enhancing prompts, structuring story narration, and using multiple adapters or repeated reference images.Source: §4 · pp. 8–9
  • The authors present these findings as directions for future research aimed at bridging remaining gaps in video generation.Source: §4 · p. 9

A. Datasets

The dataset landscape ranges from large web-video corpora with little narrative context to longer, more structured resources designed for story-driven generation.

A. Datasets

  • Web-scale corpora exceed 250 M clips collectively, but their stock-footage provenance yields noisy captions and licenses that forbid commercial use.Human-centric datasets supply face tracks, skeletons, and camera-motion labels, but most clips remain under 20 s [105, 44, 83, 102].Source: A. Datasets · p. 15
  • MovieBench [86] is described as the first movie-level dataset with hierarchical movie, scene, and shot annotations for character consistency and multi-scene storytelling.MiraData provides 1–2 min sequences with dense captions covering objects, actions, style, and camera motions [40].Source: A. Datasets · p. 15
  • Vript [93] provides six-minute films with 145-word scene-level scripts, while HD-VILA-100M [91] and Panda-70M [9] mostly contain 5–15 s clips with minimal narrative context.The contrast highlights the difference in duration and narrative detail across these datasets.Source: A. Datasets · p. 15

B. Evaluation Metrics

Video-generation evaluation has used image-derived and text-alignment metrics, but these measures can miss temporal and narrative qualities in longer clips. VBench offers fine-grained dimensions for diagnosing generated-video quality.

B. Evaluation Metrics

  • Image-derived metrics including IS [62], FID [26], FVD [69], SSIM, and LPIPS [99], alongside CLIPScore [25], have been used to assess video generation.The cited passage notes these measures offer convenient benchmarks.Source: B. Evaluation Metrics · p. 15
  • These measures can obscure temporal coherence, storytelling fidelity, and multi-scene consistency, and may diverge from human judgments on longer, more complex clips.Source: B. Evaluation Metrics · p. 15
  • VBench [34] decomposes video quality into dimensions including visual quality, motion smoothness, identity consistency, temporal flicker, spatial relationships, and text-video relevance.Its dimensions use tailored prompt sets and human-preference annotations; the passage calls it a principal benchmark for next-generation video models.Source: B. Evaluation Metrics · p. 15

TV IV VV VE Len. FPS #Frames

The model listings span different video-generation tasks and reported generation statistics, including video length, frame rate, and frame count.

TV IV VV VE Len. FPS #Frames

Loading 2507.07202v1…