Source-linked AI summary
SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, Ziwei Liu
TL;DR
Existing video generators usually produce short, single-scene clips, leaving a need for coherent story-level videos with creative transitions and prediction across scenes. SEINE uses a random-mask, text- and video-conditioned diffusion model to generate intermediate transition frames and extend to related video tasks. Experiments report superiority over existing methods for transition and prediction, while performance depends on alignment between prompts and scene conditions.
Problem
Existing AI-generated videos are usually short shot-level clips depicting single scenes, limiting coherent story-level video generation with transitions across scenes.
Method
SEINE uses a random-mask video diffusion model conditioned on scene images and textual descriptions to generate intermediate transition frames between scenes.
Results
SEINE reports superior performance over existing methods for generative transition and autoregressive prediction, evaluated with temporal consistency, semantic similarity, and video-text alignment.
Takeaways & Limitations
The model supports coherent story-level long videos and extends to image-to-video animation and autoregressive video prediction.
Takeaways & Limitations
Transition quality depends on alignment between the provided text and scene, while autoregressive prediction can cause watermarks to emerge and persist.
Abstract
from arXiv · showhide
Recently video generation has achieved substantial progress with realistic results. Nevertheless, existing AI-generated videos are usually very short clips ("shot-level") depicting a single scene. To deliver a coherent long video ("story-level"), it is desirable to have creative transition and prediction effects across different clips. This paper presents a short-to-long video diffusion model, SEINE, that focuses on generative transition and prediction. The goal is to generate high-quality long videos with smooth and creative transitions between scenes and varying lengths of shot-level videos. Specifically, we propose a random-mask video diffusion model to automatically generate transitions based on textual descriptions. By providing the images of different scenes as inputs, combined with text-based control, our model generates transition videos that ensure coherence and visual quality. Furthermore, the model can be readily extended to various tasks such as image-to-video animation and autoregressive video prediction. To conduct a comprehensive evaluation of this new generative task, we propose three assessing criteria for smooth and creative transition: temporal consistency, semantic similarity, and video-text semantic alignment. Extensive experiments validate the effectiveness of our approach over existing methods for generative transition and prediction, enabling the creation of story-level long videos. Project page: https://vchitect.github.io/SEINE-project/ .
1 INTRODUCTION
SEINE addresses the gap between short, single-scene video clips and coherent story-level videos by generating creative transitions between scenes. It introduces a conditional short-to-long diffusion model and evaluates transitions using multiple criteria.
- Motivation: Current video generation methods typically produce shot-level clips lasting only a few seconds and depicting a single scene.These short videos are described as insufficient for cinematic and film production.
- Problem and approach: SEINE frames generative transition as producing intermediate frames between two different scenes for longer, more intricate visual storytelling.The model targets smooth and creative transitions between scenes and varying-length shot-level videos.
- Problem and approach: The model uses initial and final scene frames with text and video conditioning to generate a smooth transition video between them.A flexible random-mask diffusion model controls the transition process while targeting coherence and visual quality.
- Extensions: SEINE extends to image-to-video animation and autoregressive video prediction, supporting long-shot videos and dynamic video creation.These extensions broaden the model beyond transition generation.
- Evaluation and contributions: The paper proposes generative transition and prediction as a new problem and reports superior performance on transition criteria across versatile applications.The stated criteria evaluate semantic relevance, temporal coherence and smoothness, and consistency with provided text.
2 RELATED WORKS
Prior work addresses video generation, scene transitions, and long-video synthesis through diffusion, interpolation, morphing, and sequential latent modeling. These approaches provide context for SEINE’s transition objective and story-level generation setting.
- Text-to-Video Generation: Diffusion models substantially advanced text-to-image generation, motivating their expansion to video generation.The related work positions video generation as an extension of progress in text-conditioned image synthesis.
- Transition Generation: Traditional scene transitions include fades, dissolves, wipes, IRIS effects, and cuts, while morphing estimates pixel-level similarities and transitional offsets.These methods use predefined patterns or image correspondences to connect scenes.
- Transition Generation: Generative approaches have used latent-code interpolation to capture semantic similarities in applications such as style transfer and object transfiguration.This establishes latent interpolation as a related strategy for generating visual transformations.
- Long Video Generation: Long-video generation has used GANs, VAEs, discrete-latent transformers, and time-sensitive transformers to model and sequentially produce videos.Examples include VideoGPT and TATS, which generate video through learned latent representations.
- Long Video Generation: Story-level video generation targets continuous multi-scene videos with varying shot lengths.This setting matches the broader goal of connecting distinct scenes into longer videos.
3 METHODOLOGY
SEINE builds on diffusion models to generate intermediate transition frames and extend short videos into longer sequences. Its random-mask conditioning preserves visible frames while enabling text-controlled, coherent frame generation and recursive prediction.
- Diffusion foundation: The framework builds on LaVie, a cascaded text-to-video diffusion model with temporal interpolation and video super-resolution components.LaVie-base uses Stable Diffusion with temporal attention and image-video joint fine-tuning.
- Generative transition: The transition task generates intermediate frames between source images S1 and S2 under a transition-description caption c.The generated sequence is intended to satisfy temporal coherence, semantic similarity, and video-text alignment.
- Random-mask conditioning: SEINE encodes videos into latent representations and applies a random binary mask to selectively preserve visible conditional frames while suppressing others.The masked latent code and mask are concatenated as conditional input to the model.
- Noise prediction: The U-Net receives the corrupted latent input with masking information and is trained to predict noise for generating realistic, visually coherent transition frames.The objective models noise affecting the corrupted latent code together with the textual description.
- Masking strategy: Sampling masks from a Bernoulli distribution makes approximately two or three of every n frames visible on average.This masking choice targets transition and prediction settings in which adjacent frames are especially relevant.
- Short-to-long generation: The model supports arbitrary-position frame generation, endpoint-conditioned transition synthesis, and recursive long-video prediction using recently generated frames.Text prompts control transition style and content, while recursive prediction reuses the last few generated frames.
4 EXPERIMENTS
Experiments compare SEINE with transition baselines, evaluate transition quality across three semantic and temporal criteria, and demonstrate diverse, controllable transitions and long-video prediction. Results also show stable autoregressive generation and applicability to image-to-video animation.
- Qualitative Comparison: SEINE is compared with Morphing, Stable Diffusion, and VQGAN-based interpolation for generative transition.The baselines use image or latent-code interpolation, whereas SEINE is evaluated as a video transition model.
- Qualitative Comparison: SEINE preserves smooth object motion while avoiding transparent blending and inconsistent intermediate images observed in competing methods.The comparisons include transitions such as Spider-Man becoming a sand sculpture and a cat changing pose.
- Quantitative Comparison: Transition quality is assessed using temporal coherence, semantic similarity, and video-text alignment, including CLIPSIM-text, CLIPSIM-Scenes, and CLIPSIM-frames.CLIPSIM-Scenes compares frames with source scene images, while CLIPSIM-frames measures similarity between generated frames.
- Quantitative Comparison: Human raters preferred SEINE over Morphing, VQGAN-based, and Stable Diffusion approaches in transition video quality.The evaluation used 100 videos and 10 raters, who compared pairs of videos based on overall quality.
- Diverse and Controllable Transition Generation: Identical prompts and different random seeds produce diverse, interesting, and reasonable transitions, while camera-motion prompts provide controllable effects.Zoom-in and zoom-out controls can create occlusions that reveal the subsequent scene during transition.
- Video Prediction and Applications: For 1024-frame UCF101 generation, SEINE degrades more slowly than TATS and consistently outperforms it after fine-tuning.SEINE also maintains visual quality and semantic coherence during autoregressive long-video prediction and produces image-to-video animations for square and rectangular inputs.
5 FAILURE CASES
SEINE’s failure cases include watermark persistence, abrupt transitions between dissimilar scenes, and text-video unalignment that can reduce generation quality.
- Watermarks: Watermarks from WebVid10M may emerge during autoregressive prediction and persist throughout the generated video.In transition, watermarks in input images affect intermediate frames; without input watermarks, they are less likely and usually faint.
- Failure Examples: The transition examples include changes between scenes such as a lake and autumn setting, a panda on a swing, and a sea becoming a trumpet-playing raccoon.These examples illustrate the varied scene changes used in the failure-case discussion.
- Failure Examples: Additional examples include style transformation and changes in a panda’s activity, such as moving from sketch-like to realistic or from television viewing to office work.These examples provide further cases of scene or content transformation.
- Abrupt Transition: Transitions can become abrupt or visually inconsistent when source and target scenes lack meaningful correspondences or shared elements.The authors suggest multi-step image editing to transform source scenes toward target images while preserving structure, background, and objects.
- Text-video Unalignment: Generation quality can suffer when prompts do not align with the conditional image or realistic scenario.The authors identify improved alignment between text and conditional images as future work for long-video generation.
6 CONCLUSION
The conclusion presents SEINE as a short-to-long video diffusion model for smooth transitions and autoregressive prediction, with extensions to coherent story-level video generation.
- Conclusion: SEINE uses a random-mask video diffusion model to generate transitions from textual descriptions while targeting coherence and visual quality.The model is described as extending to image-to-video animation and autoregressive video prediction.
- Conclusion: SEINE supports autoregressive prediction and image-to-video animation for creating coherent story-level long videos.The conclusion frames these extensions as applications beyond transition generation.
- Conclusion: Transition evaluation uses temporal consistency, semantic similarity, and video-text semantic alignment.The conclusion identifies these three criteria as the basis for evaluating transitions.
- Conclusion: The experiments report superior performance over existing methods for transition and prediction.The reported evaluation covers both transition and prediction tasks.
7 ETHICS STATEMENT
SEINE shares ethical concerns with other text-to-image and text-to-video diffusion models, including possible unintended bias from training data.
- Ethics Statement: SEINE may inherit ethical concerns from text-to-image and text-to-video diffusion models, including unintended bias introduced by training data.The authors also note potential implications for filmmaking, video games, and artistic creation.
- Ethics Statement: The model’s short-to-long video generation capability may have implications for filmmaking, video games, and artistic creation.This implication is stated alongside the discussion of shared ethical concerns.