FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling

Haonan Qiu; Menghan Xia; Yong Zhang; Yingqing He; Xintao Wang; Ying Shan; Ziwei Liu

Source-linked AI summary

FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling

Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, Ziwei Liu

arXiv:2310.15169v3Oct 23, 2023cs.CV

TL;DR

Video diffusion models are limited by short training clips and single-text conditioning, hindering high-fidelity long videos and changing content over time. FreeNoise analyzes initial noise, reschedules noises for long-range correlation, and injects motion for multiple prompts; experiments report improved video quality, efficiency, and visual coherence.

Problem
Video diffusion models are generally trained on few frames and commonly support only single-text conditioning, limiting long-video generation and multi-stage content control.
Method
FreeNoise is a tuning-free paradigm that reschedules noises for long-range temporal correlation and adds motion injection through cross attention for multi-prompt video generation.
Results
FreeNoise outperforms existing state-of-the-art methods in video quality and computational efficiency while achieving multi-prompt long-video generation with visual coherence.
Takeaways & Limitations
Noise rescheduling and motion injection extend pretrained video diffusion models to longer videos with multiple text prompts without tuning.
Takeaways & Limitations
Repeated locally shuffled noises weaken the introduction of new content as video length increases, which can limit subject displacement in some cases.

Abstract

from arXiv · show

With the availability of large-scale video datasets and the advances of diffusion models, text-driven video generation has achieved substantial progress. However, existing video generation models are typically trained on a limited number of frames, resulting in the inability to generate high-fidelity long videos during inference. Furthermore, these models only support single-text conditions, whereas real-life scenarios often require multi-text conditions as the video content changes over time. To tackle these challenges, this study explores the potential of extending the text-driven capability to generate longer videos conditioned on multiple texts. 1) We first analyze the impact of initial noise in video diffusion models. Then building upon the observation of noise, we propose FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models while preserving content consistency. Specifically, instead of initializing noises for all frames, we reschedule a sequence of noises for long-range correlation and perform temporal attention over them by window-based function. 2) Additionally, we design a novel motion injection method to support the generation of videos conditioned on multiple text prompts. Extensive experiments validate the superiority of our paradigm in extending the generative capabilities of video diffusion models. It is noteworthy that compared with the previous best-performing method which brought about 255% extra time cost, our method incurs only negligible time cost of approximately 17%. Generated video samples are available at our website: http://haonanqiu.com/projects/FreeNoise.html.

1 INTRODUCTION

Video diffusion models struggle to generate coherent, high-fidelity long videos because they are trained on few frames, and they commonly support only single-text conditioning. FreeNoise addresses both challenges through noise rescheduling and motion injection without tuning pretrained models.

Challenges: Limited-frame training causes generated-video quality to decline as inference length increases.Stitching short fragments avoids the training-inference gap but produces disconnected and incoherent videos.
Challenges: VideoLDM frame content depends on initial noises across all frames, making new-content introduction difficult while preserving subjects and scenes.Temporal attention creates full interaction among frame noises, so resampling one frame affects others.
Challenges and solution: Single-text conditioning cannot represent videos whose main subjects perform multiple actions over time.Motion Injection gradually adds new motion during denoising steps associated with object shapes, without additional inference time.
Contributions: FreeNoise is a tuning-free paradigm for longer-video generation that improves video quality and computational efficiency while maintaining content consistency.The paper also proposes motion injection for multi-prompt long-video generation with visual coherence.

2 RELATED WORK

The related work frames video diffusion as an extension of latent diffusion with temporal modeling, while distinguishing training strategies and auxiliary designs across existing text-to-video systems. Prior long-video methods address temporal generation but face complexity, resource, and consistency challenges.

Latent Diffusion Models: The diffusion model learns to reverse a forward process that progressively adds noise across T timesteps.The reverse process obtains less noisy samples iteratively through a noise-prediction network.
Latent Diffusion Models: Latent Diffusion Models perform diffusion and denoising in a learned low-dimensional latent space produced by an autoencoder.The encoder maps images to latent codes, and the decoder reconstructs images from those codes.
Video Latent Diffusion Models: VideoLDM extends latent diffusion to video latent space with temporal-aware layers, including temporal convolution and temporal transformers for cross-frame operations.The video latent representation adds a frame dimension to the image latent representation.
Related Text-to-Video Models: Related text-to-video models primarily differ in training strategies or auxiliary designs such as frame-rate conditioning and image-video joint training.AlignYourLatent trains only temporal blocks, whereas ModelScope fully trains the model from an initialized Stable Diffusion checkpoint.
Long Video Generation: Long-video generation is challenging because temporal complexity, resource limitations, and content-consistency requirements increase with video length.Both GAN-based and diffusion-based approaches have been proposed to address this problem.

3 METHODOLOGY

FreeNoise extends a pretrained VideoLDM to longer, semantically accurate and temporally coherent videos through noise rescheduling and window-based temporal attention. It also supports changing events with multiple text prompts by injecting motion through cross-attention during selected denoising steps.

3 METHODOLOGY: FreeNoise targets longer videos from a VideoLDM trained on Ntrain frames while preserving semantic accuracy and temporal coherence.The target video contains M frames, where M > Ntrain, without additional training.
3.1 OBSERVATION AND ANALYSIS: Independent noise sampling for longer sequences makes global coherence difficult because temporal modules were trained to attend over fixed Ntrain-frame neighborhoods.Temporal attention globally interacts with frames, while independently resampled noises can alter other frames and disrupt content consistency.
3.2 NOISE RESCHEDULING FOR LONG-RANGE CORRELATION: Noise rescheduling constructs long-range-correlated noise sequences while retaining internal randomness for longer-video inference.The method initializes Ntrain independent noise frames and extends them through local shuffling, whose unit size S divides Ntrain.
3.2 NOISE RESCHEDULING FOR LONG-RANGE CORRELATION: Temporal attention is computed within local sliding windows of U = Ntrain frames, with stride S, so each window matches the model’s training-frame noise distribution.Overlapping window outputs are fused using weights based on each frame’s distance from the window center.
3.2 NOISE RESCHEDULING FOR LONG-RANGE CORRELATION: Overlapped-window slicing and merging are restricted to temporal attention, adding no computation to other U-Net modules.This design is intended to preserve computational efficiency while processing longer sequences.
3.3 MOTION INJECTION FOR MULTI-PROMPT VIDEO GENERATION: For multi-prompt videos, the method uses the first prompt for most denoising steps and gradually injects target prompts through spatial cross-attention during object-shape and pose-related steps.Frames between [Nγ, Nτ] receive linearly interpolated prompt embeddings for a smooth transition, while the last L cross-attention layers use the target prompt throughout denoising.

4 EXPERIMENTS

Experiments evaluate FreeNoise for 64-frame generation from a 16-frame-trained VideoCrafter model, comparing quality, consistency, efficiency, multi-prompt control, and ablations. FreeNoise produces higher-fidelity, more consistent long videos while adding little inference time.

Longer Video Generation: FreeNoise preserves main subjects and scenarios while allowing content variation, outperforming direct sampling, sliding windows, and Gen-L-Video in qualitative longer-video comparisons.Direct sampling suffers from the training-inference gap; sliding windows lose long-range consistency, while Gen-L-Video exhibits content mutation.
Efficiency: Less than 20% extra inference time is incurred by FreeNoise, whereas Gen-L-Video takes nearly four times longer than direct inference on NVIDIA A100.FreeNoise limits most additional calculations to temporal-attention layers.
Quantitative Results: FreeNoise achieves the best FVD, KVD, and CLIP-SIM among the compared methods, with CLIP-SIM indicating superior content consistency.Direct inference obtains the worst FVD and KVD because of the training-inference gap.
User Study: FreeNoise receives the highest user-study scores for content consistency, video quality, and video-text alignment, with almost twice the content-consistency votes of the second-place method.Examples were displayed in random order to reduce evaluation bias.
Multi-Prompt Generation: Motion Injection enables coherent transitions between multiple prompts, while noise rescheduling alone can introduce unexpected scene changes.Examples include camel running to standing and an astronaut resting to riding a horse.
Ablation Study: Removing noise rescheduling breaks semantic continuity across frames, while using stride S = 8 produces looser cross-window interaction and gradual bowl-shape changes.Stride 4 provides sufficient content consistency; stride 2 is omitted because it would double extra inference time.

5 CONCLUSION

The study addresses limited-frame training and single-text conditioning by extending pretrained video diffusion models to high-fidelity long videos with multiple text prompts. FreeNoise combines tuning-free noise-based extension with motion injection and achieves this with approximately 17% additional time cost versus 255% for the previous best-performing method.

Conclusion: FreeNoise extends pretrained video diffusion models to high-fidelity long videos conditioned on multiple texts.The approach targets models trained on limited frame counts and supporting single-text conditions.
Conclusion: Motion injection supports multi-text video generation, while the overall paradigm maintains content consistency without model tuning.The conclusion reports extensive experimental confirmation of the paradigm’s superiority.
Conclusion: Approximately 17% additional time cost is required by FreeNoise, compared with 255% extra time cost for the previous best-performing method.The comparison concerns additional inference time.

6 ETHICS STATEMENT

The ethics statement emphasizes that FreeNoise remains rooted in the original pretrained video generation model. It therefore frames oversight around the original model’s ethical standards.

Ethics Statement: FreeNoise-generated content remains rooted in the original pretrained video generation model.The statement says regulators need only oversee the original model for ethical-standard adherence.

7 REPRODUCIBILITY STATEMENT

The authors provide detailed algorithm and implementation information and report implementations on three advanced video generation models. Code links are listed for VideoCrafter, AnimateDiff, and LaVie.

Reproducibility Statement: The paper states that researchers familiar with video diffusion models should be able to basically reproduce FreeNoise from its detailed algorithm and implementation descriptions.The authors report implementations on three advanced video generation models.
Reproducibility Statement: FreeNoise implementations are provided for VideoCrafter, AnimateDiff, and LaVie through listed code repositories.The statement supplies separate links for each model implementation.

A APPENDIX: IMPLEMENTATION DETAILS

The appendix specifies the sampling, resolution, guidance, evaluation-generation, and user-study settings used to compare longer-video inference methods.

DDIM sampling uses 50 denoising steps, eta 0, 256 × 256 resolution, and classifier-free guidance scale 15.
Quantitative comparisons generate 2048 videos per longer-inference method from 512 EvalCrafter prompts and 4 random initial noises.FVD and KVD use direct-inference 16-frame videos as references, while 64-frame videos are cut into four 16-frame segments for length alignment.
The user study asks 27 users to select the best video among the proposed method and three baselines for consistency, quality, and video-text alignment.

B APPENDIX: CASE ANALYSIS OF SIGNIFICANT MOVEMENT

The case analysis categorizes significant movement into three types and reports that FreeNoise can produce all of them during inference.

Movement Types: Significant movement is categorized as lens movement with the subject, subject movement off the screen, and subject movement within the screen.
Lens Moving with the Subject: For lens movement with the subject, the subject position changes little while movement appears through background regression.
Subject Moving off the Screen: For subject movement off the screen, the subject can suddenly reappear because of semantic constraints.
Subject Moving within the Screen: For subject movement within the screen, size limitations can make the subject turn, while the pretrained model behaves unnaturally during turning.
Inference Behavior: FreeNoise produces all three movement types, which are automatically determined during inference from sampled random noises and the given prompt.

C APPENDIX: LIMITATION DISCUSSION

The limitation discussion identifies weakening new-content introduction over increasing video length and dependence on the pretrained base model’s motion capabilities.

Repeated locally shuffled noises weaken the introduction of new content as video length increases, limiting subject displacement in some cases.
FreeNoise does not eliminate motion variation or fully fix object spatial structure, including left-to-right movement.
FreeNoise performance is constrained by the base model, which works well mainly for lens movement with the subject and struggles with subjects moving off or within the screen.