Source-linked AI summary
Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise
Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, Michael Ryoo, Paul Debevec, Ning Yu
TL;DR
Video diffusion models lack fine-grained, interactive motion control, while their spatiotemporal complexity makes adapting architectures or training pipelines difficult. Go-with-the-Flow preprocesses videos into optical-flow-warped latent noise and fine-tunes models without architectural changes. Experiments and user studies report strong visual quality, motion controllability, and temporal consistency across multiple motion-control applications.
Problem
Fine-grained motion control in video diffusion remains under-explored because spatiotemporal correlations are intricate and the appropriate motion representation is unresolved.
Method
The method preprocesses training videos into temporally correlated, optical-flow-warped noise that preserves spatial Gaussianity, then fine-tunes video diffusion models without architecture or pipeline changes.
Results
Experiments and user studies demonstrate advantages in visual quality, motion controllability, and temporal consistency across local object motion, camera movement, and motion transfer.
Takeaways & Limitations
The approach provides a unified, model-agnostic paradigm for user-friendly motion control in video diffusion models.
Takeaways & Limitations
Noise-warping methods that do not preserve Gaussianity can deteriorate per-frame performance, making Gaussianity preservation a method constraint.
Abstract
from arXiv · showhide
Generative modeling aims to transform random noise into structured outputs. In this work, we enhance video diffusion models by allowing motion control via structured latent noise sampling. This is achieved by just a change in data: we pre-process training videos to yield structured noise. Consequently, our method is agnostic to diffusion model design, requiring no changes to model architectures or training pipelines. Specifically, we propose a novel noise warping algorithm, fast enough to run in real time, that replaces random temporal Gaussianity with correlated warped noise derived from optical flow fields, while preserving the spatial Gaussianity. The efficiency of our algorithm enables us to fine-tune modern video diffusion base models using warped noise with minimal overhead, and provide a one-stop solution for a wide range of user-friendly motion control: local object motion control, global camera movement control, and motion transfer. The harmonization between temporal coherence and spatial Gaussianity in our warped noise leads to effective motion control while maintaining per-frame pixel quality. Extensive experiments and user studies demonstrate the advantages of our method, making it a robust and scalable approach for controlling motion in video diffusion models. Video results are available on our webpage: https://eyeline-labs.github.io/Go-with-the-Flow. Source code and model checkpoints are available on GitHub: https://github.com/Eyeline-Labs/Go-with-the-Flow.
1. Introduction
Video diffusion models struggle with fine-grained motion control because of intricate spatiotemporal correlations and unclear motion representations. Go-with-the-Flow addresses this by structuring latent noise with optical-flow-derived temporal correlations while preserving spatial Gaussianity, enabling model-agnostic control across several applications.
- Fine-grained interactive motion control remains under-explored because video frames have intricate spatiotemporal correlations and the best motion-control representation is unresolved.
- Existing motion-control applications center on local object motion, global camera movement, and transferring motion from reference videos to new contexts.
- The method preprocesses training videos into optical-flow-warped noise, requiring only a data change and no diffusion-model architecture or training-pipeline modifications.
- Its noise warping correlates temporal latent structure while maintaining spatial Gaussianity, supporting motion correspondence and per-frame pixel quality.
- A faster-than-real-time warping algorithm reduces the quadratic-cost bottleneck of prior HIWYN warping, making large-scale fine-tuning more practical.
- Experiments and user studies report advantages in pixel quality, controllability, temporal consistency, and subjective preference across local motion, motion transfer, and camera movement tasks.
3. Method
Go-with-the-Flow warps latent noise frame by frame according to optical flow, tracking density during expansion and contraction while preserving Gaussianity. The resulting structured noise conditions fine-tuned video diffusion models for controllable object, camera, and transferred motion.
- Method overview: Go-with-the-Flow separates noise warping from video-diffusion training and introduces motion control through noise initialization without adding model parameters.
- 3.1.1 Algorithm: The algorithm stores only the previous frame’s noise and per-pixel flow densities, using parallelizable pixel-level operations for efficient frame-by-frame warping.
- 3.1.1 Algorithm: Forward and backward optical flows produce next-frame noise and density temporally correlated with the previous frame’s corresponding quantities.
- 3.1.1 Algorithm: Expansion maps one source pixel to multiple targets, while contraction aggregates sources; density values weight aggregated noise to preserve its distribution.
- 3.1.2 Theoretical analysis: The algorithm preserves i.i.d. standard Gaussian pixels when the previous-frame noise has that distribution.
- 3.1.2 Theoretical analysis: Its per-frame time complexity is O(D), because the bipartite graph contains at most 2D edges.
- 3.4. Video diffusion inference with warped noise: At inference, degradation γ controls motion adherence: motion transfer typically uses γ between 0.2 and 0.5, while synthetic object control uses γ between 0.5 and 0.7.
- 3.4. Video diffusion inference with warped noise: The approach supports local object motion, camera movement, arbitrary motion transfer, and model-agnostic use with AnimateDiff and CogVideoX.
4. Experiments
Experiments show that the method preserves spatial Gaussianity, improves noise-generation efficiency and temporal consistency, and supports controllable motion across local, transfer, and camera-movement tasks. User studies and ablations further indicate stronger motion fidelity, visual quality, and robustness than the evaluated baselines.
- Noise quality: 0.00014 Moran’s I with p-value 0.84 and K-S statistic 0.060 with p-value 0.44 indicate preserved spatial Gaussianity and normality.Values close to zero for Moran’s I and small K-S statistics support the intended spatial distribution of warped noise.
- Efficiency: 26× faster than HIWYN, the proposed noise warping algorithm runs one order of magnitude faster than real time through linear time complexity.Profiling used an NVIDIA A100 40GB GPU at 1024×1024 resolution.
- Image diffusion applications: On DAVIS super-resolution, the algorithm outperforms all baselines in temporal consistency while remaining comparable in PSNR, SSIM, and LPIPS.Interpolation-based methods produce lower image quality because they fail to preserve spatial Gaussianity.
- Motion transfer and camera control: On DAVIS, DL3DV, and WonderJourney, the method improves motion fidelity and video quality for object motion transfer and camera movement control.The method also preserves object fidelity and 3D consistency under intricate or overlapping motions, including in qualitative comparisons.
- Ablations: Warped-noise motion conditioning improves DAVIS I2V FID, FVD, CoTracker mIoU, and Vbench relative to the Original CogVideoX-5B variant.The ablation attributes the improvement to adding realistic motion guidance beyond text prompts and the initial frame.
5. Conclusion
The method introduces faster-than-real-time noise warping for motion control in video diffusion and establishes a unified paradigm for diverse applications. Experiments and user studies report advantages in visual quality, motion controllability, and temporal consistency.
- Faster-than-real-time noise warping incorporates motion control into video diffusion noise sampling.The algorithm is presented as bridging structured motion and latent noise.
- The method preprocesses video data for video diffusion fine-tuning, supporting a unified paradigm for user-friendly motion-controllable video generation.
- Extensive experiments and user studies demonstrate superiority in visual quality, motion controllability, and temporal consistency.
Social impact statement
The framework is presented as computationally efficient and accessible, with potential applications in creative industries. The paper also acknowledges risks from misuse and advocates safeguards for ethical deployment.
- The framework is described as enabling creators, developers, and artists to produce dynamic content with minimal resources or specialized training.
- Potential misuse includes creating deepfakes or misleading media.
- The paper advocates responsible use, content labeling, and detection mechanisms to support ethical deployment.
6. Gaussianity preservation of our noise warping algorithm
The algorithm preserves Gaussian white noise while transferring noise through optical-flow-induced expansion and contraction. Its graph-based aggregation and renormalization maintain independent standard-Gaussian pixels across frames.
- Gaussianity preservation: Weighted sums of independent standard Gaussians remain Gaussian, and the proof computes cross-covariances to establish independence.
- Gaussianity preservation: Each output pixel is formed from Gaussian edge contributions and renormalized to have mean 0 and unit variance.
- Exact recovery: An expansion followed by contraction can deterministically recover the original noise and density under the stated flow conditions.
- Exact recovery: In the recovery example, the final pixel has density 1 and the original noise q.
7. Qualitative results of training-free image diffusion based video editing
Noise warping methods that fail to preserve Gaussianity degrade per-frame performance in training-free image diffusion-based video editing.
- Nearest-neighbor and bilinear interpolation destroy Gaussianity in warped noise.The passage links this failure to qualitative artifacts shown in Fig. 11.
- Destroyed Gaussianity consequently deteriorates per-frame performance on pretrained image-to-image diffusion models.The reported degradation is illustrated in Figs. 12 and 13.
8. The advantage of noise warping
Noise warping provides motion information while discarding input-video structure that cannot be inferred from motion alone. Compared with activation manipulation, the flow-based approach can avoid certain structural distortions.
- Noise warping discards input-video structure that cannot be inferred from motion alone.This trade-off can be advantageous for motion-driven editing.
- MotionClone’s activation manipulation gives the windmill an extra set of arms, whereas optical-flow-based warping relies solely on motion information.The comparison highlights a difference between manipulating model activations and conditioning noise with optical flow.
- Figure 11 visualizes noise produced by HIWYN, bilinear, and nearest-neighbor interpolation during forward roller-coaster motion.The forward movement forces the noise to expand significantly, and HIWYN appears visibly non-Gaussian early in the video.
- Figure 12 compares noise-warping algorithms for video super-resolution on the DAVIS dataset using DeepFloyd IF.
9. Comparison to the video diffusion base model without finetuning
Video diffusion models can respond to warped noise without fine-tuning, but the resulting videos suffer from reduced per-frame quality and potential visual artifacts.
- Without fine-tuning, output motion still roughly follows the flow of warped noise.This response occurs despite CogVideoX being trained on pure Gaussian noise.
- Warped noise can cause visual artifacts because it is statistically distinct from the pure Gaussian noise used to train CogVideoX.The passage also notes that per-frame quality suffers without fine-tuning.
- Figure 13 compares noise-warping algorithms for portrait video relighting using DifFRelight.
10. User study settings and statistics
The paper evaluates user preferences, controllability, and temporal consistency for local object motion control and turnable camera movement. It also reports applicability beyond the primary base model, including AnimateDiff and HunyuanVideo.
- The user study covers local object motion control and turnable camera movement video generation.Questionnaires measure overall subjective preference, controllability, and temporal consistency.
- The method is data- and model-agnostic, adding motion control to arbitrary video diffusion models by processing noise sampling during fine-tuning.
- The approach works with AnimateDiff fine-tuned on WebVid, with qualitative results reported in Fig. 16.
- The community has trained a version on HunyuanVideo, supporting the method’s applicability to more advanced video diffusion base models.
12. Pseudo code
The pseudo code and supplementary figures document real-time noise warping, user-study setup, and AnimateDiff fine-tuning with warped noise. The implementation derives flow between frames and uses it to transform noise for motion-controlled generation.
- Visual evaluation: Figure 14 compares the input, derived optical flow, proposed outputs, a baseline, and an ablation on a cut-and-drag windmill rotation.The caption reports that frame-structural information can give MotionClone an extra set of arms, whereas the proposed method receives motion information.
- Base-model fine-tuning: Figure 16 shows AnimateDiff T2V fine-tuned with warped noise derived from an input video and initialized under different text prompts.
- Noise warping: The pseudo code defines a warp_noise routine that takes previous and current frames, previous noise, and a previous weight as inputs.
- Noise warping: The noise-warping implementation computes optical flow between consecutive frames and uses the negative flow as a cheap backward-flow approximation.The flow computation is explicitly agnostic to the optical-flow algorithm.
- Implementation: The supplementary material points readers to the noise-warping pseudo code and publicly available source code and model checkpoints.