Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise

1. Introduction

• Fine-grained motion control remains difficult in video diffusion, while existing approaches can require architectural changes, precise parameterized signals, and trade visual quality for rigid control [45] [64].Source: §1 · p. 2
• The method correlates latent noise over time by warping a 2D Gaussian noise slice with optical flow, requiring only data preprocessing and no changes to model architecture or training pipelines.The authors present it as plug-and-play with video diffusion base models and compatible with other controls.Source: §1 · pp. 2–3
• Unlike HIWYN [8]’s quadratic-cost warping, the algorithm warps between consecutive frames, tracks noise and flow density in both flow directions, and samples conditional white noise to preserve spatial Gaussianity in real time.It accounts for expansion and contraction dynamics.Source: §1 · pp. 2–3
• The approach supports local object motion, reference-based camera movement, and motion transfer, with experiments and user studies reporting advantages in pixel quality, controllability, temporal consistency, text alignment, and user preference.Motion transfer can use optical flow or other representations, including flows from 3D rendering and depth warping.Source: §1 · p. 3

2.1. Image and video diffusion models

• Noise warping lets image diffusion generate temporally correlated videos without fine-tuning, but HIWYN [8] has spatial-Gaussianity and runtime shortcomings; the work also fine-tunes CogVideoX [64] and AnimateDiff [16].Training-free evaluations apply the algorithm to DifFRelight [18] for video relighting and DeepFloyd IF for video super-resolution.Source: §2.1 · p. 3

2.2. Motion controllable video generation

• Prior motion-control work spans object trajectories [13] [39] [45] [49] [58] [60], camera poses or directional moves [17] [30] [58] [59] [62], and reference-video transfer [14] [34] [38] [57].The paper positions its black-box flow approach across these control paradigms.Source: §2.2 · p. 4

3. Method

• The method separates noise warping from diffusion training and controls motion through noise initialization, adding no parameters to the video diffusion model.After fine-tuning on videos paired with warped noise, the model supports motion control at inference.Source: §3 · p. 4

3.1.1 Algorithm

• The fast, frame-by-frame algorithm tracks previous-frame noise and per-pixel flow density, using parallelizable pixel operations to warp noise according to forward and backward optical flow.Each frame depends only on the previous frame’s noise and density.Source: §3.1.1 · pp. 4–5
• The algorithm handles expansion and contraction with flow-linked noise transfer, weighting aggregated values by density to preserve distributions over time.Forward flow moves noise pixels; backward flow fills gaps, and density tracks how many pixels were aggregated.Source: §3.1.1 · p. 5

3.1.2 Theoretical analysis

• If previous-frame pixels are i.i.d. standard Gaussians, the algorithm outputs i.i.d. standard Gaussian pixels, and its per-frame time complexity is O(D).The edge count is bounded by 2D.Source: §3.1.2 · p. 5

3.2. Training-free image diffusion models with warped noise

• Prior work [8] and [11] uses flow-warped noise for training-free, temporally consistent video edits with per-frame image diffusion models.The passage gives relighting [18] and super-resolution as example tasks.Source: §3.2 · p. 5

3.3. Fine-tuning video diffusion models with warped noise

• Fine-tuning CogVideoX T2V and I2V variants changes only the training noise: optical-flow-warped noise replaces regular Gaussian noise under the original objective and pipeline.The warped noise tensor is created iteratively for each training video from its framewise optical flow.Source: §3.3 · pp. 5–6

3.4. Video diffusion inference with warped noise

• At inference, warped noise initializes deterministic sampling without pipeline changes; synthetic polygon flows enable local motion control, while driving-video flows support motion transfer and camera movement.Synthetic object motion typically uses degradation γ = 0.5–0.7, while motion transfer typically uses γ = 0.2–0.5.Source: §3.4 · pp. 6–7

3.5. Implementation details

• The method is data- and model-agnostic, extending motion control to arbitrary video diffusion models by processing noise sampling during fine-tuning.The passage also reports use with AnimateDiff fine-tuned on WebVid.Source: §3.5 · p. 7

4.1. Gaussianity

• The method preserves spatial Gaussianity, achieving Moran’s I of 0.00014 with p-value 0.84 and K-S statistic 0.060 with p-value 0.44.Bilinear, bicubic, and nearest-neighbor warping have Moran’s I values of 0.24 to 0.30 and K-S statistics of 0.17 to 0.37, with p-values below 0.05.Source: §4.1 · p. 7, Table 1 · p. 8

4.2. Efficiency

• Noise generation is 26× faster than HIWYN and more than an order of magnitude faster than real time, enabling on-the-fly warping during fine-tuning.Profiling used an NVIDIA A100 40GB GPU to generate noise at 1024×1024 resolution; the method also runs faster than InfRes.Source: §4.2 · p. 7, Table 1 · p. 8

4.3.1 DeepFloyd IF video super-resolution

• On DeepFloyd IF super-resolution, the method outperforms all baselines in warping error and is comparable in PSNR, SSIM, and LPIPS except against interpolation methods, which yield low-quality generations.The evaluation uses 43 DAVIS videos downsampled to 64×64 and superresolved to 256×256.Source: §4.3.1 · p. 8, Table 1 · p. 8

4.3.2 DifFRelight video relighting

• For DifFRelight relighting, the method achieves consistently advantageous image and temporal metrics across baselines, while bilinear and bicubic noise show low warping error alongside significantly worse image quality.The evaluation covers four subjects across four camera-motion scenarios with fixed lighting conditions.Source: §4.3.2 · p. 8, Table 1 · p. 8

4.4.1 Local object motion control

• User-defined masks and cut-and-drag trajectories, plus synthetic camera-rotation flows, guide localized noise warping to follow intended motion while preserving object structure.Precise flows from input templates guide inference in the I2V model.Source: §4.4.1 · pp. 8–9
• On both VIPSeg and the 40 videos from the user study, the method outperforms all training-based and training-free baselines, matching the authors’ qualitative observations.It maintains high object fidelity and 3D consistency, even with intricate or overlapping motions.Source: §4.4.1 · p. 9

4.4.2 Motion transfer and camera movement control

• On DAVIS I2V, the method improves FID, FVD, and motion metrics; it also outperforms MotionClone on DL3DV and WonderJourney camera-control results.For first-frame editing, it preserves the original camera movement and added-object identity, while MotionClone adds unintended objects and AnyV2V gradually loses the foreground object.Source: §4.4.2 · pp. 11–12, Table 2 · p. 11

4.4.3 Ablation studies

• Ablations show that warped-noise motion conditioning improves DAVIS I2V reconstruction and VBench quality over text-only CogVideoX-5B, and γ ≈0.5 is generally suitable.Increasing γ tightens motion control; training on a fraction of the dataset performs worse than the full model, and CogVideoX-2B performs worse than the main CogVideoX-5B model.Source: Table 2 · p. 11, §4.4.3 · p. 12

• The proposed faster-than-real-time noise warping algorithm incorporates motion control into video diffusion noise sampling.The authors characterize it as bridging chaos and order in generative modeling.Source: §5 · p. 12
• Experiments and user studies demonstrate superiority in visual quality, motion controllability, and temporal consistency.The authors describe the method as a robust and versatile solution for motion control in video diffusion models.Source: §5 · p. 12
• Preprocessing video data for video diffusion fine-tuning provides a unified paradigm for user-friendly, motion-controllable video generation applications.Source: §5 · p. 12

Social impact statement

• The framework could democratize high-quality video generation for filmmaking and animation by enabling creators to produce dynamic content with minimal resources or specialized training.The paper characterizes this contribution as computationally efficient and accessible.Source: Social impact statement · p. 12
• Potential misuse includes deepfakes and misleading media; the authors advocate responsible use, content labeling, and detection mechanisms to support ethical deployment.Source: Social impact statement · p. 12
• Compatibility with diverse models is presented as a way to encourage transparency and research collaboration in addressing societal concerns while maximizing positive impact.Source: Social impact statement · p. 12

6. Gaussianity preservation of our noise warping algorithm

• The section combines a formal proof with an illustrative expansion-contraction example to show preservation of the underlying Gaussian distribution during warping.The example follows noise across three frames.Source: §6 · p. 16
• Each up-sampled noise value Xi is Gaussian because it is constructed as a weighted sum of independent standard Gaussians.The proof identifies q(x, y) and the Zi values as i.i.d. standard Gaussians.Source: §6 · p. 16
• The stated finding is that all Xi values are independent, establishing independence among the Xi variables themselves, without specifying a noise distribution or relationship to output pixels.Source: §6 · p. 16
• The output pixel remains Gaussian with mean 0, and renormalization using its computed variance restores a standard Gaussian.For a target with no incoming edges, the algorithm instead samples an independent standard Gaussian.Source: §6 · p. 16
• In Example 1, expansion of a density-1 frame-1 pixel into two frame-2 pixels yields density 1/2 at each under Algorithm 1.Source: §6 · p. 16
• After expansion and subsequent contraction, the example deterministically recovers the original pixel’s noise and density.The final pixel has density 1 and noise q, matching the original pixel’s state.Source: §6 · p. 16

7. Qualitative results of training-free image diffusion based video editing

• Nearest-neighbor and bilinear interpolation destroy Gaussianity and consequently deteriorate per-frame performance in pretrained image-to-image diffusion models [8].The passage points to Fig. 11 for the loss of Gaussianity and Figs. 12–13 for the per-frame performance decline.Source: §7 · p. 16

• Unlike MotionClone, which gives the windmill an extra set of arms, our optical-flow-based method avoids those artifacts.MotionClone manipulates diffusion-model activations rather than using optical flow to guide the video trajectory.Source: §8 · p. 16
• Noise warping conditions motion while discarding input-video structural information that cannot be inferred from motion alone.Source: §8 · p. 16
• Forward movement expands the noise in a long roller-coaster video, while HIWYN [8] produces visibly non-Gaussian noise early on.Figure 11 visualizes the warped noise, HIWYN [8], bilinear interpolation, and nearest-neighbor interpolation.Source: Figure 11 · p. 17

9. Comparison to the video diffusion base model without finetuning

• Without fine-tuning, CogVideoX outputs roughly follow warped-noise flow, but per-frame quality suffers and statistical mismatch with its training Gaussian noise can cause visual artifacts.The warped noise is statistically distinct from the pure Gaussian noise on which CogVideoX was trained.Source: §9 · p. 17

• Fig. 15 presents the user study questionnaires and statistics for local object motion control and turnable camera movement video generation, assessing preference, controllability, and temporal consistency.Source: §10 · p. 17

11. Model Agnostic

• The method adds motion control to arbitrary video diffusion models by processing only noise sampling during fine-tuning.The authors describe it as data- and model-agnostic.Source: §11 · p. 17
• AnimateDiff T2V fine-tuned with warped noise uses input-video-derived noise to initialize generations under different text prompts.The example model was fine-tuned on WebVid [2]; the passage identifies AnimateDiff [16].Source: §11 · p. 17, Figure 16 · p. 21
• The authors report a community-trained Go-with-the-Flow version on HunyuanVideo and say the method will generalize to more advanced future base models.Source: §11 · p. 18

12. Pseudo code

• In the windmill cut-and-drag example, MotionClone adds extra arms, while warped noise conveys optical-flow motion without double windmills.The figure attributes the extra arms to per-frame structural information and says the method uses motion information from optical flow alone.Source: Figure 14 · p. 19
• With γ = .5 on CogVideoX before fine-tuning, warped noise causes visual artifacts but still conveys motion cues, including a camera zoom toward the windmill.The figure attributes the artifacts to warped noise being statistically distinct from the Gaussian noise used to train CogVideoX.Source: Figure 14 · p. 19
• Across questions on local object motion and turnable camera movement, the method wins the most user preferences, according to the reported user-study statistics.Source: Figure 15 · p. 20, §12 · p. 20
• Figure 17 presents the noise-warping pseudocode, and the authors point readers to GitHub for source code and model checkpoints.Source: §12 · p. 18, Figure 17 · p. 22