Optical-Flow Guided Prompt Optimization for Coherent Video Generation

1. Introduction

• Temporal consistency remains a challenge in text-to-video diffusion, while existing approaches may require fine-tuning or introduce interventions that can adversely affect results.Video guidance also faces sequence-wide computation costs and potential instability.Source: §1 · p. 2
• Applying diffusion guidance to video is difficult because dependencies span frames, making full-sequence backpropagation costly and potentially unstable.Fine-tuning video diffusion models is also challenging because of their large size.Source: §1 · p. 2
• MotionPrompt optimizes learnable prompt tokens during inference, using the prompt’s global influence to guide all frames without retraining or computing gradients for every frame.The method appends a placeholder string for the learnable tokens and preserves the original prompt’s semantic meaning.Source: §1 · p. 2
• A discriminator evaluates optical flow between random frame pairs from real and generated videos, and its gradients guide prompt optimization toward more natural motion.During sampling, the discriminator evaluates a subset of generated frames; the method is intended to minimize impact on samples already close to real videos.Source: §1 · p. 2

Latent Video Diffusion Models

• Latent video diffusion reduces the computational cost and memory demands of high-resolution video generation, but temporal consistency remains challenging.FreeInit refines initial noise but can be computationally expensive and result in detail loss; UniCtrl and VideoGuide can negatively affect performance in some cases.Source: §2 · pp. 2–3

Video Synthesis with Optical Flow

• Prior optical-flow approaches use flow generation, reference flow, or flow completion, whereas MotionPrompt uses a discriminator to guide sampling toward more realistic optical flow.The paper hypothesizes that improving generated optical flow would enhance temporal consistency and motion smoothness in text-to-video generation.Source: §2 · p. 3

Prompt Optimization

• Prompt optimization has been studied mainly for image-domain tasks and remains under-explored for video generation; the paper presents on-the-fly prompt tuning for video diffusion models.The authors describe this as the first application of on-the-fly prompt tuning to video diffusion models.Source: §2 · p. 3

3.1. Conditional Video Diffusion Model

• The video diffusion model encodes clean video into latents, conditions sampling on a text embedding, and generates samples through reverse diffusion.The text embedding is obtained by applying the text encoder to the prompt.Source: §3.1 · p. 3

3.2. Prompt Optimization for Video Guidance

• Optimizing the text embedding using gradients from a subset of frames indirectly controls the video latent, avoiding gradients for every frame.Only appended learnable token embeddings are optimized; the original prompt tokens are retained, and the original prompt is restored after the specified optimization range.Source: §3.2 · p. 4

3.3. Defining Loss Function from Optical Flow

• The objective combines discriminator and total-variation losses with an embedding regularizer that keeps optimized tokens near the original embedding space.The total-variation term regularizes the optical flow toward a smooth field.Source: §3.3 · pp. 4–5
• A discriminator trained on optical flows from real and generated videos guides sampled-frame motion toward flows classified as real.The discriminator can take a single flow between selected frames rather than requiring flow from an entire video sequence.Source: §3.3 · p. 5

4.1. Implementation Details

• The evaluation uses Lavie, AnimateDiff, and VideoCrafter2 with 800 VBench prompts, a 50-step DDIM sampler, and matched seeds for baselines and MotionPrompt.Discriminators are trained on optical flow from complete, clean videos and process optical flow from denoised estimates at inference.Source: §4.1 · pp. 5–6

4.2. Results

• Compared with three baselines, MotionPrompt suppresses appearance changes and sudden object shifts while preserving color tone and prompt-specific scene details.The baselines sometimes fail to preserve object appearance or quantity, or produce objects that suddenly appear or disappear.Source: §4.2 · p. 6, Figure 3 · p. 6
• MotionPrompt improves object consistency, reduces temporal flickering, and enhances motion smoothness with minimal impact on text alignment.The authors note a possible trade-off between consistency and motion dynamics, while visualization results indicate a balance between dynamics and coherence.Source: §4.2 · p. 6, Table 1 · p. 7
• 100 participants evaluating 90 videos preferred MotionPrompt overall in an A/B study of temporal quality.The study included 30 videos per model.Source: §4.2 · p. 7, Table 2 · p. 7

5.1. Ablation Study

• All tested loss-component configurations outperform the baseline, while the selected moderate λ2 balances smoother motion against declines in other metrics.The authors also report that adding ℓdisc improves results; comparisons involving λ3 support their hypothesis that embedding deviation from the trained text space contributed to video-quality degradation.Source: §5.1 · p. 7, Table 3 · p. 7
• Three optimization iterations provide an effective balance; one is insufficient, while seven smooth motion but reduce dynamic degree and consistency and increase computation time.Starting early improves motion smoothness but reduces consistency, whereas starting too late weakens its impact; the authors establish an optimal range for the optimization start.Source: §5.1 · p. 7

5.2. Discriminator Generalization

• Cross-dataset inference generally improves performance when the discriminator is trained on videos generated by Lavie and VideoCrafter2 rather than the inference dataset.The authors conjecture that the higher baseline performance of those models makes their videos harder to distinguish from real ones, leading to stricter discriminator training.Source: §5.2 · pp. 7–8, Table 4 · p. 8

5.3. Token Variations

• Token-embedding cosine similarity decreases during sampling, and videos with higher subject consistency exhibit less token variation.The authors interpret this as less optimization when the discriminator judges a video closer to real, preserving its original characteristics.Source: §5.3 · p. 8, Figure 4 · p. 8

5.4. Extensions: Image-to-Video Diffusion Model

• With MotionPrompt, DynamiCrafter’s appearance-detail differences and artifacts around objects are significantly mitigated during sampling.The vanilla image-to-video model already produces relatively consistent videos due to its reference image.Source: §5.4 · p. 8, Figure 5 · p. 8

• MotionPrompt uses a discriminator trained to distinguish real-video optical flow from generated-video optical flow, together with text optimization.Source: §6 · p. 8
• Qualitative and quantitative experiments demonstrated the method’s effectiveness.Source: §6 · p. 8
• Optimization adds slightly more generation time, but is limited to 10 to 15 steps, keeping costs low relative to performance gains.Source: §6 · p. 8
• Because its objective is not grounded in physics, MotionPrompt’s improvements may not always produce physically plausible outcomes.Source: §6 · p. 8

Supplementary Material

• Supplementary Section A details the training configurations for the discriminator and the inference configurations.Source: Supplementary Material · p. 11
• Supplementary Section A also presents the evaluation setups.Source: Supplementary Material · p. 11
• Supplementary Section B presents additional experimental analyses, while Section C provides further results from the framework.Source: Supplementary Material · p. 11

A.1. Training Details of the Discriminator

• The optical-flow discriminator was trained with batch size 32, SGD at learning rate 0.0005 and momentum 0.9, for approximately 20 epochs; the best-validation-loss checkpoint was selected.Source: §A.1 · p. 11

A.2. Hyperparameters for Evaluation

• Table 5 lists evaluation hyperparameters by model; the “# of frames” counts decoded frames used for optical-flow calculation, with six frames yielding three adjacent-frame flows.Source: §A.2 · p. 11, Table 5 · p. 11

A.3. User study

• In the Prolific A/B study, 100 participants compared 90 video pairs and chose the more natural, smoothly moving video, either A, B, or both.Source: §A.3 · p. 11

B. Additional Analysis

• The additional analyses focus primarily on AnimateDiff.Source: B. Additional Analysis · p. 11
• Repeated FreeInit tends to reduce detail and cause saturation, whereas one FreeInit application combined with MotionPrompt mitigates these issues while improving temporal quality.Source: Figure 6 · p. 11

B.1. Ablation on Token Optimization

• Adding three tokens improved some temporal-quality metrics but reduced dynamic degree and overall consistency, so the authors retained one token as the default.Source: §B.1 · p. 11
• Appending the learnable token to the prompt’s end was more effective than placing it at the front.Source: §B.1 · pp. 11–12
• Initializing the token with “the” yielded a smaller improvement than the final configuration but still outperformed baseline; cosine similarity to the initialization token decreased over time.Source: §B.1 · p. 12

B.2. Synergies with the Existing Method

• FreeInit initializes noise, while MotionPrompt guides generation through prompt optimization, making their mechanisms complementary.The section presents results from using both methods together.Source: §B.2 · p. 12
• Applying MotionPrompt after one FreeInit noise-initialization step improves temporal quality while relatively minimizing the video-quality compromise of multiple FreeInit steps.Repeated FreeInit steps sharply reduce Overall Consistency and Image Quality; videos may lose high-frequency details and show saturation.Source: §B.2 · pp. 12–13

B.3. Computational Cost

• Computational cost is reported as GPU memory usage and computing time averaged over 100 generations.Figure 8 is identified as the computational-cost figure.Source: §B.3 · p. 13, Figure 8 · p. 13

C. Additional Results

• Additional qualitative results include DynamiCrafter, an image-to-video model, as well as AnimateDiff, Lavie, and VideoCrafter2.The figures include additional results for each of these models.Source: C. Additional Results · p. 13, Figure 9 · p. 14, Figure 10 · p. 15
• Across the reported models, MotionPrompt enhances temporal quality without significantly altering generated content.The additional results include DynamiCrafter, AnimateDiff, Lavie, and VideoCrafter2.Source: C. Additional Results · p. 13
• In Lavie, the last frame often deviates from preceding frames, and MotionPrompt mitigates this issue to a large extent.The cited examples are rows 3, 6, and 9 in Figure 10.Source: C. Additional Results · p. 13, Figure 10 · p. 15