PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis

Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, Diana Marculescu

arXiv:2312.17681v1cs.CVcs.MM

TL;DR

Video-to-video synthesis must maintain consistency across frames, but text-guided edits and imperfect optical flow make that challenging. FlowVid combines spatial conditions with flow-warped references, editing the first frame before propagating changes. It reports a 45.7% user-study preference rate and faster generation than three comparison methods.

  • Problem

    Text-guided video-to-video synthesis must maintain temporal consistency, while applying image-to-image models independently can produce pixel flickering.

    Source: §1 · pp. 1–2

  • Method

    FlowVid edits the first frame with an image-to-image model, then propagates edits using a video diffusion model conditioned on spatial controls and flow-warped frames.

    Source: §1 · p. 2

  • Results

    FlowVid’s user-study preference rate was 45.7%, compared with CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%), and it generated 120 frames in 1.5 minutes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The decoupled edit-propagate design supports stylization, object swaps, local edits, and autoregressive generation of lengthy videos.

    Source: §1 · p. 2

    FlowVid relies on the edited first frame being structurally aligned with the input, and large occlusions from fast motion can lead it to guess or hallucinate missing regions.

    Source: §5.5 · p. 8

Abstract

from arXiv · show

Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. However, the advancement of video-to-video (V2V) synthesis has been hampered by the challenge of maintaining temporal consistency across video frames. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video. Contrary to prior methods that strictly adhere to optical flow, our approach harnesses its benefits while handling the imperfection in flow estimation. We encode the optical flow via warping from the first frame and serve it as a supplementary reference in the diffusion model. This enables our model for video synthesis by editing the first frame with any prevalent I2I models and then propagating edits to successive frames. Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility: FlowVid works seamlessly with existing I2I models, facilitating various modifications, including stylization, object swaps, and local edits. (2) Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF, Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).

1. Introduction

FlowVid addresses temporal inconsistency in video editing by combining spatial conditions with optical-flow guidance that can tolerate estimation errors. It edits a first frame and propagates the edit, with reported gains in speed and user preference.

1. Introduction

  • Video editing with text prompts is difficult because independently applying image-to-image models can produce pixel flickering, while estimated optical flow may be inaccurate.Earlier spatial-temporal attention methods improve consistency but do not fully attain it.Source: §1 · p. 2
  • FlowVid uses flow-warped frames as temporal guidance and adds spatial conditions, such as depth maps, to rectify inaccurate flow rather than enforce it as a hard constraint.The paper’s example describes incorrect leg flow corrected by spatial controls.Source: §1 · p. 2, Figure 2 · p. 2
  • The model edits the first frame with a prevalent image-to-image model, then propagates edits with a trained video model; its decoupled design supports autoregressive generation of lengthy videos.The contributions also include stylization, object swaps, and local editing.Source: §1 · p. 2
  • 1.5 minutes generates 120 frames at 512×512, 3.1× faster than CoDeF, 7.2× faster than Rerender, and 10.5× faster than TokenFlow; FlowVid’s preference rate was 45.7%.The user-study comparison was CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).Source: §1 · p. 2

2. Related Work

Related work spans image-editing diffusion models, video diffusion approaches to frame consistency, and optical-flow-based edit propagation. FlowVid combines spatial controls with imperfect flow as supplementary guidance.

2.1. Image-to-image Diffusion Models

  • Image-to-image diffusion methods add spatial controls such as depth maps, but applying them independently to video frames can yield unsatisfactory consistency.ControlNet, T2I-Adapter, and Composer are examples of methods using spatial conditions.Source: §2.1 · p. 3

2.2. Video-to-video Diffusion Models

  • Video diffusion methods use cross-frame attention or semantic correspondences to improve consistency; FlowVid additionally trains with imperfect temporal flow alongside spatial controls.Other approaches include per-video image representations, which can degrade with large motion.Source: §2.2 · p. 3

2.3. Optical flow for video-to-video synthesis

  • Earlier flow-based methods use flow for pixel consistency, occlusion inpainting, or canonical-image guidance, generally treating estimated flow as accurate supervision; FlowVid instead uses it without rigid constraints.Ground-truth flow is typically unavailable in real-world videos, where pretrained models commonly estimate flow.Source: §2.3 · p. 3

3. Preliminary

The preliminaries describe latent diffusion’s noise-removal process and ControlNet’s use of spatial conditions to guide image generation.

3. Preliminary

  • Latent diffusion compresses an image into a lower-resolution latent, adds Gaussian noise over time, and trains a time-conditional U-Net to reverse that process.The latent has dimensions H/8 × W/8 × 4, and text prompt τ is also a condition.Source: §3 · p. 3
  • ControlNet converts spatial conditions such as edges or depth maps into latents and combines them with the noisy input latent to provide spatial guidance during generation.The condition latent interacts with the diffusion model at multiple feature resolutions.Source: §3 · p. 3

4. FlowVid

FlowVid combines spatial controls with optical-flow clues to synthesize temporally consistent videos while treating estimated flow as imperfect. Its design inflates an image diffusion model for video, trains with joint spatial-temporal conditions, and propagates edits from a modified first frame.

4.1. Inflating image U-Net to accommodate video

  • Spatial-temporal attention queries each frame using its features while drawing keys and values from the first and previous frames.The model adapts image-model layers for video and expands only the major U-Net; the ControlNet U-Net processes spatial conditions.Source: §4.1 · p. 4

4.2. Training with joint spatial-temporal conditions

  • FlowVid uses estimated optical flow as a soft condition alongside spatial controls, rather than enforcing potentially imperfect flow as a rigid constraint.Flow warps the first frame into later-frame references; occlusions can leave blank regions, while spatial conditions help guide structure.Source: §4.2 · p. 4

4.3. Generation: edit the first frame then propagate

  • Generation edits the first frame with an I2I model, then uses flow-warped edited frames and spatial conditions to guide video synthesis.The flow-warped video serves as a texture reference, while spatial controls regularize generation when flow is inaccurate.Source: Figure 3 · p. 4, §4.3 · p. 5
  • Autoregressive generation reuses each batch’s last frame as the next batch’s starting frame, while color calibration reduces progressive graying.Calibration matches each generated frame’s mean and variance to the first edited frame; Figure 4 contrasts uncalibrated gray drift with more stable color.Source: §4.3 · p. 5, Figure 4 · p. 5
  • DDIM-inversion self-attention features from the original-prompt video are reused during target-prompt generation to help preserve structure and motion.The method saves intermediate self-attention maps and substitutes their keys and values during generation.Source: §4.3 · p. 5

5. Experiments

Experiments compare FlowVid with representative V2V methods, evaluate its quality and runtime, and examine condition choices and limitations. Results favor FlowVid in user preference and speed, while ablations show how spatial and temporal conditions contribute.

5.1. Settings

  • The study trains on 100k Shutterstock videos and evaluates on DAVIS and Shutterstock videos with designed prompts; generation uses keyframes, RIFE interpolation, and canny-edge or depth controls.The training setup uses 512×512 crops and v-parameterization; evaluation includes 25 DAVIS videos and 50 Shutterstock videos.Source: §5.1 · pp. 5–6

5.2. Qualitative results

  • FlowVid’s qualitative comparisons report stronger prompt alignment and overall video quality, while baselines show flicker, blur, motion failures, or prompt-following errors.The examples include per-frame ControlNet, CoDeF, Rerender, and TokenFlow.Source: §5.2 · pp. 6–7, Figure 5 · p. 6

5.3. Quantitative results

  • FlowVid was preferred 45.7% of the time, versus CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%) in a study judging temporal consistency and text alignment.The study used 25 DAVIS videos and 115 prompts.Source: Table 1 · p. 7, §5.3 · p. 7
  • 1.5 minutes was FlowVid’s runtime for a 120-frame, 512×512 video, making it 3.1×, 7.2×, and 10.5× faster than CoDeF, Rerender, and TokenFlow, respectively.The comparison includes image processing, model operation, and frame interpolation.Source: §5.3 · p. 7, Table 1 · p. 7

5.4. Ablation study

  • Spatial-only conditions had a 9% winning rate against the full model, compared with 38% when flow-warped video was added and 42% after adding an occlusion mask.The full setting combines spatial controls, flow-warped video, a flow-occlusion mask, and the first frame; the first frame adds texture guidance when few original pixels remain.Source: §5.4 · pp. 7–8, Figure 6 · p. 7
  • Canny edges better preserve input structure for stylization, while depth maps allow more editing flexibility for object swaps.In examples, Canny preserves a panda’s mouth placement for an ink-painting edit, while depth avoids retaining panda eyes in a koala swap.Source: §5.4 · p. 8, Figure 7 · p. 8
  • Epsilon-prediction can introduce unnatural global color shifts across frames, whereas v-prediction avoids the reported gray shift.The comparison uses the same flow-warped video condition.Source: §5.4 · p. 8, Figure 8 · p. 8

5.5. Limitations

  • FlowVid’s results are limited when the edited first frame is structurally misaligned with the input or fast motion creates large occlusions.The model may propagate first-frame errors or guess—and sometimes hallucinate—content in missing regions.Source: §5.5 · p. 8, Figure 9 · p. 8

6. Conclusion

FlowVid combines spatial and temporal conditions to synthesize consistent videos while treating optical flow as a supplementary reference. Its decoupled edit-propagate design supports edits from existing image-to-image models and autoregressive generation of longer videos.

6. Conclusion

  • FlowVid uses joint spatial-temporal conditions, with optical flow as a supplementary reference rather than a strict constraint.The model adapts existing image-to-image models to edit the first frame and propagate edits to later frames.Source: §6 · p. 9
  • The decoupled edit-propagate design supports first-frame editing with existing image-to-image models and propagation to consecutive frames.The design also supports lengthy videos through autoregressive evaluation.Source: §6 · p. 9
  • Qualitative and quantitative comparisons with current methods highlight FlowVid’s efficiency and high quality.Source: §6 · p. 9

A. Webpage Demo

The section points readers to FlowVid’s demo webpage to view video results.

A. Webpage Demo

B.1. CLIP scores

The CLIP evaluation covers temporal consistency and prompt alignment across 116 DAVIS video-prompt pairs. FlowVid ranks highest in prompt alignment and second in temporal consistency, while TokenFlow leads temporal consistency but sometimes makes weaker edits.

B.1. CLIP scores

  • FlowVid ranks highest in prompt alignment and second in temporal consistency in the CLIP evaluation.The evaluation analyzes 116 video-prompt pairs from DAVIS.Source: §B.1 · p. 11
  • The evaluation measures temporal consistency as mean cosine similarity across sequential frame pairs and prompt alignment as mean cosine similarity between the prompt and all video frames.Source: §B.1 · p. 11
  • TokenFlow shows superior temporal consistency, but sometimes underperforms at modifying videos, leaving outputs close to the original input.Source: §B.1 · p. 11

B.2. Runtime breakdown

The runtime breakdown compares FlowVid and other methods on a 4-second, 120-frame, 512 × 512 video. FlowVid’s batch keyframe processing is faster than Rerender’s sequential processing, while its non-reference-based interpolation contrasts with Rerender’s reference-based approach.

B.2. Runtime breakdown

  • The runtime benchmark uses a 512 × 512 video with 120 frames, equivalent to four seconds at 30 FPS.FlowVid and Rerender use a keyframe interval of four.Source: §B.2 · p. 11
  • Figure 10 presents the runtime breakdown for generating a four-second, 512 × 512 video at 30 FPS on one A100-80GB GPU.Source: Figure 10 · p. 11
  • FlowVid generates 31 keyframes in 1.1 minutes, compared with Rerender’s 5.2 minutes.FlowVid processes 16 images simultaneously, while Rerender processes one image at a time.Source: §B.2 · p. 11
  • Rerender takes 5.6 minutes to interpolate 90 non-keyframes using reference-based Eb-Synth, while FlowVid uses non-reference-based RIFE interpolation.Source: §B.2 · p. 11
  • CoDeF and TokenFlow require per-video preparation: 3.5 minutes for CoDeF’s canonical-image reconstruction training and 10.4 minutes for TokenFlow’s 500-step DDIM inversion.Source: §B.2 · p. 11
Loading 2312.17681v1…