Source-linked AI summary
FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis
Wonjoon Jin, Qi Dai, Chong Luo, Seung-Hwan Baek, Sunghyun Cho
TL;DR
Video diffusion models have limited user-defined camera control, and detailed camera-parameter methods require frame-level labels. FloVD generates optical flow and then conditions video synthesis on it; experiments report accurate camera control and natural object motion compared with prior approaches.
Problem
Video diffusion models often lack user-defined camera control, while detailed camera-control methods require ground-truth camera parameters for every frame.
Source: §1 · pp. 1–2Method
Results
Extensive evaluation demonstrates high-quality video generation with accurate camera control and natural object motion, outperforming previous approaches across metrics.
Source: §1 · p. 2, §5.3 · p. 7Takeaways & Limitations
Optical flow lets FloVD use arbitrary training videos without ground-truth camera parameters while leveraging background motion for detailed camera control.
Source: §6 · p. 8Errors in the object-motion or semantic-segmentation models may result in unnatural synthesized object motion.
Source: §6 · p. 8
Abstract
from arXiv · showhide
We present FloVD, a novel video diffusion model for camera-controllable video generation. FloVD leverages optical flow to represent the motions of the camera and moving objects. This approach offers two key benefits. Since optical flow can be directly estimated from videos, our approach allows for the use of arbitrary training videos without ground-truth camera parameters. Moreover, as background optical flow encodes 3D correlation across different viewpoints, our method enables detailed camera control by leveraging the background motion. To synthesize natural object motion while supporting detailed camera control, our framework adopts a two-stage video synthesis pipeline consisting of optical flow generation and flow-conditioned video synthesis. Extensive experiments demonstrate the superiority of our method over previous approaches in terms of accurate camera control and natural object motion synthesis.
1. Introduction
Existing video diffusion models offer limited user-defined camera control, while more detailed approaches require camera-labeled training data. FloVD uses optical flow to represent camera and object motion, supporting detailed control and training on arbitrary videos.
1. Introduction
- Video diffusion models often lack user-defined camera controls; text- or stroke-based methods [6] [23] [26] [33] [39] mainly support zoom and pan.Source: §1 · p. 1
- Camera-parameter methods [2] [9] [16] [32] [33] [34] [35] [40] [42] offer detailed control but require frame-level ground-truth parameters, restricting datasets and limiting generalization.Such datasets are challenging to acquire and often consist primarily of static scenes; these methods can produce unnatural object motion and inaccurate camera control.Source: §1 · pp. 1–2
- Optical flow can be estimated directly from videos, avoiding ground-truth camera labels; background flow also encodes 3D correlations across viewpoints for detailed control.Source: §1 · p. 2
- Given an image and camera parameters, FloVD generates future frames along a desired trajectory through optical-flow generation and flow-conditioned video synthesis.Source: §1 · p. 2
- FloVD separately generates background camera flow from camera parameters and image structure, and object flow from an object-motion model, then combines them.Source: §1 · p. 2
- Extensive evaluation demonstrates FloVD’s ability to generate high-quality videos with accurate camera control and natural object motion.Source: §1 · p. 2
2. Related Work
Camera-controllable video generation ranges from coarse text- or stroke-based controls to detailed parameter-driven controls that depend on frame-level camera annotations. FloVD uses optical flow to avoid those annotations, while prior flow-based synthesis pipelines focus on temporal coherence rather than camera control.
2. Related Work
- Text- and stroke-based methods [6] [23] [26] [39] offer basic camera control through high-level instructions such as zoom and pan, but lack detailed control for user-defined specific camera motions.Source: §2 · p. 2
- Detailed camera-control methods [2, 9, 33, 35, 37, 42] condition on ground-truth camera parameters for every frame, often restricting training datasets and generalization.Source: §2 · p. 2
- FloVD instead uses optical-flow maps, which can be estimated from arbitrary videos, to support training without ground-truth camera parameters.Source: §2 · p. 2
- Prior two-stage flow-based synthesis methods [7] [13] [17] [19] [21] [41] target temporal coherence, not camera control, and do not separate camera from object motion.Source: §2 · p. 2
3. FloVD Framework
FloVD turns an input image and camera trajectory into video through two stages: generating camera-object optical flow, then synthesizing video conditioned on that flow. Camera flow represents viewpoint motion, while object flow represents moving objects.
3. FloVD Framework
- FloVD’s two stages generate camera and object flow maps, integrate them, and condition the video synthesis model on the result.The synthesized video starts from the input image and follows the supplied camera trajectory.Source: §3 · p. 3
3.1. Flow Generation
- Camera flow is computed by warping image-derived 3D coordinates with camera parameters, while OMSM generates object flow maps independent of background motion.OMSM uses a latent video diffusion model to denoise latent object-motion features into flow maps.Source: §3.1 · p. 3
- Flow integration combines camera and object motion using an object mask, but omitting object z-motion makes the integrated flow physically inaccurate; FVSM can still synthesize natural motion from noisy flows.The mask removes incorrectly synthesized flow vectors in static regions.Source: §3.1 · p. 4
3.2. Flow-Conditioned Video Synthesis
- FVSM encodes camera-object flow at multiple levels and adds those embeddings to denoising U-Net features to synthesize video frames.The U-Net also receives the encoded input image and a noisy latent video volume.Source: §3.2 · p. 4
4. FloVD Training
FloVD trains OMSM and FVSM separately using optical-flow maps, without requiring ground-truth camera parameters. OMSM’s two-stage training reduces camera motion in synthesized object flows, while FVSM learns to condition video synthesis on flow.
4.1. Training Datasets
- The full dataset contains 500K clips, while its curated subset contains around 100K clips without camera motion; OMSM uses both datasets and FVSM uses only the full dataset.The curated subset is selected by detecting static background regions and thresholding their average optical-flow magnitude.Source: §4.1 · p. 4
4. FloVD Training
- Training estimates optical flow for each clip and uses it as ground-truth object flow for OMSM and camera-object flow for FVSM.FloVD trains OMSM and FVSM separately using optical-flow maps, without requiring ground-truth camera parameters.Source: §4.1 · p. 4, §4 · p. 4
4.2. Training Object Motion Synthesis
- After OMSM’s second training stage, synthesized object-flow maps show minimal camera motion, unlike the camera motion remaining after the first stage.The first stage trains on the full dataset; the second fine-tunes on the curated dataset without camera motion.Source: §4.2 · pp. 4–5
4.3. Training Flow-Conditioned Video Synthesis
- FVSM starts from a pretrained video diffusion model and trains only its flow encoder, using denoising score matching.At inference, camera-object flow is synthesized through 3D warping and OMSM; training-time optical flows and inference-time flows both represent camera and object motions as flow vectors.Source: §4.3 · p. 5
5. Experiments
FloVD is evaluated for camera control and video synthesis quality, then analyzed through component ablations, flow-conditioned synthesis, and application examples. Across the comparisons, it combines camera control with natural object motion and supports video editing and cinematic camera effects.
5.1. Implementation Details
- FVSM synthesizes 14 frames at 320 × 576 resolution; FVSM and OMSM use separate training schedules, and FVSM uses quadratic timestep sampling.FVSM trains for 50K iterations with 16 clips per batch; OMSM trains for 100K iterations, then receives 50K iterations of fine-tuning, using 8 flow maps per batch.Source: §5.1 · p. 5
5.2. Evaluation Protocol
- Camera control is evaluated by estimating camera parameters from synthesized videos and comparing them with the input parameters on 1,000 RealEstate10K test clips.The reported measures are mean rotation error, mean translation error, and mean camera-extrinsic-matrix error.Source: §5.2 · p. 5
- Video quality evaluation uses FVD, FID, and IS across seven camera trajectories on Pexels-random and object-motion subsets.The small, medium, and large motion subsets are categorized by average object-flow magnitudes below 20 pixels, between 20 and 40 pixels, and above 40 pixels, respectively.Source: §5.2 · pp. 5–6
5.3. Comparison
- FloVD trained on RealEstate10K outperforms MotionCtrl and CameraCtrl across all camera-control metrics.The comparison also reports accurate camera control for CameraCtrl and FloVD, unlike MotionCtrl, which does not reliably follow input trajectories.Source: Table 1 · p. 6, §5.3 · p. 6
- FloVD reports better sample quality on Pexels-random and better object-motion synthesis on Pexels-small, Pexels-medium, and Pexels-large than previous methods.CameraCtrl degrades on large object motions, while MotionCtrl can score well on FVD when generated viewpoints remain close to the input despite visual artifacts.Source: Table 2 · p. 7, §5.3 · p. 7
5.4. Analysis
- Adding OMSM enables object motion but can introduce artifacts; the final model with larger-scale training data produces natural-looking motion without noticeable artifacts.The ablation reports that each component consistently improves Pexels-random evaluation metrics.Source: §5.4 · pp. 7–8, Table 3 · p. 8
- Flow-conditioned synthesis produces realistic-looking results close to 3D-warped images while avoiding their distortions and holes.The warped images can still provide useful camera-control hints despite those artifacts.Source: §5.4 · p. 8
5.5. Applications
- The framework supports temporally consistent video editing at no extra cost and cinematic controls such as dolly zoom.For editing, FVSM uses flow maps from the input video and an edited first frame; the dolly-zoom example keeps the subject a similar size while the background converges inward.Source: §5.5 · p. 8
6. Conclusion
FloVD uses optical flow to represent camera and object motion, enabling detailed camera control and natural video synthesis without ground-truth camera parameters. The authors note that object-motion and segmentation errors can still produce unnatural motion.
6. Conclusion
- FloVD represents camera and object motions with optical-flow maps, enabling arbitrary training videos without ground-truth camera parameters and detailed control from background motion.Source: §6 · p. 8
- Extensive experiments demonstrate realistic video synthesis with natural object motion and accurate camera control.Source: §6 · p. 8
- Errors from object-motion synthesis or semantic segmentation may produce unnatural motion; point prompts can alleviate segmentation-estimation errors.Source: §6 · p. 8