Source-linked AI summary
MotionBridge: Dynamic Video Inbetweening with Flexible Controls
Maham Tanveer, Yang Zhou, Simon Niklaus, Ali Mahdavi Amiri, Hao Zhang, Krishna Kumar Singh, Nanxuan Zhao
TL;DR
Distant-frame inbetweening requires synthesizing missing content, while existing video-generation techniques may not give users fine control over intermediate frames. MotionBridge unifies trajectory, keyframe, mask, guide-pixel, and text controls using separate control generators and dual-branch embedders, with curriculum training. The authors report flexible, high-quality inbetweening and applications to image-to-video and text-to-video generation, while noting limits in trajectory alignment and mask-based 3D motion.
Problem
Distant-frame inbetweening must synthesize novel content, and users want results that follow their artistic expression rather than merely a possible interpolation.
Source: §1 · p. 2Method
MotionBridge unifies trajectory, keyframe, mask, guide-pixel, and text controls, using two generators, dual-branch embedders, and curriculum training to learn the controls.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors conclude that multimodal controls make inbetweening more dynamic, customizable, and contextually accurate, and show that the framework can be applied beyond inbetweening.
Source: §6 · p. 8, §5 · p. 8Motion trajectories must remain aligned with image content or motion may be overshadowed by the stronger image condition, and mask control is limited to 2D translation rather than 3D movement such as rotation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
By generating plausible and smooth transitions between two image frames, video inbetweening is an essential tool for video editing and long video synthesis. Traditional works lack the capability to generate complex large motions. While recent video generation techniques are powerful in creating high-quality results, they often lack fine control over the details of intermediate frames, which can lead to results that do not align with the creative mind. We introduce MotionBridge, a unified video inbetweening framework that allows flexible controls, including trajectory strokes, keyframes, masks, guide pixels, and text. However, learning such multi-modal controls in a unified framework is a challenging task. We thus design two generators to extract the control signal faithfully and encode feature through dual-branch embedders to resolve ambiguities. We further introduce a curriculum training strategy to smoothly learn various controls. Extensive qualitative and quantitative experiments have demonstrated that such multi-modal controls enable a more dynamic, customizable, and contextually accurate visual narrative.
1. Introduction
Video inbetweening must bridge missing information between distant frames, but a plausible interpolation alone may not follow a creator’s intent. MotionBridge proposes a unified framework for controllable large-motion inbetweening, using multiple controls and separate design choices to represent and learn them.
1. Introduction
- Distant input frames make motion estimation and compensation difficult because realistic interpolation must synthesize novel content missing between them.Users also seek results that follow their artistic expression, not just a possible interpolation.Source: §1 · p. 2
- MotionBridge supports diverse large motions with trajectories, keyframes, masks, guide pixels, and text in a unified controllable inbetweening framework.The authors describe it as an initial effort for controllable video inbetweening.Source: §1 · p. 2
- MotionBridge separates content controls such as masks from motion controls such as trajectories, encoding them with dual-branch embedders to reduce ambiguity.It also uses two generators to extract control signals and curriculum training to introduce controls sequentially.Source: §1 · p. 2
- The authors report quantitative and qualitative evaluations and practical applications, including controllable image-to-video generation and text-to-video quality improvement by reducing ambiguity.The supplied introduction gives no benchmark values for these outcomes.Source: §1 · p. 2
2. Related Work
Related work spans generative video methods, frame interpolation, and controllable video generation. MotionBridge distinguishes its aim as interpolating between two given frames with flexible multimodal controls.
2.1. Video Generation
- Keyframe-conditioned video-generation methods use initial frames to guide later frames, whereas MotionBridge targets interpolation between two given frames with flexible multimodal controls.The related-work passage also describes generative approaches including diffusion models and methods that use text or training-free guidance.Source: §2.1 · p. 2
2.2. Video Inbetweening
- As input frames grow farther apart in time or space, inbetweening must hallucinate information absent from the inputs, while users seek artistic control over plausible results.Earlier interpolation approaches still re-synthesize an intermediate frame from the input frames.Source: §2.2 · pp. 2–3
2.3. Motion Control
- Some controllable video-generation methods learn template motions through per-template fine-tuning, while others use inputs such as depth maps or sketches that can be difficult for average users to create.Trajectory-control methods permit object and camera motion, but the cited later approaches remain limited to single-image inputs.Source: §2.3 · p. 3
3. Method
MotionBridge combines motion and content guidance for controllable video inbetweening. It converts extracted motion and regional content into separate controls, embeds them through two branches, and introduces conditions progressively during training.
3. Method
- MotionBridge combines motion guidance such as trajectories with content guidance such as masks and guide pixels to customize video inbetweening.The paper uses a DiT-based model as its training backbone.Source: §3 · pp. 3–4
3.1. Preliminary
- The preliminary diffusion pipeline iteratively denoises a noisy video using a learned function conditioned on inputs.The passage describes the noisy video as X_T, the denoised output as X_0, and c as the conditions.Source: §3.1 · p. 4
3.2. Control Generation
- The control-generation design addresses large-motion ambiguity and artifacts with a Sparse Motion Generator for motion paths and an Augmented Frame Generator for visual context.Together, the two mechanisms are described as supporting controlled, natural-looking motion.Source: §3.2 · p. 4
3.2.1 Sparse Motion Generator
- The Sparse Motion Generator derives trajectories from dense optical flow and feature tracking, then Gaussian filtering expands the single-pixel paths into sparse trajectories.It maps motion direction and speed into RGB point controls so the representation can follow the keyframe input process in DiT.Source: §3.2.1 · p. 4, Figure 3 · p. 4
3.2.2 Augmented Frame Generator
- The Augmented Frame Generator extracts a masked region from a keyframe and translates it along a trajectory to create guide pixels appended to the keyframes.Users can set guide regions explicitly to place content; masks and guide pixels reduce the need for extensive trajectory drawing, and the content condition is randomly dropped out 20% of the time during training.Source: §3.2.2 · pp. 4–5
3.3. Curriculum Learning for Multi-modal Control with Dual-Branch Encoders
- A dual-branch encoder separately embeds content inputs, including keyframes and guide pixels, and motion inputs before feeding their representations to the transformer denoiser.The content and motion branches have similar structure, and the latent inputs are transformed into patches by embedders.Source: §3.3 · p. 5
- Training proceeds from image-only interpolation to optical-flow conditioning, then sparse motion inputs and guided pixels, after direct motion-input training was found to be ignored.The authors say this phased approach helps the model interpret limited motion information.Source: §3.3 · p. 5
- The authors motivate two branches by the difference between content and motion conditions and report testing a single-branch alternative.The passage identifies this as a design choice examined in an ablation.Source: §3.3 · p. 5
4. Experiments
MotionBridge is evaluated through qualitative comparisons, quantitative benchmarks, a user study, backbone generalization, and ablations. Results indicate strong motion control and visual quality, while the ablations highlight the roles of curriculum training, dual-branch embedders, and masks.
4. Experiments
- The evaluation compares generative quality and motion control against diffusion-based interpolation methods and FILM [26], using FVD, LPIPS [47], and a trajectory-based Motion metric.The Motion metric computes Fréchet Distance between paths extracted from generated optical flow and the input trajectory.Source: §4 · p. 5
4.1. Qualitative Results
- Qualitative examples show trajectory-guided object motion, multi-object control, smooth animation, prompt-adapted outcomes, and guide-pixel-conditioned interpolation.Comparisons report fewer distortions and artifacts than FILM [26], Dynamicrafter, TimeReversal [6], and SEINE [4].Source: §4.1 · pp. 5–6
4.2. Quantitative Evaluation
- MotionBridge produces comparable or better visual quality than recent interpolation methods and outperforms related works on motion control by a large margin.The quantitative evaluation randomly samples 100 videos from DAVIS and Objectron.Source: §4.2 · pp. 6–7, Table 1 · p. 7
4.3. User Study
- In a user study, participants assessed motion faithfulness and visual quality, and the authors report that MotionBridge follows motion while maintaining high-quality outputs.Ten results were shown side by side in random order for participant preference judgments.Source: §4.3 · p. 7, Figure 7 · p. 7
4.4. Generalizability
- OpenSora results are presented to demonstrate that the method can be applied to a different DiT structure.The authors also report quantitative results using OpenSora in Table 7.Source: §4.4 · p. 7, Figure 8 · p. 7
4.5. Ablation Study
- Training directly from sparse motions failed to integrate motion cues, while a single embedding branch introduced more artifacts than separating content and motion conditions.The authors also report that masks improve reliability and predictability and support more complicated motions; Figure 10 contrasts masked and unmasked movement.Source: §4.5 · pp. 7–8, Figure 9 · p. 8, Figure 10 · p. 8
5. Applications
MotionBridge supports applications beyond interpolation, including looping-video generation and animation from a single image. The examples show trajectory-guided continuous playback and plausible motion from a still frame.
5. Applications
- The model can integrate with existing text-to-video and image-to-video models to refine results with added controls.The paper presents looping-video generation and single-image animation as additional use cases.Source: §5 · p. 8
- With identical input frames, MotionBridge generates seamless looping videos that follow input motion trajectories for continuous playback.Figure 11 indicates that periodic motion can be produced by shortening the trajectory to less than the clip duration.Source: §5 · p. 8, Figure 11 · p. 8
- Although trained for inbetweening, MotionBridge can generate plausible motion from a single frame.Figure 12 shows an example combining trajectory and mask controls, as well as a trajectory-only example, without single-image-animation training.Source: §5 · p. 8, Figure 12 · p. 8
6. Conclusion
MotionBridge is a DiT-based framework for controllable video inbetweening that generates interpolated frames guided by keyframes, trajectories, and masks. Its use is constrained by trajectory-image alignment requirements and mask control limited to 2D translation.
6. Conclusion
- MotionBridge generates high-quality interpolated frames guided by keyframes, trajectories, and masks.The conclusion presents it as a DiT-based framework for controllable inbetweening.Source: §6 · p. 8
- The method’s versatility is demonstrated through extensive experiments and applications.Source: §6 · p. 8
- Trajectories must align with image content, and mask control is limited to 2D translation, excluding 3D movements such as rotation.The authors identify adding 3D transformations as a direction for future work.Source: §6 · p. 9