Source-linked AI summary
MultiCOIN: Multi-Modal COntrollable Video INbetweening
Maham Tanveer, Yang Zhou, Simon Niklaus, Ali Mahdavi Amiri, Hao Zhang, Krishna Kumar Singh, Nanxuan Zhao
TL;DR
Video inbetweening must bridge gaps between frames, yet existing controllable approaches offer limited ways to express diverse user intent and fine-grained motion. MultiCOIN combines trajectory, depth, target-region, and text controls in a DiT-based framework with separate content and motion branches. Experiments report improved trajectory alignment over Framer and multimodal versatility.
Problem
Existing inbetweening methods struggle with large, complex motion and offer limited ways to express user intent or finely control intermediate-frame details.
Source: §1 · p. 2Method
MultiCOIN adapts trajectories, depth, target regions, and text to a DiT-based video model, encoding content and motion controls in separate branches and training them in stages.
Source: §1 · pp. 2–3Results
MultiCOIN achieves significantly better trajectory alignment than Framer and shows multimodal versatility, with experiments covering multi-object trajectory and depth control and region- and text-guided content.
Source: §1 · p. 3Takeaways & Limitations
MultiCOIN supports flexible interpolation controls that can be used individually or in combination across a wide variety of use cases.
Source: §5 · p. 9Strong content conditioning can dominate and suppress motion cues, leaving trajectory alignment with image content as a challenge.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Video inbetweening creates smooth and natural transitions between two image frames, making it an indispensable tool for video editing and long-form video synthesis. Existing works in this domain are unable to generate large, complex, or intricate motions. In particular, they cannot accommodate the versatility of user intents and generally lack fine control over the details of intermediate frames, leading to misalignment with the creative mind. To fill these gaps, we introduce MultiCOIN, a video inbetweening framework that allows multi-modal controls, including depth transition and layering, motion trajectories, text prompts, and target regions for movement localization, while achieving a balance between flexibility, ease of use, and precision for fine-grained video interpolation. To achieve this, we adopt the Diffusion Transformer (DiT) architecture as our video generative model, due to its proven capability to generate high-quality long videos. To ensure compatibility between DiT and our multi-modal controls, we map all motion controls into a common sparse and user-friendly point-based representation as the video/noise input. Further, to respect the variety of controls which operate at varying levels of granularity and influence, we separate content controls and motion controls into two branches to encode the required features before guiding the denoising process, resulting in two generators, one for motion and the other for content. Finally, we propose a stage-wise training strategy to ensure that our model learns the multi-modal controls smoothly. Extensive qualitative and quantitative experiments demonstrate that multi-modal controls enable a more dynamic, customizable, and contextually accurate visual narrative.
1. Introduction
Video inbetweening needs to honor users’ creative intent, but existing controllable methods offer limited control over complex motion and intermediate-frame details. MultiCOIN combines motion, depth, region, and text controls, adapting them to DiT and separating motion from content guidance.
1. Introduction
- Users need interpolations that respect artistic intent, yet prior controllable work focused on trajectories and lacked depth transitions and region-based localization.Text prompts also offer a means of edit control, while depth and target regions can help resolve ambiguities in motion.Source: §1 · p. 2
- MultiCOIN combines trajectories, depth, target regions, and text to support precise paths, depth cues, motion localization, and semantic guidance.The framework aims to balance flexibility, ease of use, and precision in fine-grained interpolation.Source: §1 · p. 2
- DiT’s patch-based, temporally compressed representation disrupts the spatial correlations of raw motion controls and complicates alignment of content at different times.MultiCOIN maps controls into the video/noise input domain; input frames and target regions are placed at designated temporal positions with masks.Source: §1 · p. 2
- MultiCOIN separates content and motion controls into two branches before denoising, and gradually trains from denser, concrete controls to sparser, higher-level ones.The authors report greater training stability and robustness while preserving motion fidelity and content consistency.Source: §1 · p. 3
- MultiCOIN achieves significantly better trajectory alignment than Framer and demonstrates versatility from complementing trajectories with additional controls.The reported experiments also cover multi-object control using trajectory and depth, with content guided by keyframes and target regions.Source: §1 · p. 3
2. Related Work
Video generation research has explored multiple generative architectures and conditioning strategies. Video inbetweening methods address frame synthesis, while controllable inbetweening adds user-guided motion and complementary controls.
2.1. Video Generation
- Video generation research spans GANs, temporally aware networks, diffusion models, and keyframe- or text-conditioned approaches; MultiCOIN targets interpolation between two frames with flexible multimodal controls.Related examples include Stable Video Diffusion [2], Tune-a-video [43], and keyframe-guided methods [8] [18] [48].Source: §2.1 · p. 3
2.2. Video Inbetweening
- Earlier inbetweening methods use optical flow or adaptive kernels, while Framer [40] uses motion trajectories; MultiCOIN reports better trajectory alignment and combines complementary controls.The paper frames controllable inbetweening as a way to produce outputs that follow users’ artistic expression.Source: §2.2 · p. 3
3. Method
MultiCOIN uses a DiT video-diffusion backbone with point-based motion controls and regional content guidance. Its generators, separate content and motion branches, and staged training organize these controls for interpolation.
3. Method
- The DiT-based framework combines trajectory and depth controls, target regions and masks, and text guidance through modules and dual-branch embedders for motion and content.Users may provide motion and content guidance individually or together.Source: §3 · pp. 3–4, Figure 2 · p. 4
3.1. Preliminary
- DiT iteratively denoises noisy video, and its transformer backbone models long-range dependencies and global context.Training uses a diffusion loss measuring mean squared error between predicted and input noise.Source: §3.1 · p. 4
3.2. Control Generation
- Large-motion control uses sparse motion-depth paths and augmented frames to add visual context for natural-looking motion.The generators address ambiguity and artifacts in large-motion interpolation through motion paths and additional visual context.Source: §3.2 · p. 5
3.2.1. Sparse Motion-Depth Generators
- The Sparse Motion-Depth Generator converts flow and depth to RGB, samples them along high-motion trajectories, and expands sparse controls with Gaussian flow and disk depth filters.For single-point depth inputs, corner anchors at three multiples above and below the mean provide global depth context.Source: §3.2.1 · p. 5, Figure 3 · p. 5
3.2.2. Augmented Frame Generator
- Augmented Frames add target regions as a content “nudge,” which can accompany trajectories or be used alone to localize motion.Training creates target regions from trajectories and optical-flow segmentation; binary masks distinguish valid from invalid image regions.Source: §3.2.2 · pp. 5–6
3.3. Stage-wise Training with Dual-Branch Encoders
- Dual-branch encoders separately encode keyframes, target regions, and masks as content and motion/depth points as motion; staged training introduces conditions gradually.Training progresses from image-only interpolation to dense flow and depth, then sparse motion inputs, and finally guided pixels; direct sparse-input training struggled to localize motion.Source: §3.3 · p. 6
4. Experiments
MultiCOIN is evaluated through qualitative and quantitative comparisons, with results indicating stronger motion control than Framer alongside comparable or better visual quality. The experiments also examine its multimodal controls and the effects of stage-wise training and dual-branch encoders.
4. Experiments
- Evaluation compares trajectory control with Framer, uses SIFT-matched points for linear paths, and assesses quality and motion on DAVIS and UCF Sports.The metrics are SSIM, FVD, LPIPS, and a flow-derived Fréchet Distance Motion metric.Source: §4 · pp. 6, 8
4.2. Qualitative Evaluation
- Against Framer, MultiCOIN produces smoother transitions with fewer distortions and artifacts; the comparison also illustrates depth and text controls.The authors attribute stronger spatio-temporal alignment to embedding motion in the video latent space rather than introducing it indirectly through ControlNet.Source: Figure 6 · p. 8, §4.2 · p. 8
4.1. Qualitative Results
- MultiCOIN combines trajectory, depth, text, and target-region controls; depth and trajectories together enable 2D translation with depth variation.The qualitative examples also show more than two input frames and image deformation.Source: Figure 5 · p. 7, §4.1 · p. 8
- Results with more than two input frames are shown with and without motion control, and image deformation is presented as another application.These examples appear in Figures 7 and 8.Source: Figure 7 · p. 8, §4.1 · p. 8, Figure 8 · p. 8
4.3. Quantitative Evaluation
- On DAVIS and UCF, MultiCOIN’s motion metric clearly improves over Framer; visual quality generally matches or surpasses it, except for DAVIS-SSIM (Table 1).The reported comparison uses Framer under trajectory control.Source: §4.3 · pp. 8–9, Table 1 · p. 9
4.4. Ablation Study
- Direct training on sparse motion and depth inputs keeps FVD and LPIPS on par with stage-wise training, but mislocalizes motion and misinterprets depth.The reported results also say motion control deteriorates severely without stage-wise training, despite preserved overall frame fidelity.Source: §4.4 · p. 9, Figure 9 · p. 9, Table 2 · p. 9
- Replacing the dual-branch design with one concatenated-condition branch increases artifacts, reduces motion control, and raises FVD.The ablation compares a single branch that concatenates conditions with the input noise against the dedicated content and motion branches.Source: §4.4 · p. 9, Figure 9 · p. 9, Table 2 · p. 9
5. Conclusion
MultiCOIN is a DiT-based framework for controllable inbetweening using trajectories, depth, target regions, and text prompts, individually or in combination. Experiments demonstrate versatility, while strong content conditioning can suppress motion cues.
5. Conclusion
- MultiCOIN uses a DiT-based framework to generate interpolated frames conditioned on trajectories, depth, target regions, and text prompts, individually or in combination.The conclusion describes experiments as demonstrating versatility and effectiveness across varied use cases.Source: §5 · p. 9
- Strong content conditioning can dominate and suppress motion cues; the authors propose lightweight preprocessing to better balance the controls.The proposed direction aims to preserve user intent while maintaining quality.Source: §5 · p. 9
- The conclusion reports qualitative and quantitative experiments demonstrating versatility and effectiveness across a wide variety of use cases.Source: §5 · p. 9