Source-linked AI summary
Following Motion for Sequential Modeling in Video Frame Interpolation
Jaehyun Park, Nam Ik Cho
TL;DR
VFI needs efficient long-range modeling, but fixed SSM scanning orders are poorly aligned with dynamic motion trajectories. MGMVFI uses optical-flow-guided serialization and contextual synthesis within an adapted Mamba architecture, achieving state-of-the-art performance on challenging benchmarks.
Problem
Fixed SSM scanning orders are motion-agnostic, causing semantically inconsistent aggregation and limiting modeling of dynamic VFI trajectories.
Method
MGMVFI uses optical flow for motion-guided serialization, contextual synthesis for unreliable regions, and an adapted Mamba architecture with modified EDFFN refinement.
Results
MGMVFI achieves state-of-the-art performance on challenging VFI benchmarks, particularly for complex and dynamic motions.
Takeaways & Limitations
Motion-guided serialization provides a motion-aware sequence-modeling direction for video interpolation.
Takeaways & Limitations
MGMVFI depends on external optical flow; flow errors can corrupt token ordering, and a single motion-adaptive serialization may underperform with multiple objects and small motions.
Abstract
from arXiv · showhide
State Space Models (SSMs) have surfaced as a promising architecture in Video Frame Interpolation (VFI), as they can capture long-range dependencies with linear computational complexity. However, their predefined scanning order limits their effectiveness in modeling the dynamic motion trajectories inherent in VFI problems. To tackle this challenge, we propose Motion-Guided Mamba for Video Frame Interpolation (MGMVFI), an adaptation of the selective state space model tailored explicitly for VFI. MGMVFI introduces Motion-Guided Serialization (MGS), which leverages optical flow to define a motion-adaptive 1D input order for the SSM. This aligns the causal state updates with semantically related tokens, enabling motion-consistent feature propagation, particularly for large and dynamic motions. Additionally, to mitigate the unreliable feature representations caused by inaccurate optical flow estimates, we introduce contextual synthesis that utilizes the surrounding spatial context for robust inter-frame feature synthesis. These components are seamlessly integrated within our tailored Mamba architecture, which also employs a lightweight refinement block to enhance local detail reconstruction at a reduced computational cost. Extensive experiments on standard VFI benchmarks demonstrate that MGMVFI achievesstate-of-the-artperformance,particularly on complex and dynamic motions, thereby establishing a new direction for sequence modeling in video interpolation.
1 Introduction
VFI synthesizes intermediate frames but requires models that capture long-range spatial and temporal dependencies efficiently. MGMVFI addresses motion-misaligned SSM scans with motion-guided serialization, contextual synthesis, and an adapted Mamba architecture.
- Motivation: VFI synthesizes intermediate frames between consecutive inputs for applications including frame-rate up-conversion, slow motion, and novel-view synthesis.Increasing temporal resolution improves visual smoothness and downstream video-task performance.
- Motivation: CNNs have limited receptive fields for large motions, while Transformers capture global dependencies with quadratic complexity that limits high-resolution scalability.These trade-offs motivate efficient long-range sequence modeling for VFI.
- Problem: Predefined horizontal, vertical, spiral, Z-, interleaved, and Hilbert scans are motion-agnostic and can misalign SSM sequences with video dynamics.Scan direction and motion magnitude affect sequential distances between related features, complicating trajectory estimation.
- Proposed Approach: MGS uses optical flow to dynamically serialize features along motion paths, placing semantically related patches adjacent for causal SSM updates.Patch-level interleaving keeps related features adjacent regardless of scan direction.
- Proposed Approach: Contextual synthesis replaces unreliable flow-region representations using valid surrounding spatial context, complementing MGS within the adapted Mamba architecture.The architecture also integrates a modified EDFFN to improve local texture and detail reconstruction at low cost.
2 Related Work
VFI research has progressed from CNN and Transformer approaches toward SSM-based sequence modeling, while generative methods target perceptual quality at higher computational cost. MGMVFI extends this SSM-based line with motion-guided serialization and contextual feature processing.
- CNN-based VFI: Early VFI methods predominantly used CNNs, including flow-based approaches that estimate motion and warp frames toward the intermediate timestep.Later refinements include intermediate-flow estimation, bilateral motion estimation, and contextual features.
- Transformer-based VFI: Transformers were adopted to address CNN receptive-field limits through attention, with EMA-VFI balancing performance and efficiency as a state-of-the-art baseline.Transformer-based kernel methods also addressed content-agnostic weighting.
- Generative VFI: Generative diffusion models improve perceptual quality in complex-motion scenarios but incur significant computational cost.EDEN combines a Transformer-based tokenizer with generative modeling for this direction.
- SSM-based Vision Models: SSMs have been adapted to image and video tasks through selective scanning, hierarchical bidirectional updates, and motion-aware trajectory modeling.These adaptations target global receptive fields, efficient sequence modeling, and complex temporal dynamics.
- SSM-based VFI: VFIMamba introduced S6-based VFI with interleaved frame scanning, while LCMamba used Hilbert-curve and 3D voxel-level scans to preserve spatial-temporal continuity.MGMVFI builds on this S6-based VFI line with motion-guided serialization.
3 Proposed Method
MGMVFI processes interpolated-frame features with motion-guided serialization, contextual synthesis for unreliable flow regions, and Mamba-based inter-frame modeling. Its pipeline uses optical flow to align feature sequences, repairs corrupted regions from spatial context, and feeds fused tokens into an SSM.
- Motion-Guided Feature Processing: MGMVFI extracts multiscale features and bidirectional optical flow, then processes them through motion-guided feature modeling before reconstruction.A shared CNN encoder produces F0 and F1, while a pretrained flow model estimates f0→1 and f1→0.
- Motion-Guided Serialization: MGS warps features to the intermediate grid so raster scans serialize tokens along estimated motion trajectories.This makes semantically related patches consecutive and lets the SSM focus on inter-frame feature modeling.
- Motion-Guided Serialization: Sequential Feature Similarity measures adjacent-token feature similarity, with higher S indicating smoother consecutive features and a simpler SSM modeling target.The metric averages an exponential distance-based similarity over the N-token sequence.
- Contextual Synthesis: A forward-backward flow consistency check identifies unreliable motion paths caused by occlusion, disocclusion, large motion, or brightness changes.The resulting binary masks flag regions where bidirectional flow agreement fails, including zero-vector failures under lighting variation.
- Contextual Synthesis: Masked adapters replace unreliable sampled features with surrounding valid context using a 7 × 7 hollow convolution, with a symmetric adapter for the opposite flow direction.Refined maps select sampled or synthesized features according to consistency masks, then add them to the original features.
- Inter-Frame Feature Modeling: The pipeline concatenates corresponding fused features into sequence tokens and feeds the resulting 1D sequence to a four-scan SS2D Mamba block.The SS2D block uses horizontal, vertical, and backward raster scans, aggregates their outputs, gates the input features, and applies a final linear layer.
4 Implementation Details
The model is trained on Vimeo-90K with cropped and augmented triplets, multiscale feature processing, and a composite reconstruction-plus-warping objective. Training uses AdamW, cosine learning-rate decay, and precomputed bidirectional RAFT flow.
- Training Data and Augmentation: Training uses Vimeo-90K triplets with 256 × 256 random crops and spatial, temporal, and rotational augmentations.Feature processing is repeated at ×8 and ×16 downsampled resolutions with S = 2 and N = 3.
5 Experiment Results
MGMVFI is evaluated against diverse VFI baselines and benchmarks, including challenging motion, high-resolution, qualitative, ablation, flow-sensitivity, and scan-order analyses. Results show strong reconstruction quality, competitive efficiency, and benefits from its proposed modules and motion-guided serialization.
- Quantitative Evaluation: MGMVFI achieves state-of-the-art performance on Vimeo-90K and UCF101 and on the Hard and Extreme SNU-FILM splits.On SNU-FILM Extreme, it exceeds VFIMamba by 0.07 dB.
- Efficiency and Scalability: 427.5 ms inference runtime demonstrates competitive efficiency despite motion-guided serialization.The EDFFN selectively refines high-frequency details in the frequency domain.
- Qualitative Evaluation: MGS produces coherent motion-path sequences that improve structural fidelity in rapid-motion qualitative comparisons.Prior fixed interleaved and Hilbert-curve scans are described as producing semantically inconsistent patches and structural distortion.
- Qualitative Evaluation: Contextual Synthesis and EDFFN improve background reconstruction and high-frequency texture detail, including a reported 0.14 dB EDFFN gain.The masked adapter uses surrounding valid spatial context to synthesize features for information gaps.
- Ablation Study: Progressive ablations raise SNU-FILM Hard PSNR from 29.80 dB with EDFFN to 30.24 dB with MGS and 31.04 dB with Contextual Synthesis.The full model also reaches 36.67 dB on Vimeo-90K and reconstructs complex occlusions and disoccluded backgrounds.
- Sensitivity and Scan Analysis: MGMVFI remains robust with lightweight optical-flow estimators, while MGS achieves a Sequential Feature Similarity of 0.46.The flow-estimator performance drop is approximately 0.12 dB, and MGS improves over Interleaved by 31%.
6 Limitations
MGMVFI remains constrained by its reliance on external optical flow and its single motion-adaptive serialization. These limitations can affect token ordering and reduce performance in scenes with multiple objects or small motions.
- External flow errors can corrupt token ordering and propagate through the SSM.Contextual synthesis helps but does not address the underlying flow-dependence problem.
- A single motion-adaptive serialization may underperform in scenes containing multiple objects and small motions.
- Joint flow refinement and multi-trajectory serialization are identified as future research directions.
7 Conclusion
MGMVFI redefines S6-based VFI around motion-adaptive serialization, contextual synthesis, and efficient detail refinement. The framework achieves state-of-the-art performance on challenging benchmarks and is presented as a foundation for broader video synthesis and understanding tasks.
- MGMVFI uses optical flow to create motion-adaptive serialization that preserves temporal coherence and moves beyond fixed scanning patterns.
- Contextual synthesis mitigates occlusions, while the optimized EDFFN block improves efficiency and restores fine details.
- MGMVFI achieves state-of-the-art performance on challenging video frame interpolation benchmarks.
- The authors position motion-guided serialization as a scalable foundation for broader video synthesis and understanding tasks.