Source-linked AI summary
MotionSpec: Spectral Trajectory Supervision for Motion-Consistent Video Generation
Ziqi Ni, Rui Li, Shiqi Jiang, Wei Zhou
TL;DR
Text-to-video generation can produce high-fidelity images without realistic motion, while standard objectives provide limited explicit motion supervision. MotionSpec aligns spectral trajectories and adjacent-frame optical flow; experiments report improved motion consistency and coherence while preserving visual fidelity.
Problem
High-fidelity text-to-video generation can still produce temporal discontinuities, inconsistent action progression, and structural distortions, while standard objectives lack explicit motion-correspondence supervision.
Source: §1 · p. 1Method
MotionSpec combines amplitude-and-phase alignment of dense anchor-relative trajectory spectra with adjacent-frame optical-flow alignment.
Source: §1 · p. 1Results
Combining LFC and STC yields OIS 56.53 and TCS 98.82, outperforming either component alone on these dimensions and achieving the highest average VMBench score.
Source: §3.2 · p. 3Takeaways & Limitations
MotionSpec improves motion smoothness, structural stability, and temporal coherence without sacrificing visual fidelity, with consistent gains across motion-oriented evaluation metrics.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Recent advances in text-to-video generation have enabled high-fidelity visual synthesis, yet realistic motion remains challenging. Generated videos may exhibit temporal discontinuities, inconsistent action progression, and structural distortions during complex movements. Even when individual frames appear realistic, the underlying motion may evolve in inconsistent or implausible ways. Standard generative objectives provide limited motion-specific supervision, leaving motion evolution insufficiently constrained. In this paper, we propose MotionSpec, a motion supervision framework centered on Spectral Trajectory Consistency (STC). STC constructs dense anchor-relative motion trajectories and transforms them into motion spectral volumes via a temporal Fourier transform. By aligning the spectral amplitude and phase of predicted and target trajectories, STC constrains both motion strength across temporal frequencies and the temporal organization of motion. To complement this trajectory-level supervision, we introduce Local Flow Consistency (LFC), which aligns consecutive-frame optical flow between predicted and target videos to stabilize local motion transitions. Experiments demonstrate that MotionSpec consistently improves motion consistency, temporal coherence, and plausibility while preserving visual fidelity.
1. INTRODUCTION
High visual quality in text-to-video generation does not ensure realistic motion, and conventional objectives lack explicit motion-correspondence supervision. MotionSpec adds spectral trajectory and local flow objectives; experiments report improved motion quality, temporal coherence, and structural stability while preserving visual appearance.
1. INTRODUCTION
- Conventional denoising and flow-matching objectives do not directly constrain motion correspondences, despite motion discontinuities and distortions in generated videos.These problems concern both motion magnitude and how motion evolves over time.Source: §1 · p. 1
- Generated videos show lower nonzero-frequency spectral amplitudes and more concentrated per-video amplitude distributions than real videos, indicating weaker trajectory variation.The largest absolute amplitude gaps occur at low temporal frequencies.Source: §1 · p. 1, Figure 1 · p. 1
- Spectral Trajectory Consistency aligns amplitude and phase in dense motion-trajectory spectra to constrain motion energy across frequencies and its temporal organization.The trajectories are anchor-relative and transformed into motion spectral volumes.Source: §1 · p. 1
- MotionSpec complements spectral supervision with Local Flow Consistency, which aligns adjacent-frame optical flow to stabilize local transitions.Experiments report improved motion quality, temporal coherence, and structural stability while preserving visual appearance.Source: §1 · pp. 1–2
2. METHOD
MotionSpec combines local adjacent-frame flow alignment with spectral supervision of anchor-relative trajectories. Its training objective adds both motion losses to flow matching, using the spectral terms to constrain motion across temporal frequencies and over longer horizons.
2.2. Local Flow Consistency
- LFC aligns adjacent-frame optical flow from predicted and target videos, with a soft mask derived from target-flow magnitude emphasizing moving regions.The flow representation is extracted with a frozen, differentiable optical-flow estimator.Source: §2.2 · p. 2, §2.1 · p. 2
2.3. Spectral Trajectory Consistency
- Adjacent-frame flow alone leaves longer-term motion evolution unconstrained, motivating anchor-relative trajectories that capture patterns such as motion decay, trajectory drift, and inconsistent long-term dynamics.The trajectory horizon L sets the future offsets from each anchor, and sliding anchors with stride s sample multiple temporal segments.Source: §2.3 · p. 2
- STC transforms anchor-relative trajectories into motion spectral volumes and aligns their amplitude and phase to constrain motion energy across frequencies and long-range temporal organization.Sliding anchors cover multiple temporal segments; phase supervision is adjusted because phase becomes unreliable when target spectral amplitude is near zero.Source: §2.3 · pp. 2–3
2.4. Training Objective
- The final objective combines short-term flow and long-term spectral trajectory supervision with flow matching, with separate weights controlling the two motion losses.The method also uses a timestep-dependent weight because optical flow estimated from predicted video is less reliable at highly noisy timesteps.Source: §2.4 · p. 3
3. EXPERIMENTS
Experiments compare MotionSpec against fine-tuning and objective ablations, with quantitative and qualitative results plus analyses of spectral components and time-domain supervision.
3.1. Experimental Setup
- Training uses 10K motion- and prompt-filtered OpenVid-1M videos covering human, object, and complex motion.The backbone is Wan2.1-T2V-1.3B, and experiments use 256 × 256 × 81 clips.Source: §3.1 · p. 3
3.2. Quantitative Results
- The combined LFC and STC model achieves the best overall performance, improving across all five VMBench dimensions over FM-only.FM-only raises the average VMBench score from 61.57 to 62.18; the full model reaches OIS 56.53 and TCS 98.82, improves appearance from 81.95 to 82.20, and raises PAS from 22.91 to 23.74.Source: §3.2 · p. 3, Table 1 · p. 3
3.3. Qualitative Results
- Qualitative examples show more consistent body orientation and motion, clearer limbs during running, and more stable clothing and hoop shapes than Wan2.1.The comparisons include riding toward the camera, running, and hula-hooping.Source: §3.3 · pp. 3–4
3.4. Ablation Studies
- With LFC enabled, amplitude or phase supervision improves motion metrics over LFC alone, and combining both terms performs best.Amplitude mainly improves smoothness and structural stability, while phase provides a larger temporal-coherence gain.Source: §3.4 · p. 4, Table 2 · p. 4
- Time-domain supervision improves motion and average scores over LFC alone, but STC raises the average score to 65.07.STC separates motion energy from temporal organization across frequency components, unlike point-wise matching at temporal offsets.Source: §3.4 · p. 4, Table 3 · p. 4
4. CONCLUSION
MotionSpec combines spectral trajectory supervision and local flow consistency to improve motion coherence and stability while preserving visual fidelity.
4. CONCLUSION
- STC supervises motion frequency structure and temporal organization beyond adjacent frames, while LFC stabilizes local transitions; together they improve smoothness, structural stability, and coherence without sacrificing visual fidelity.The paper reports consistent gains across motion-oriented evaluation metrics.Source: §4 · p. 4