Source-linked AI summary
Mode Seeking meets Mean Seeking for Fast Long Video Generation
Shengqu Cai, Weili Nie, Chao Liu, Julius Berner, Lvmin Zhang, Nanye Ma, Hansheng Chen, Maneesh Agrawala, Leonidas Guibas, Gordon Wetzstein, Arash Vahdat
TL;DR
Generating coherent, high-fidelity videos spanning minutes is difficult because long-video training data is scarce and expensive. This paper decouples global coherence from local realism with two training heads, closing the fidelity–horizon gap while enabling fast, few-step long-video synthesis.
Problem
Long-video generation requires coherent, high-fidelity temporal streams, but long-video training remains data-constrained and expensive compared with short-video generation.
Method
A Decoupled Diffusion Transformer uses supervised flow matching on long videos for global structure and reverse-KL distribution matching to a frozen short-video teacher for local realism.
Results
The method jointly improves local sharpness, motion, and long-range consistency across quantitative, qualitative, and ablation results.
Takeaways & Limitations
The decoupled design closes the fidelity–horizon gap and provides fast, few-step inference for minute-scale video synthesis.
Takeaways & Limitations
The approach assumes access to an expert pretrained short-video teacher with an available velocity field.
Abstract
from arXiv · showhide
Scaling video generation from seconds to minutes faces a critical bottleneck: while short-video data is abundant and high-fidelity, coherent long-form data is scarce and limited to narrow domains. To address this, we propose a training paradigm where Mode Seeking meets Mean Seeking, decoupling local fidelity from long-term coherence based on a unified representation via a Decoupled Diffusion Transformer. Our approach utilizes a global Flow Matching head trained via supervised learning on long videos to capture narrative structure, while simultaneously employing a local Distribution Matching head that aligns sliding windows to a frozen short-video teacher via a mode-seeking reverse-KL divergence. This strategy enables the synthesis of minute-scale videos that learns long-range coherence and motions from limited long videos via supervised flow matching, while inheriting local realism by aligning every sliding-window segment of the student to a frozen short-video teacher, resulting in a few-step fast long video generator. Evaluations show that our method effectively closes the fidelity-horizon gap by jointly improving local sharpness, motion and long-range consistency. Project website: https://primecai.github.io/mmm/.
1. Introduction
Long-video generation is difficult because minute-scale sequences require temporal extrapolation, sustained narrative structure, and scarce long-form data, while mixed-duration training can sacrifice short-timescale fidelity. The proposed decoupled approach uses limited long videos for global coherence and a frozen short-video teacher for local realism, enabling fast few-step synthesis.
- Motivation: Minute-scale video generation requires new events, causal chains, and narrative structure beyond the short-clip horizon, making it fundamentally harder than image-resolution scaling.A one-minute video is described as temporal extrapolation rather than interpolation from a five-second clip.
- Data bottleneck: High-quality short clips are abundant online, but sustained minute-scale videos are much scarcer, making long-video training data-constrained and expensive.Short-video generators also encode web-scale diversity and high-fidelity modes that long-video training struggles to reproduce.
- Fidelity–horizon gap: Mixed-duration training can extend sequence length while sacrificing the sharp local dynamics, detail, and liveliness characteristic of expert short-video teachers.The model must relearn a high-fidelity short-video prior under constrained long-video data and compute.
- Method: The method decouples global coherence from local fidelity with a Decoupled Diffusion Transformer using supervised Flow Matching on limited long videos and Distribution Matching to a frozen short-video teacher.The two heads decode from a unified intermediate representation and target long-context structure versus local quality.
- Teacher alignment: Aligning every sliding-window segment through mode-seeking reverse KL lets the student inherit high-fidelity short-timescale priors without additional short-video data.The teacher alignment uses DMD-style distillation.
- Inference and outcome: Using only the Distribution Matching head at inference enables fast, few-step long-video synthesis with sharper local dynamics, improved motion, and consistent long-range context.The reported outcome is closure of the fidelity–horizon gap while preserving long-range consistency.
2. Related Work
Related work addresses long-video generation through length extrapolation, context persistence, and computational efficiency. Existing methods extend temporal horizons, retrieve or sparsify context, and optimize or prune spatiotemporal attention.
- Long video generation: Long-video generation methods extend pretrained models beyond their training horizon through noise rescheduling, temporal-frequency techniques, and related extensions.These approaches aim to improve length generalization and real-time rollout quality.
- Context learning and compression: Context-learning methods preserve long-range information by retrieving relevant history or using learned routers and policies to sparsify attention.Retrieval indexes include field of view, geometry, and view position, while routers select salient context chunks or token groups.
- Efficient video diffusion designs: Efficient video diffusion methods reduce computation with kernel-level optimizations, structured sparse attention, and training- or inference-time pruning.Examples include FlashAttention, sliding or tiling windows, radial masks, and learned sparse routing or token selection.
3. Method
The method decouples global long-horizon coherence from local fidelity with a shared long-context encoder and separate flow-matching and distribution-matching heads. Supervised flow matching learns minute-scale structure from long videos, while sliding-window reverse-KL alignment distills local high-fidelity modes from a short-video teacher for fast few-step generation.
- Local Distribution Matching: The DM head regularizes every on-policy sliding window with mode-seeking reverse-KL alignment to a frozen short-video teacher, concentrating student mass on high-fidelity teacher modes.The intractable reverse-KL gradient is approximated using teacher and student score/velocity differences on identically noised windows, following DMD/VSD-style training.
- Decoupled Training: Decoupling the heads prevents gradient interference between mean-seeking flow matching and mode-seeking teacher alignment while allowing both objectives to update the shared encoder.Each head receives only its corresponding training signal, preserving long-context learning while distilling short-video generation capability.
- Architecture: A shared video diffusion-transformer encoder with full-range temporal attention produces a unified spatiotemporal representation for both velocity heads.The encoder processes noisy long-video latents with conditioning and timestep information, then feeds two lightweight transformer decoders.
- Global Flow Matching: The FM head learns the global velocity field through supervised flow matching on full-length videos, anchoring long-horizon temporal dependencies and narrative structure to real trajectories.Gradients update both the shared encoder and FM head.
- Inference: The resulting model supports direct few-step inference with the DM head, avoiding multi-stage training and distillation while enabling fast minute-scale video generation.Local teacher alignment alone cannot provide minute-scale coherence, so global SFT supplies the complementary long-range signal.
4. Results
Results show that the decoupled design addresses complementary weaknesses of SFT-only and teacher-only baselines: it preserves long-range structure while restoring local realism. Qualitative comparisons and ablations support the necessity of separate global and local objectives, sliding-window teacher matching, and long-clip SFT.
- Evaluation: The evaluation uses 200 long-video prompts to generate 30-second videos, assessed with VBench-Long metrics and Gemini-3-Pro consistency.Metrics include subject and background consistency, motion smoothness, temporal flickering, dynamic degree, aesthetic quality, and imaging quality.
- Qualitative comparison: Teacher-only methods retain local realism for approximately 5 seconds but lack long-term narrative grounding and degrade during autoregressive minute-level rollouts.Their short-video teachers do not provide a mechanism for modeling long-range narrative structure.
- Analysis on our method: Our decoupled method uses global Flow Matching SFT as a long-context anchor and local sliding-window Distribution Matching to inject the teacher’s high-fidelity texture and motion priors.The two heads separately address long-scale prompts and temporal dependencies versus local realism.
- Qualitative comparison: SFT-only baselines preserve rough long-context structure but lose local realism, with washed-out textures, softened edges, and weakened scene continuity.The scarcity of high-quality long-video supervision limits SFT’s ability to maintain fidelity across diverse domains.
- Ablation study: Removing the decoupled dual-head design and jointly training one velocity predictor with SFT and teacher alignment causes the largest drop across all metrics.The ablation supports the claim that mean-seeking and mode-seeking objectives interfere when forced into a single predictor.
5. Conclusion
Mode Seeking meets Mean Seeking scales video diffusion from seconds to minutes despite scarce long-video data. Its Decoupled Diffusion Transformer separates long-range coherence learning from local distribution matching through a shared long-context encoder and two specialized heads.
- 5. Conclusion: The proposed paradigm targets scaling video diffusion from seconds to minutes under scarce long-video data.It is presented as a simple training paradigm for minute-scale video generation.
- 5. Conclusion: A Decoupled Diffusion Transformer uses a shared long-context encoder with separate objectives and two heads.The architecture separates the mean-seeking and mode-seeking training objectives.
- 5. Conclusion: The mean-seeking SFT flow-matching head learns minute-scale coherence from real long clips.This head is trained on real long-video clips to capture coherence at minute scale.
- 5. Conclusion: The mode-seeking sliding-window distribution-matching head aligns student windows to a frozen short-video teacher via reverse- divergence.The supplied passage ends after “reverse-,” so the divergence name is incomplete there.
A. Implementation Details
The model is trained with variable-length video batching and sequence parallelism to support efficient long-context training across A100 and GB200 GPUs.
- Training Infrastructure: Training uses dynamic length-based bucketing to reduce padding waste and idle time, with DeepSpeed Ulysses sequence parallelism for long contexts.The sequence-parallelism group size is 4 on A100 GPUs and 2 on GB200 GPUs.
B. Data
The method trains on over 100,000 videos from public datasets and filtered internet collections, spanning 10 seconds to minutes while capping training clips at 61 seconds. The dataset averages 31 seconds per video and excludes single-shot videos, with longer clips subsampled.
- The data combines all available Sekai videos, a MiraData subset, and randomly collected internet videos filtered to remove single-shot clips.
- More than 100k videos span durations from 10 seconds to minutes, with an average length of 31 seconds.
- Training clips have a temporal upper bound of 61 seconds, and videos exceeding it are subsampled.
C. Sliding Window DMD Implementation
Sliding-window DMD encounters a semantic mismatch in joint image-video latent spaces because middle-cropped windows begin with video latents while the teacher expects an image latent first.
- Sliding Window DMD Implementation: Middle-cropped sliding windows can begin with a video latent, conflicting with the teacher’s expectation that the first clip latent is an image latent.This boundary mismatch arises in video latent diffusion models whose latent spaces include both image and video frame latents for image-video joint training.
D. Gemini Evaluation Prompt.
The prompt instructs evaluators to score a video's semantic consistency from 0 to 100, assessing temporal coherence while penalizing unrealistic changes and static outputs. Responses must contain only one integer in the inclusive 0–100 range.
- Scoring criteria: Evaluators score semantic consistency on a 0-100 scale.The evaluation concerns a video generation result.
- Scoring criteria: Semantic consistency requires objects, identities, attributes, and the overall scene to remain coherent over time.Sudden unrealistic changes, identity swaps, content drift, and frame-to-frame contradictions are penalized.
- Motion requirement: Static or nearly frozen videos should receive a low score rather than a high consistency score.Trivial consistency without motion or temporal change does not demonstrate consistency under motion.
- Output format: The evaluator must return only a single integer between 0 and 100, inclusive.No other words are permitted.
E. Limitation and Future Work
The method is orthogonal to causal autoregressive approaches and suggests combining long-context bidirectional modeling with causal training or sampling.
- Future Work: The method is orthogonal to causal autoregressive methods.
- Future Work: A follow-up could use the method as a base model for causal autoregressive training.
- Future Work: Alternatively, the long-context bidirectional model could be distilled into a causal sampler by adding a causal attention mask during training.