Source-linked AI summary
Motif-Video 2B: Technical Report
Junghwan Lim, Wai Ting Cheung, Minsu Ha, Beomgyu Kim, Taewhan Kim, Haesol Lee, Dongpin Oh, Jeesoo Lee, Taehyun Kim, Minjae Kim, Sungmin Lee, Hyeyeon Cho, Dahye Choi, Jaeheui Her, Jaeyeon Huh, Hanbin Jung, Changjin Kang, Dongseok Kim, Jangwoong Kim, Youngrok Kim, Hyukjin Kweon, Hongjoo Lee, Jeongdoo Lee, Junhyeok Lee, Eunhwan Park, Yeongjae Park, Bokki Ryu, Dongjoo Weon
TL;DR
Video generation research has largely relied on massive datasets, parameter counts, and compute, raising whether strong quality is possible at a much smaller budget. Motif-Video 2B separates generation roles architecturally and combines that design with efficient training mechanisms. It reaches 83.76% on VBench, surpassing Wan2.1-14B with substantially fewer parameters, data, and compute.
Problem
Leading video generation models rely on massive datasets, parameter counts, and compute, while shared pathways can cause interference among text alignment, temporal consistency, and detail recovery.
Method
Motif-Video 2B uses a three-stage role-separated backbone, Shared Cross-Attention for long-sequence text control, and TREAD plus early REPA with a frozen video encoder for efficient training.
Results
83.76% on VBench surpasses Wan2.1-14B using fewer than 10M clips, under 100,000 GPU hours, and 7× fewer parameters.
Takeaways & Limitations
The results support architectural specialization and efficiency-oriented training as a way to narrow or exceed the quality gap associated with much larger video models.
Takeaways & Limitations
The model occasionally produces sub-object artifacts, including distorted anatomy, degraded body structure, and attribute leakage, which the authors primarily attribute to data coverage.
Abstract
from arXiv · showhide
Training strong video generation models usually requires massive datasets, large parameter counts, and substantial compute. In this work, we ask whether strong text-to-video quality is possible at a much smaller budget: fewer than 10M clips and less than 100,000 H200 GPU hours. Our core claim is that part of the answer lies in how model capacity is organized, not only in how much of it is used. In video generation, prompt alignment, temporal consistency, and fine-detail recovery can interfere with one another when they are handled through the same pathway. Motif-Video 2B addresses this by separating these roles architecturally, rather than relying on scale alone. The model combines two key ideas. First, Shared Cross-Attention strengthens text control when video token sequences become long. Second, a three-part backbone separates early fusion, joint representation learning, and detail refinement. To make this design effective under a limited compute budget, we pair it with an efficient training recipe based on dynamic token routing and early-phase feature alignment to a frozen pretrained video encoder. Our analysis shows that later blocks develop clearer cross-frame attention structure than standard single-stream baselines. On VBench, Motif-Video~2B reaches 83.76\%, surpassing Wan2.1 14B while using 7$\times$ fewer parameters and substantially less training data. These results suggest that careful architectural specialization, combined with an efficiency-oriented training recipe, can narrow or exceed the quality gap typically associated with much larger video models.
1 Introduction
Motif-Video 2B asks whether architectural specialization and efficient training can make competitive video generation accessible below the scale of leading open models. It separates conflicting generation roles and combines this design with mechanisms for long-context text alignment, temporal structure, and compute-efficient training.
- Leading open video models use hundreds of millions of curated clips and 5B–14B parameters, concentrating competitive training among few groups.
- Video generation must jointly satisfy prompt following, cross-frame consistency, and fine-detail recovery, creating objective interference when these goals share capacity.
- Motif-Video 2B separates early text-video fusion, joint representation learning, and detail reconstruction across a three-stage backbone.
- 83.76% on VBench surpasses Wan2.1-14B with fewer than 10M clips, under 100,000 GPU hours, and 7× fewer parameters.
- Shared Cross-Attention reuses self-attention key–value weights to stabilize text–video alignment when long sequences dilute text-token influence.
- DDT and TREAD support the design through inter-frame attention structure and dynamic token routing, while early REPA with a V-JEPA teacher improves training efficiency.
2 Related Work
Related work establishes large-scale video generation as the prevailing regime while showing that modality-aware architectures and efficient diffusion training can reduce computational demands. Motif-Video 2B builds on these architectural and optimization directions for video generation.
- Production-scale video generation: Open video models commonly train on hundreds of millions of clips with 5B–14B parameters, while proprietary systems appear comparably or more heavily scaled.
- Video and image diffusion transformer architectures: Dual-stream and single-stream diffusion transformer designs preserve modality-specific processing early before merging streams for joint generation.
- Efficient training for diffusion models: REPA aligns early hidden states with frozen visual encoders, while TREAD routes selected tokens from shallow to deep layers to reduce training cost.
3 Model Architecture
Motif-Video 2B organizes its backbone around separate responsibilities for modality fusion, joint text-video representation learning, and detail reconstruction. Shared Cross-Attention addresses text dilution in long joint sequences, while the DDT decoder exhibits stronger local inter-frame attention.
- Backbone design: The architecture assigns early fusion, joint representation learning, and detail reconstruction to separate dual-stream, single-stream, and DDT stages.It uses 12 dual-stream layers, 16 single-stream layers, and 8 DDT decoder layers.
- Modality fusion: Dual-stream layers preserve modality-specific processing before text and video tokens exchange information through cross-attention.This avoids premature sharing of attention capacity between modalities during early processing.
- Joint representation: Single-stream layers merge text and video tokens for joint representation learning, but long sequences weaken text influence under joint-token competition.As video tokens become much more numerous than text tokens, their aggregate attention share grows relative to text.
- Detail reconstruction: The DDT decoder delegates high-frequency detail reconstruction to final layers and shows stronger attention between temporally adjacent frames than dual- or single-stream layers.This observed pattern is consistent with a role focused on temporal consistency, although the paper leaves the contribution of DDT design versus depth unresolved.
- Long-context text conditioning: 720p training produces measurable degradation in prompt following and semantic alignment as the video-to-text token imbalance increases.The paper reports this failure mode as largely absent at lower resolutions.
- Shared Cross-Attention: Sequential refinement generalizes dilution correction by posing a second text query from the self-attention output rather than merely renormalizing existing attention.The method appends lightweight cross-attention after self-attention in each single-stream block.
O · Attn(Q, K, V), (4)
Shared Cross-Attention reuses self-attention’s text key and value geometry while learning a new query for sequential refinement. Zero-initialization preserves the pretrained block initially, but stable integration depends on grounding K and V in the existing representation manifold.
- Design asymmetry: Shared Cross-Attention reuses the enclosing self-attention layer’s K and V projections while introducing a learned query projection for the post-self-attention video state.The design keeps text keys and values on the model’s established representational manifold while allowing a distinct refinement query.
- Key-value sharing: The shared K and V projections preserve compatibility with the residual stream because self-attention already trained text values to contribute to joint representations.The paper explicitly distinguishes this geometric grounding from parameter saving.
- Query projection: A learned query is necessary because the post-self-attention video state contains local context that the pre-attention input does not.Reusing self-attention’s query would ask the same question again and undermine sequential refinement.
- Initialization: Zero-initializing Wcross_O makes the augmented block identical to the base block at step 0 and lets the cross-attention contribution grow during training.The paper treats this as an optimization-stability choice rather than complete geometric sharing.
- Stability comparison: After 1,000 matched training steps, the SkyReels-V4-style variant collapses to near-black incoherent outputs, whereas Shared Cross-Attention continues producing coherent scenes.Both variants began from the same pretrained checkpoint and used zero-initialized output projections.
- Scope of evidence: The supported claim is narrower than an eventual performance comparison: grounding K and V stabilizes a new module interfacing with an already-trained self-attention pathway.The paper notes that a SkyReels-V4-style module trained from scratch might recover.
4 Training Strategy
Motif-Video 2B uses a compute-conscious training strategy that combines progressive image–video training, early representation alignment, token routing, and iterative recipe refinement. The full recipe targets efficient 2B-scale video training while addressing quality and cost constraints.
- The training strategy is designed for a tight compute budget, requiring each iteration to maximize learning efficiency and measurable progress.
- Representation alignment: The recipe front-loads representation learning with early REPA alignment to a frozen visual encoder, then removes the objective before it becomes a capacity bottleneck.V-JEPA is used as the video teacher because its latent representations capture temporal structure.
- Progressive curriculum: Training begins with 144p image pre-training, then progresses through joint image–video stages with increasing resolution and frame count.Image stages initialize spatial generation, while joint training stabilizes per-frame quality and develops temporal modeling.
- Recipe refinement: Shared Cross-Attention is introduced at Stage 9 after 720p semantic degradation, while REPA is disabled from Stage 4 onward.The curriculum therefore changes objectives and conditioning mechanisms as training progresses.
- Recipe refinement: The 720p pretraining stage is initialized from the 480p SFT checkpoint rather than the 480p pretraining checkpoint, a pragmatic choice that was not ablated.The authors report no instability or regression during the 720p stage but do not present this as a validated finding.
- Token routing: TREAD routing reduces 720p theoretical FLOPs from approximately 4,913 to 3,563 per forward pass, a 27.5% reduction and estimated 1.38× speedup.Measured throughput increases by 1.31× in videos per second at 720p.
- Evaluation scope: The complete recipe is evaluated end to end rather than through isolated component ablations.The authors state that individual contributions are not separately identified.
5 Data
Motif-Video 2B uses curated, progressively filtered data and metadata-aware training infrastructure to improve data quality and utilization under a constrained compute budget.
- Corpus construction: The corpus combines internal and public image-video sources processed through shared sanitation, filtering, deduplication, and stage-wise quality controls.
- Quality filtering: Quality filtering removes specific failure modes using complementary aesthetic, luminance, suitability, technical, and motion signals.The pipeline avoids collapsing these signals into a single learned ranking.
- Captioning: Caption long, short, and truncated variants are sampled with probabilities 0.5, 0.3, and 0.2 to reduce train–test mismatch.
- Data loading: The offline bucket-balanced sampler moves filtering, bucketing, and rank assignment into planning while preserving sequential shard reads.
- Data loading: Throughput rises from N to approximately 4.6N, while utilization increases from roughly 20% to roughly 76%.Adding SA further raises throughput to approximately 5.4N.
6 Experiments
Experiments validate the architecture through attention analyses, benchmark comparisons, qualitative samples, and human evaluation, while also exposing weaknesses in anatomy and long-horizon stability.
- Quantitative evaluation: 83.76% VBench Total Score surpasses Wan2.1-T2V-14B at 83.69% and HunyuanVideo at 83.24% under the standard open-source setting.Wan2.2-T2V scores 84.23% but uses prompt optimization and is treated separately.
- Quantitative evaluation: Motif-Video 2B leads open-source models on Spatial Relationship at 83.02% and reaches 80.44% on Semantic Score.
- Shared Cross-Attention: Shared Cross-Attention contributes 7.6% of self-attention residual magnitude on average, with a maximum of 21.7%, and no block below 5.2%.
- Shared Cross-Attention: The cross-attention and self-attention residuals are nearly orthogonal, with global cosine similarity approximately −0.008.The analysis characterizes Shared Cross-Attention as injecting text information along distinct directions.
- Shared Cross-Attention: Cross-attention activity peaks near σ ≈1.0 and stabilizes after step 22 at σ ≈0.96 during denoising.
- Human evaluation: Human raters preferred Wan2.1-14B over Motif-Video 2B on prompt-following and video-fidelity, despite the latter’s higher VBench Total Score.
- Human evaluation: The 40-prompt human study cannot support fine-grained claims about small ELO differences or uniform superiority over baselines.
7 Discussion
The discussion finds that Motif-Video 2B’s strong VBench performance does not fully translate to perceptual quality, especially for temporal stability and fine-scale artifacts. The authors identify data coverage, unresolved component attribution, and scaling questions as important boundaries for interpreting the design.
- Interpretation of Results: 83.76% is the highest Total Score among evaluated open-source models, yet internal comparisons show a perceptual gap favoring Wan2.1-T2V-14B.The larger model leads by only 0.07 points on the aggregate metric, despite the consistent perceptual preference.
- Interpretation of Results: VBench can reward semantically correct outputs that remain perceptually unconvincing, including distorted anatomy and other sub-object artifacts.These failures preserve category or action correctness while human viewers immediately notice the artifact.
- Interpretation of Results: 83.31% is the Wan2.1-T2V-1.3B Total Score, with Motif-Video 2B ahead by 0.45 points overall and 4.79 points on Semantic Score.The models trade wins on quality dimensions, and the comparison is not fully controlled because Wan2.1 used roughly two orders of magnitude more data.
- Interpretation of Results: Motif-Video 2B’s results support matching larger models on compositional and semantic axes, but not matching Wan2.1-14B in perceived quality.The report attributes the remaining gap to capacity limits on temporal-stability axes while retaining quality parity within its parameter class.
- Interpretation of Results: Data coverage is identified as the current ceiling: long-tail domain gaps and dynamic-motion degradation are assessed as symptoms of scarce diverse, coherent video data.The authors propose scaling corpus quantity, motion diversity, and domain breadth as the natural next step.
- Limitations: Temporal failures include physically implausible dynamics, coherence loss in complex multi-agent scenes, and unintended mid-clip scene transitions.These failure modes are not surfaced by static-frame metrics and are documented as boundaries for future iterations.
- Limitations: The recipe is evaluated jointly rather than through per-component ablations, so individual contributions from Shared Cross-Attention, DDT, REPA phasing, and TREAD routing remain unresolved.The report instead provides attention analyses, a comparison involving Shared Cross-Attention, and an end-to-end VBench result.
- Limitations: Two training-recipe questions remain open: whether holistic early-stopped REPA helps, and how denser teacher features would change when REPA should be disabled.Both are presented as natural extensions rather than corrections.
8 Conclusion
The conclusion argues that competitive text-to-video generation can be achieved without massive scale when model design explicitly separates otherwise entangled objectives. Motif-Video 2B combines specialized architecture with an efficient training recipe and reaches strong semantic and compositional benchmark performance under a micro-budget.
- 8 Conclusion: Motif-Video 2B reaches 83.76% on VBench with 2B parameters, fewer than 10M clips, and under 100,000 H200 GPU hours.The model matches or exceeds models 7× its size on compositional and semantic dimensions.
- 8 Conclusion: Shared Cross-Attention, a three-stage backbone, and a DDT decoder separate text conditioning, modality fusion, joint representation, and detail reconstruction.The conclusion links these choices to stabilized long-sequence conditioning and inter-frame attention structure in later blocks.
- 8 Conclusion: TREAD token routing and phase-constrained REPA with a V-JEPA teacher form the report’s micro-budget training recipe.The recipe combines a 27% per-step FLOP reduction with structured early-phase learning.
A Additional results
This section presents additional qualitative results for text-to-video and image-to-video generation.
- A Additional results: Additional qualitative results cover both text-to-video and image-to-video generation in Figures 16 and 17.Figure 17 shows the input image followed by representative generated video frames.
B Sampling Configuration
The VBench samples use a fixed generation setup covering resolution, frame count, frame rate, sampler settings, timestep scheduling, and a negative prompt. The appendix also specifies the qualitative artifact description used in sampling configuration.
- B Sampling Configuration: VBench samples are generated at 1280 × 736 resolution, 121 frames, and 24 fps.These settings define the spatial and temporal sampling conditions for the reported scores.
- B Sampling Configuration: The sampler uses Video APG with momentum = 0.0, η = 0.0, r = 27, and classifier-free guidance scale 8.The configuration follows Waver for the sampler and uses fixed parameter values.
- B Sampling Configuration: The timestep schedule is linear-quadratic, with the linear-to-quadratic transition at t = 250.The schedule follows Meta Movie Gen.
- B Sampling Configuration: Figure 17 presents the input image in the leftmost panel and representative generated video frames in the remaining panels.This figure documents additional image-to-video results.
- B Sampling Configuration: The negative prompt is fixed at every sampling call and follows Wan’s configuration.The supplied passage introduces the full negative-prompt string, which describes overlays, rigid motion, flat framing, and dull lighting.
C Training Configuration
The training configuration combines adaptive optimization, resolution-dependent timestep sampling and shifting, and classifier-free guidance dropout. These choices adjust training behavior across stages and support conditional generation.
- AdamW uses β1 = 0.9, β2 = 0.99, ϵ = 10−8, zero weight decay, and global gradient clipping at 1.0.
- The learning rate is adjusted across stages using qualitative inspection, VBench scores, and current training resolution, with warmup after configuration changes.
- Below 360p, training samples timesteps with a logit-normal distribution; from 360p onward, it switches to cosine mode sampling.The early distribution emphasizes high-noise regions, while the later distribution shifts density toward intermediate timesteps.
- Resolution-dependent timestep shifting increases the shift factor with resolution, reaching σ = 7.0 at the highest training resolution.Larger shifts bias sampling toward higher-noise timesteps as token counts increase.
- Classifier-free guidance training independently drops text and image conditions with probability 10% each.Dropped samples re-encode the empty string through the T5Gemma2 text encoder rather than using a zero tensor.
D Implementation Details for FSDP2 Wrapping Order
The implementation applies activation checkpointing, compilation, and FSDP2 wrapping in that order so checkpointed and compiled transformer blocks can be independently sharded. Because Accelerate’s built-in path does not reliably target these blocks, the authors patch two entry points.
- Activation checkpointing precedes torch.compile, and both precede FSDP2 wrapping.This ordering aligns compile regions with checkpointed units and enables independent parameter sharding and communication.
- Accelerate’s built-in FSDP2 path does not reliably apply activation checkpointing to individual blocks inside the model’s transformer ModuleLists.The authors therefore replace fsdp2 apply ac with a version that directly iterates the relevant ModuleLists.
E Cross-Attention Ablation Details
Figure 18 evaluates Shared Cross-Attention by comparing matched generations with the mechanism enabled versus disabled in all 16 single-stream encoder blocks. The ablation links its removal to distinct failures in verb semantics, noun preservation, and compositional object structure.
- Without cross-attention, the model reverses the action in “A person is pushing cart,” preserving scene layout but losing verb semantics.
- Without cross-attention, “A green bird” produces green foliage without a bird, retaining the adjective while dropping the noun.
- Figure 18 compares enabled and disabled Shared Cross-Attention generations for the same prompts and seed at 360p, 50 steps, and 121 frames.The disabled condition removes cross-attention from all 16 single-stream encoder blocks while leaving dual-stream and DDT decoder blocks unchanged.
- Without cross-attention, “A clock and backpack” merges the two objects into one hybrid form instead of preserving them as distinct entities.