PaperlayerView exact revision on arXiv

Source-linked AI summary

MUG-V 10B: High-efficiency Training Pipeline for Large Video Generation Models

Yongshun Zhang, Zhongyi Fan, Yonghang Zhang, Zhangzikang Li, Weifeng Chen, Zhongwei Feng, Chaoyue Wang, Peng Hou, Anxiang Zeng

arXiv:2510.17519v2cs.CVcs.AI

TL;DR

Large-scale video generation training must address text-video alignment, long sequences, and complex spatiotemporal dependencies. MUG-V presents an end-to-end training framework for these challenges, and reports competitive or superior evaluation performance, particularly for e-commerce scenarios.

  • Problem

    Training large-scale video generators is challenging because it must address text-video alignment, long visual token sequences, and complex spatiotemporal patterns.

    Source: §1 · p. 1

  • Method

    MUG-V integrates data processing, VideoVAE compression, a 10B Diffusion Transformer, multi-stage training, and infrastructure optimizations in an end-to-end framework.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    MUG-V 10B delivers competitive or superior performance across qualitative and quantitative evaluations, particularly in e-commerce scenarios.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The released framework includes model weights, Megatron-Core-based training code, and inference pipelines for video generation and enhancement.

    Source: §1 · p. 2

    Preference-based video optimization remains challenging because current reward models have limited capacity and evaluation spans multiple axes, so the authors use human-annotated preferences.

    Source: §4.2.3 · p. 7

Abstract

from arXiv · show

In recent years, large-scale generative models for visual content (\textit{e.g.,} images, videos, and 3D objects/scenes) have made remarkable progress. However, training large-scale video generation models remains particularly challenging and resource-intensive due to cross-modal text-video alignment, the long sequences involved, and the complex spatiotemporal dependencies. To address these challenges, we present a training framework that optimizes four pillars: (i) data processing, (ii) model architecture, (iii) training strategy, and (iv) infrastructure for large-scale video generation models. These optimizations delivered significant efficiency gains and performance improvements across all stages of data preprocessing, video compression, parameter scaling, curriculum-based pretraining, and alignment-focused post-training. Our resulting model, MUG-V 10B, matches recent state-of-the-art video generators overall and, on e-commerce-oriented video generation tasks, surpasses leading open-source baselines in human evaluations. More importantly, we open-source the complete stack, including model weights, Megatron-Core-based large-scale training code, and inference pipelines for video generation and enhancement. To our knowledge, this is the first public release of large-scale video generation training code that exploits Megatron-Core to achieve high training efficiency and near-linear multi-node scaling, details are available in https://github.com/Shopee-MUG/MUG-V.

1 Introduction

Large-scale video generation must handle text-video alignment, long visual token sequences, and complex spatiotemporal patterns. MUG-V addresses these challenges with an end-to-end training framework and releases its model, training code, and inference pipelines.

1 Introduction

  • Large-scale video generation must address text-video alignment, long visual token sequences, and complex spatiotemporal patterns.Compared with image generation, video must preserve static-content fidelity while learning diverse motion dynamics.Source: §1 · p. 1
  • MUG-V adopts latent diffusion transformers with flow matching and designs an end-to-end framework spanning data processing, compression, training, infrastructure, and evaluation.The framework investigates efficiency and resource cost at each stage alongside techniques to improve generative quality.Source: §1 · p. 2
  • MUG-V combines scalable data processing, high-ratio VideoVAE compression, a stable 10B-parameter DiT, multi-stage training, and efficient infrastructure.The framework also includes inference pipelines for video generation and enhancement.Source: §1 · p. 2
  • The authors describe the full stack as the first public release of large-scale video-generation training code using Megatron-Core for high efficiency and near-linear multi-node scaling.The released components include model weights, large-scale training code, and inference pipelines.Source: §1 · p. 2

2 Data Processing

The data pipeline turns screened raw footage into diverse clip-caption pairs for pretraining and carefully curated, human-labeled material for post-training. Its annotations assess video quality, product consistency, preference, and correctness.

2 Data Processing

  • The pipeline screens and processes video into diverse clip-caption pairs, reserving a smaller high-quality subset for post-training.The authors describe video-text pairs as a scalable training source that jointly encode visual dynamics and semantic descriptions.Source: §2 · p. 2

2.1 Scalable Video Processing Pipeline

  • Raw videos pass licensing, privacy, prohibited-content, and scene, subject, and motion-diversity screening before fine-grained processing.Scene splitting combines PySceneDetect for most cuts with CSS for transitions such as gradual fades.Source: §2.1 · p. 2
  • Four-stage visual filtering checks sharpness, aesthetic score, motion amplitude, and post-processing or camera-shake artifacts.The motion filter drops clips with average flow magnitude below 1 or above 20; a fine-tuned multimodal model flags overlays, borders, effects, speed alteration, and camera shake.Source: §2.1 · pp. 2–3
  • A Qwen2-VL-72B captioner is fine-tuned for descriptions of objects, appearance, motion, and background, then distilled to Qwen2-VL-7B for large-scale captioning.Caption parsing also supports category rebalancing and near-duplicate removal.Source: §2.1 · p. 3

2.2.1 High-quality Clip Labeling

  • Post-training data combines top-scoring clips with human-reviewed clips checked for motion continuity, content stability, and visual fidelity.The authors also up-weight human-centric clips because articulated human motion remains a bottleneck and dominates real user queries.Source: §2.2.1 · p. 3

2.2.2 Preference Optimisation Data Labeling

  • Annotators evaluate generated videos using pairwise preference judgments and absolute checks of prompt match, subject preservation, and physical or rendering errors.Pairwise labels cover aesthetics, motion smoothness, and error severity; absolute labels are pass or fail.Source: §2.2.2 · pp. 3–4

3 Model Design and Architecture

MUG-V uses a compressed video-latent pipeline: a Video VAE encodes clips, and a 10B Diffusion Transformer generates them under text, image, or combined conditioning. The design emphasizes compact representations, flexible encoding, and global spatiotemporal modeling.

3 Model Design and Architecture

  • A Video VAE compresses pixel-space frames before a 10B-parameter DiT models spatiotemporal dynamics in the latent domain.The image-conditioning strategy injects reference-image visual tokens into the DiT context stream for text- or key-frame-conditioned generation.Source: §3 · p. 4

3.1 Video Variational Autoencoder (Video VAE)

  • The Video VAE downsamples time, height, and width by 8, then 2 × 2 spatial patchification combines four latents into one DiT token.The encoder alternates 2D spatial convolutions with 3D motion-modeling convolutions to reduce FLOPs relative to an all-3D counterpart.Source: §3.1 · p. 4, §3.1.1 · p. 4

3.1.1 Video VAE Architecture

  • Joint eightfold downsampling across dimensions encodes appearance and motion, with bottleneck width C = 24 selected as a quality-storage trade-off.Reconstruction improves as channel capacity increases, then reaches diminishing returns.Source: §3.1.1 · p. 4

3.1.2 Minimal Encoding Strategy

  • Minimal encoding derives each latent token solely from its corresponding eight-frame chunk, avoiding temporal context mixing that can imbalance information across tokens.The encoder supports arbitrary sequence lengths and image-to-video, continuation, and first-, middle-, or last-frame conditioning; the decoder instead trains on windows of 1, 4, or 8 contiguous latents.Source: §3.1.2 · pp. 4–5
  • The minimal-encoding design removes causal-convolution information-density imbalance while retaining compatibility with downstream tasks and diverse clip lengths.The paper reports that this contributes significantly to overall training efficiency.Source: §3.1.2 · p. 5

3.2 MUG-V 10B Diffusion Transformer Model

  • The DiT jointly supports text-to-video, image-to-video, and text-plus-image-to-video synthesis, using input patchifying, text-condition networks, stacked blocks, and output unpatchifying.The backbone follows DiT architecture and focuses on design choices beyond the overall organization shared with prior models.Source: §3.2 · p. 5
  • MUG-V uses full attention because its high-compression Video VAE and patchification make its cost manageable while enabling direct global token interactions.Three-dimensional RoPE encodes spatial and temporal coordinates for visual tokens.Source: §3.2 · p. 5
  • For image- or frame-conditioned generation, conditioned regions use the supplied latent with zero diffusion noise while remaining tokens follow the standard noisy trajectory.During pretraining, the strategy clarifies the timestep signal and yields superior fidelity to the provided visual content at inference.Source: §3.2 · p. 5

4 Model Training

Training combines composite and adaptive Video VAE losses with parameter expansion, a staged pretraining curriculum, and preference-focused post-training. The stages aim to reduce experimentation cost and improve video-generation quality while maintaining stable, resource-efficient training.

4.1 Video VAE Training

  • The Video VAE combines reconstruction, KL-divergence, and adversarial losses, with adversarial fine-tuning reserved for the final stage and its weight kept small.Reconstruction combines MSE, L1, and perceptual terms; the authors monitor validation PSNR/SSIM because excessive adversarial weight can shift hues or over-enhance details.Source: §4.1 · pp. 5–6
  • Adaptive reconstruction weighting replaces the plain L1 component with a saliency-weighted loss so fast motion and high-frequency edges contribute larger gradients without extra data passes.The map uses the spatial Laplacian and temporal forward difference; it targets fine-detail regions that oscillate after global structure has stabilized.Source: §4.1 · p. 6

4.2 MUG-V 10B Diffusion Transformer Training

  • The 10B DiT training strategy combines parameter expansion and hyperparameter search, curriculum pretraining, and supervised and preference-based post-training.The authors present these measures as supporting stable, resource-efficient training without compromising video-generation quality.Source: §4.2 · p. 6

4.2.1 Parameter Expansion

  • A compact 2B model is trained first, then expanded in hidden width to the 10B scale for continued training using transferred hyperparameters.The expansion tiles parameters while preserving functional behavior and adds random perturbations; the small-model stage reduces experimentation cost.Source: §4.2.1 · p. 6

4.2.2 Multi-stage Pre-training Curriculum

  • Pretraining moves from mixed images and 360p clips, to longer 360p clips, then to curated 720p five-second clips.Stages 1 and 2 expose the model to over ten times more samples than Stage 3; Stage 3 uses around 12M high-quality videos to refine detail.Source: §4.2.2 · p. 7

4.2.3 Post-training and Alignment

  • Post-training uses annealed SFT on around 0.3M selected clips and preference optimization to address fine-grained artifacts and physical-plausibility failures.The authors approximate post-hoc EMA with exponentially decayed model ensembling; preference labels target error-free generation and motion quality, while retaining SFT as a regularizer mitigates undesirable statistical biases.Source: §4.2.3 · p. 7

5 Infrastructure

The Megatron-Core infrastructure targets long video sequences, billion-parameter scale, and numerical precision through parallelism, balanced data pipelines, and kernel fusion.

5 Infrastructure

  • The Megatron-Core framework addresses long sequences, model scale, and numerical precision with model parallelism, balanced data loading, and fused kernels.These are the three infrastructure optimization areas named by the authors.Source: §5 · pp. 7–8

5.1 Model Parallel Strategy

  • The 10B model combines TP, SP, PP, and DP: SP shards activations, PP partitions layers across nodes, and DP enlarges the effective batch.Benchmarking identified a configuration with near-linear efficiency scaling.Source: §5.1 · p. 8

5.2 Data Loading and Computation Balance

  • Asynchronous prefetching and caching overlap data transfer with computation, while dynamic balanced sampling reduces GPU idle cycles from variable-length videos.Batches are balanced across ranks by comparable computational cost.Source: §5.2 · p. 8

5.3 Kernel Fusion

  • A fused GPU kernel combines bias addition, modulation, and residual accumulation; block-level fusions pair LayerNorm with QKV projection and fold masking into softmax.Static shape inference also removes unnecessary padding.Source: §5.3 · p. 8
  • Combined kernel optimizations lower memory traffic, increase arithmetic intensity, and deliver an end-to-end speed-up.The section describes fusion at both the low-level kernel and DiT-block levels.Source: §5.3 · p. 8

6 Applications & Model Performance

The evaluation combines VBench-I2V metrics with human judgments tailored to e-commerce video generation. MUG-V 10B ranks third on the VBench-I2V leaderboard and achieves strong pass-rate and high-quality-rate results in human evaluation, though minor artifacts and geometric distortions remain.

6 Applications & Model Performance

  • Because automatic metrics can miss fine-grained product defects, the evaluation supplements standard scores with human judgments of usability and quality.The paper gives altered fabric textures and incorrect hand poses as examples of defects relevant to product fidelity.Source: §6 · pp. 8–9

6.1 Quantitative Evaluation of Video Generation

  • At submission time, MUG-V 10B ranked third on the VBench-I2V leaderboard, behind Magi-1 and PI.The evaluation used a weighted score over temporal consistency, motion dynamics, and perceptual aesthetics/distortion dimensions.Source: §6.1 · p. 9, Table 1 · p. 9

6.2 Human Evaluation on E-commerce Video Generation Tasks

  • Human evaluation used randomly ordered five-second clips, with three annotators assigning final labels by consensus of at least two votes.Inputs were sampled from public model showroom images, and each method used its default prompt generator.Source: §6.2 · p. 9
  • Human evaluation found strong pass-rate and high-quality-rate results, while residual minor artifacts and geometric distortions still limit overall quality.A clip was deployable only if it was sufficiently realistic and preserved product consistency with the input image.Source: §6.2 · p. 9

7 Related Works

The related-work discussion traces diffusion and flow-matching foundations into video-generation approaches, then positions MUG-V 10B within latent-video DiT systems and e-commerce alignment.

7.1 Diffusion Models

  • Diffusion models evolved from denoising processes to flow matching, which trains transport from a simple prior to data and can offer faster sampling and simpler training dynamics [12] [13].The discussion also notes diffusion and flow matching applications across images, 3D, audio, motion, and discrete domains.Source: §7.1 · pp. 9–10

7.2 Video Generation Models

  • Early video systems had limits in duration, resolution, and temporal coherence; latent video diffusion used Video VAEs to enable longer, higher-fidelity clips [79] [80] [81] [82] [83] [84].Source: §7.2 · p. 10
  • Current DiT-based video generators operate on latent video representations, with advances in conditioning, longer-context handling, and faster sampling [85] [86].The discussion cites Sora, Sora2, and Veo3 as proprietary examples of long-duration, high-resolution generation.Source: §7.2 · p. 10
  • Autoregressive video models offer long-horizon structure and discrete controllability but often trade off visual fidelity and incur compression artifacts [94].Hybrid systems combine autoregressive planning with diffusion or flow decoders [96] [98] [99] [100].Source: §7.2 · p. 10
  • MUG-V 10B follows the latent-video DiT paradigm, emphasizing efficient training, modern curriculum, text- and image-to-video conditioning, and e-commerce alignment.Source: §7.2 · p. 10

8 Conclusion

The conclusion presents MUG-V 10B as an end-to-end DiT training framework whose evaluations show competitive or superior performance, particularly for e-commerce scenarios.

8 Conclusion

  • Across qualitative and quantitative evaluations, MUG-V 10B delivers competitive or superior performance, particularly in e-commerce scenarios.The framework integrates scalable data processing, high-compression Video VAE, a DiT generator, multi-stage pre- and post-training, and systems-level optimization.Source: §8 · p. 10

A Additional Technical Details

The appendix supplies further technical details on Video VAE minimal encoding and preference optimisation. It describes an implementation based on reshaping and an iterative post-training approach that includes RDPO.

A.1 More Details of Video VAE

  • Video VAE minimal encoding is implemented with tensor reshapes and adds no computational overhead to the overall process.The algorithm groups input frames into chunks of eight, encodes them, and reshapes latent tokens for decoding.Source: §A.1 · p. 11

A.2 More Details of Preference Optimisation

  • RDPO uses reverse-sampled real videos to construct preference pairs automatically, without manual annotation.The authors report that flow paths from reverse samples are statistically superior to those from random noise and associated trajectories.Source: §A.2 · p. 11
  • Post-training uses a multi-stage iterative schedule, while balancing supervised fine-tuning and reinforcement learning remains an open problem.The authors observe that well-tuned reinforcement learning can rapidly strengthen generation along specific dimensions.Source: §A.2 · p. 11

B Additional Experiments

These additional experiments assess Video VAE reconstruction and show human-evaluation comparisons for e-commerce video generation. The reported reconstruction results balance fidelity against the VAE’s compression ratio.

B.1 Video VAE Reconstruction

  • Figure 1 pairs input video frames and local details with enlarged patches from reconstructed clips.The evaluation uses real-world clips curated for validation.Source: §B.1 · p. 11, Figure 1 · p. 12

B. Additional Experiments

  • Figure 2 presents human-evaluation comparisons of generated e-commerce video quality.Source: Figure 2 · p. 12

B.1 Video VAE Reconstruction

  • At 8 × 8 × 8 compression, the Video VAE surpasses most comparators on PSNR, SSIM, LPIPS, and FloLPIPS, with slightly lower SSIM than CogVideoX VAE at 720p.Qualitative examples show fine details, including drifting smoke and rapidly changing textures, reproduced faithfully.Source: Table 2 · p. 11, §B.1 · p. 13

B.2 Evaluation Results of E-commerce Video Generation

In mixed blind evaluations, MUG-V 10B scores higher on e-commerce video pass rate and high-quality rate, though both metrics remain relatively low in absolute terms. The paper also presents qualitative generation examples and identifies unresolved challenges in controllability, fidelity, duration, and resolution.

B.2 Evaluation Results of E-commerce Video Generation

  • MUG-V 10B achieves superior pass and high-quality rates in mixed blind e-commerce video evaluations, although both rates remain relatively low in absolute terms.The pass rate reflects videos without noticeable artifacts or errors; the high-quality rate reflects motion coherence, visual fidelity, and aesthetics.Source: §B.2 · p. 13

B.3 Visualizing Generated Videos

  • The qualitative examples cover text-to-video, image-to-video, and e-commerce generation across multiple models.For e-commerce comparisons, each model uses its default prompts or prompt-rewriting tools.Source: §B.3 · p. 13, Figure 3 · p. 14, Figure 4 · p. 15

C Challenges and Future Work

  • Open challenges include faithful and controllable conditioning, fine-grained appearance fidelity, and scaling to longer or higher-resolution videos.The authors link subtle fidelity degradation to VAE compression and DiT noise initialization, and identify long-sequence training, inference efficiency, and long-range temporal consistency as scaling needs.Source: C Challenges and Future Work · p. 13

Related papers

Loading 2510.17519v2…