PaperlayerView exact revision on arXiv

Source-linked AI summary

Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE

Yangming Shi, Shixiang Zhu, Tao Shen, Zhimiao Yu, Dengsheng Chen, Taicai Chen, Yunfei Yang, Juan Zhou, Chen Cheng, Liang Ma, Xibin Wu, Benxuan Yan, Ge Li, Tuoyu Zhang, Dan Li, Chang Liu, Zhenbang Sun

arXiv:2605.02641v1cs.CV

TL;DR

Unified frameworks for video generation and editing remain at an early stage amid data complexity and computational bottlenecks. Mamoda2.5 combines an AR–Diffusion architecture with a DiT-MoE backbone and few-step distillation and reinforcement learning, achieving top-tier video-generation and editing benchmark results.

  • Problem

    Unified frameworks for video generation and editing remain at an early stage, constrained by data complexity and computational bottlenecks.

    Source: §1 · p. 4

  • Method

    Mamoda2.5 unifies multimodal input conditioning and visual generation and editing in an AR–Diffusion pipeline with a DiT-MoE backbone and joint few-step distillation and reinforcement learning.

    Source: §1 · p. 4, §2 · p. 5, §8 · p. 27

  • Results

    Mamoda2.5 achieves top-tier generation quality on VBench 2.0 and outperforms most evaluated open-source video-editing models, surpassing proprietary models including Kling O1 on OpenVE-Bench.

    Source: §8 · p. 27

  • Takeaways & Limitations

    Deployment for content moderation and creative restoration in advertising scenarios demonstrates practical value in improving ad delivery efficiency.

    Source: §8 · p. 28

    Mamoda2.5 currently supports image and video generation and editing, with audio processing identified as a future addition.

    Source: §9 · p. 28

Abstract

from arXiv · show

We present Mamoda2.5, a unified AR-Diffusion framework that seamlessly integrates multimodal understanding and generation within a single architecture. To efficiently enhance the model's generation capability, we equip the Diffusion Transformer backbone with a fine-grained Mixture-of-Experts (MoE) design (128 experts, Top-8 routing), yielding a 25B-parameter model that activates only 3B parameters, significantly reducing training costs while scaling up the model capacity. Mamoda2.5 achieves top-tier generation performance on VBench 2.0 and sets a new record in video editing quality, surpassing evaluated open-source models and matching the performance of current top-tier proprietary models, including the Kling O1 on OpenVE-Bench. Furthermore, we introduce a joint few-step distillation and reinforcement learning framework that compresses the 30-step editing model into a 4-step model and greatly accelerates model inference. Compared to open-source baselines, Mamoda2.5 achieves up to $95.9\times$ faster video editing inference. In real-world applications, Mamoda2.5 has been successfully deployed for content moderation and creative restoration tasks in advertising scenarios, achieving a 98% success rate in internal advertising video editing scenario.

1 Introduction

Mamoda2.5 is a unified AR–Diffusion model that uses sparse fine-grained MoE to support visual generation and editing, achieving strong video-generation and editing results. Its few-step editing model reduces inference steps while retaining the framework’s efficiency focus.

1 Introduction

  • Unified video generation and editing remain challenging because spatiotemporal modeling is compute-intensive, while editing also requires encoding the source video and high-quality paired training data.The introduction identifies fine-grained MoE with learned token-level routing as an understudied direction for video generation; existing dense editing models have limited performance in complex scenarios.Source: §1 · p. 4
  • The fine-grained MoE scales Mamoda2.5 to 25B total parameters with only 3B activated per forward pass, using 128 experts and Top-8 routing.Together with a 4×16×16 VAE rather than the more typical 4×8×8, sparse activation and compressed tokens improve generation performance and training efficiency.Source: §1 · pp. 4–5
  • Mamoda2.5 achieves top-tier open-source video-generation quality on par with HunyuanVideo 1.5 [21] and LongCat-Video [22], and state-of-the-art video-editing performance, outperforming most evaluated open-source models and Kling O1.The introduction also describes its performance across generation and editing scenarios as rivaling or surpassing dedicated task-specific models [17] [19] [20] [21].Source: §1 · pp. 4–5
  • The 30-step Mamoda2.5-Edit model achieves over 12× faster editing inference than comparable baselines, while joint distillation and reinforcement learning produce a 4-step student without CFG.The student reduces the step count by 7.5× and eliminates CFG overhead for an additional ∼2× savings.Source: §1 · p. 5
  • One model supports text-to-image, text-to-video, image editing, video editing, and multimodal understanding, replacing separate task-specific models.The introduction attributes its performance across scenarios to a unified conditional visual-generation strategy in a single architecture and a scalable synthetic-data pipeline.Source: §1 · pp. 4–5

2 Model Architecture

Mamoda2.5 unifies multimodal understanding, planning, generation, and editing in an AR–Diffusion pipeline, with a fine-grained DiT-MoE backbone and compressed video latents. Learnable MetaQueries bridge the AR understanding module to diffusion, while in-context conditioning supports multiple visual tasks in one architecture.

2 Model Architecture

  • A three-stage pipeline combines AR conditioning, iterative DiT-MoE latent denoising, and a VAE; its 4×16×16 causal VAE yields 4× fewer spatial tokens than a 4×8×8 VAE.The VAE encodes images and videos into latents for training and decodes generated latents to pixels at inference.Source: §2 · p. 5, Figure 3 · p. 5

2.1 Autoregressive Understanding Module

  • The AR backbone models multimodal inputs and uses learnable MetaQueries to activate generation experts, replacing autoregressive visual-token prediction with conditional features for DiT.Mamoda2.5 retains the AR architecture and pretrained weights of Mamoda 2.0 [24].Source: Figure 3 · p. 5, §2.1 · p. 6

2.2 Visual Generation with DiT-MoE

  • Mamoda2.5 replaces DiT feed-forward sublayers with fine-grained MoE layers, using 128 routed experts with Top-8 activation and one shared expert per token.Sigmoid Top-K routing evaluates expert relevance independently, while loss-free Expert Bias adjusts selection to suppress overused experts without adding interference gradients to the training objective.Source: §2.2 · pp. 6–7

2.3 Upcycling: Dense-to-MoE Initialization

  • To upcycle a dense Wan2.2 5B model, Mamoda2.5 transfers attention and normalization weights, samples distinct neuron subsets for expert FFNs, and randomly initializes routers with zero Expert Bias.Because each expert width is 1,024 versus the dense FFN width of 14,336, this procedure breaks expert symmetry; 128×1,024/14,336 ≈9.1× oversampling brings collective dense-FFN coverage close to 100%.Source: §2.3 · p. 7

2.4 Multi-Task Conditional Generation

  • Text-to-image, text-to-video, image editing, and video editing are formulated as conditional generation by concatenating refined conditions and latents for global self-attention.An auxiliary byte-level ByT5 encoder adds character-aware features to improve spelling accuracy and text-layout fidelity.Source: §2.4 · p. 8

3 Data Curation

Mamoda2.5’s training data span five multimodal understanding and generation/editing categories. Its video-editing data pipeline synthesizes paired clips with shared early denoising, filters and recaptions them, and creates reverse-direction examples.

3.1 Data Composition

  • Training data cover multimodal understanding, text-to-image and text-to-video generation, image editing, and video editing.Understanding data derive from Honey-1M [34] and are refined for chain-of-thought supervision; text-to-image data come from Mamoda2 [24], while text-to-video data combine in-house real-world and synthetic clips. Image-editing data are cleaned from Mamoda2 [24] and cover diverse editing categories with Chinese and English instructions; video-editing data include add, remove, replace, style transfer, and subtitle editing, with style-transfer data from open-source datasets [35] [36] [37].Source: §3.1 · p. 8

3.2 Video Editing Data Synthesis Pipeline

  • Existing staged video-editing synthesis pipelines can propagate errors between stages and limit final data quality.These pipelines preprocess videos into conditioning signals, generate edits, and use vision-language models to filter samples.Source: §3.2 · p. 9
  • Prompt-modified text-to-video generation can produce aligned pre-edit and post-edit pairs but often fails to preserve their structural consistency.Differences in subject position, identity, and motion continuity limit the pairs’ suitability for high-quality editing data.Source: §3.2 · p. 9
  • Sharing early denoising steps substantially improves structural consistency between pre-edit and post-edit videos.Fixing the initial noise alone is usually insufficient because prompt-dependent deviations accumulate during multi-step denoising; sharing early steps suppresses noise drift while preserving later detail refinement.Source: §3.2 · p. 9
  • A three-stage pipeline synthesizes paired videos for add, remove, and replace editing, then filters them and doubles the examples through inversion.LLMs generate paired prompts; a video-generation model creates pre-edit and post-edit clips with shared denoising steps; VLMs recaption and filter the results before each pair is inverted into its opposite-direction counterpart.Source: §3.2 · p. 9, Figure 5 · p. 9

4 Model Training

Mamoda2.5 is trained through progressive generation and editing stages, followed by joint few-step distillation and reinforcement learning. System optimizations support efficient video training and inference.

4 Model Training

  • Mamoda2.5 jointly optimizes next-token prediction for understanding and flow matching for generation through a five-stage training pipeline.The stages progress from low- and high-resolution generation pretraining to joint generation-understanding SFT, visual editing training, and 4-step joint distillation and reinforcement learning.Source: §4 · p. 10, Figure 6 · p. 10

4.1 Pretraining and Supervised Fine-Tuning

  • Pretraining scales from 480 px images to short, low-resolution videos and then to 720 px, 24 fps, while video data is resampled to the target frame rate.The schedule allocates extra steps to short video training and reduces the image-data proportion over time to balance spatial quality with temporal consistency. SFT mixes understanding, generation, and editing data, improving generated-image and video aesthetics and realism while understanding remains competitive with the original backbone.Source: §4.1 · pp. 10–11

4.2 Editing-Specific Training

  • Editing training first pretrains on instruction-based image edits, then mixes image and video editing data while scaling resolution from 480 px to 720 px.Mixed training uses a 1:9 image-to-video sampling ratio and uniformly samples video length from 49 to 121 frames.Source: §4.2 · p. 11

4.3 Joint Few-Step Distillation and Reinforcement Learning

  • Joint distillation and RL train the few-step Student simultaneously: the DMD term anchors it to the Teacher distribution while reward optimization can push quality beyond the Teacher.A cold start applies only the DMD loss for the first C steps; DiffusionNFT optimizes the flow-matching objective through a contrastive mechanism without expensive likelihood estimation.Source: §4.3 · p. 12
  • The joint framework enables high-quality few-step inference without classifier-free guidance, can surpass the multi-step Teacher in editing quality, and cuts training cost by an order of magnitude versus sequential alternatives.Its reward system combines signals for edit quality, visual quality, background consistency, and text-rendering accuracy.Source: §4.3 · p. 12

4.4 System Optimization

  • System optimizations include online encoding that reuses cached features and inference methods combining mixed parallelism, sparse attention, and adaptive caching.Online encoding combines Encoder DP with interleaved encoding to reduce redundant encoding and improve computation-communication overlap; inference uses SSTA and separates attention and FFN caching with drift correction and sliding-window block offloading.Source: §4.4 · p. 13
  • At 480 px and 81 frames, the 4-step Mamoda2.5-Edit-Distill takes 9.2 seconds, 95.9× faster than VInO and 41.7× faster than OmniVideo2.The 30-step editor takes 69 seconds (12.8× faster than VInO, 5.6× faster than OmniVideo2); 720 px, 93-frame video generation takes 110 seconds (over 12× faster than Wan2.2 A14B, 5× faster than HunyuanVideo 1.5, and 18× faster than LongCat Video).Source: §4.4 · pp. 13–14, Table 2 · p. 14

5 Evaluation

Mamoda2.5 is evaluated on video and image generation and editing, alongside multimodal understanding; results highlight strong video-editing performance and retained visual-language comprehension.

5.1.1 Text-to-Video Generation

  • Mamoda2.5 achieves top-tier VBench 2.0 results among open-source text-to-video generators, though a performance gap remains versus state-of-the-art proprietary models.VBench 2.0 assesses creativity, commonsense, controllability, human, and physics dimensions; the evaluation compares proprietary and open-source systems.Source: §5.1.1 · p. 14, Table 3 · p. 15

5.1.2 Instruction-based Video Editing

  • 3.86 Overall on spatially aligned OpenVE-Bench is Mamoda2.5’s highest score among evaluated models, ahead of the top-tier proprietary model (3.73) and Kling O1 (3.69).The comparison covers seven task categories and excludes the non-aligned setting. Qualitative examples also show stronger object placement, detail preservation, local replacement, and temporal consistency; the authors attribute gains partly to their data pipeline and ByT5 encoder for text editing.Source: §5.1.2 · pp. 14–16, Table 4 · p. 15
  • Mamoda2.5 achieves the highest scores across all FiVE-Bench metrics among competing open-source models, including on motion-rich video edits.FiVE-Bench evaluates 100 videos across six task categories and 420 object-level prompt pairs using FiVE-Acc submetrics.Source: §5.1.2 · p. 19, Table 5 · p. 19
  • On Reco-Bench scored with Gemini-2.5-pro, Mamoda2.5 achieves the best overall performance across all four editing categories in comparison with VInO.The authors use Gemini-2.5-pro because they found Gemini-2.5-flash less discriminative, and focus the re-scored comparison on VInO, the leading open-source editing model.Source: §5.1.2 · p. 19, Table 6 · p. 19

5.2.1 Text-to-Image Generation

  • Mamoda2.5 performs best among evaluated video-generation models on GenEval and remains highly competitive with dedicated image-generation models.GenEval assesses compositional generation across six dimensions, including object counting, spatial relationships, and color attribution.Source: §5.2.1 · p. 20, Table 7 · p. 20

5.2.2 Instruction-based Image Editing

  • Mamoda2.5 ranks first on ImgEdit’s Replace category and shows competitive performance among open-source models, with strong overall image-editing quality.The evaluation uses ImgEdit and GEdit-EN, which assess editing operations and structural consistency, perceptual quality, and overall quality, respectively.Source: §5.2.2 · p. 20, Table 8 · p. 21

5.3 Multimodal Understanding

  • Mamoda2.5 closely matches Qwen3-VL-8B-Instruct on multimodal understanding, surpassing it on MMMU and MathVista while remaining competitive on perception-oriented tasks.The results indicate that adding generation and editing capabilities preserves core visual-language comprehension.Source: §5.3 · p. 20, Table 9 · p. 21

6 Experiments

Experiments test MoE scaling and upcycling, conditioning choices, editing data, and joint distillation/RL. The reported results include faster convergence, stronger instruction-following and editing performance, and a 4-step student that surpasses the 30-step teacher on overall editing quality.

6.1.1 Experimental Setup

  • T2I ablations guide T2V training; early T2V runs showed the same MoE convergence trend, but full-scale T2V ablations were not conducted.All models share activated parameter counts; Sigmoid gating with loss-free Expert Bias outperforms Softmax with auxiliary load-balancing loss on training-loss convergence, so subsequent experiments use Sigmoid plus Expert Bias.Source: §6.1.1 · pp. 21–22

6.1.2 MoE vs. Dense: Efficient Scaling with Fine-Grained Experts

  • E64A8 achieves approximately 2.2× convergence speedup over Dense, with convergence ordering E64A8 > E32A4 > E16A2 > Dense under matched activated parameters.The authors attribute the trend to finer experts enabling greater specialization and more diverse expert assemblies; Mamoda2.5 extrapolates to E128A8.Source: Figure 13 · p. 22, §6.1.2 · p. 22

6.1.3 Upcycling Ablations

  • Expert Attn reaches loss 0.08 in ∼8K steps versus ∼40K for From scratch, an approximately 5× speedup, while improving GenEval by +12.47 and DPGBench by +4.10 over that baseline.Expert Attn also improves GenEval by +1.45 and DPGBench by +0.12 beyond Attn init; Expert Attn + Drop has nearly Attn init’s training loss and GenEval −0.60 versus Expert Attn, despite DPGBench +0.03. The authors attribute Drop’s counterproductive effect to destroying too much pretrained knowledge in the already-sparse sampled weights.Source: Figure 14 · p. 23, §6.1.3 · p. 23
  • Random neuron sampling was consistently effective in practice compared with structured contiguous partitioning and activation-magnitude-based neuron selection, and transferred pretrained representations despite fundamental architectural differences across substantially different architectures.DiT blocks are eligible for transfer; the condition encoder, conditioning injection mechanism, and router are initialized independently.Source: §6.1.3 · p. 23

6.2 Conditioning Architecture Ablations

  • Qwen3-VL-8B paired with in-context conditioning enables a task-agnostic architecture to handle diverse multimodal conditions without structural modification.The four-way comparison trains from scratch under identical hyperparameters, data, and compute to isolate encoder and injection choices from pretrained initialization.Source: §6.2 · p. 24
  • GenEval 68.50 and DPGBench 78.84 make Qwen3-VL-8B with in-context conditioning the best tested combination, exceeding the umT5 plus cross-attention baseline by +1.20 and +1.76, respectively.In-context injection improves both encoders (umT5: +0.63/+0.68; Qwen3-VL-8B: +2.98/+2.40); with cross-attention, Qwen3-VL-8B scores 65.52 versus umT5’s 67.30.Source: Table 10 · p. 24, §6.2 · p. 24

6.3 Visual Editing Ablations

  • Mamoda2.5-Edit-Data consistently beats Reco-Data across add, remove, and replace scenarios in editing accuracy, video quality, naturalness, and overall score.In a separate ImgEdit comparison, Mamoda2.5 25B-A3B outperforms Wan2.2 5B Dense after both are fine-tuned on the same image-editing dataset, despite using fewer activated parameters.Source: §6.3 · pp. 24–25, Table 12 · p. 25, Table 11 · p. 25

6.4 Joint Distillation and RL

  • ∼15× inference acceleration lets the 4-step CFG-free Student maintain competitive editing quality and surpass the 30-step CFG Teacher’s hybrid reward after around 100 training steps.The acceleration combines 7.5× from reducing steps and ∼2× from eliminating CFG. In a qualitative case, the FSD model correctly added a red car beside a moving train when the Teacher failed.Source: Figure 15 · p. 25, §6.4 · pp. 25–26, Figure 16 · p. 26

7 Applications

Mamoda2.5 is deployed in advertising for content moderation and creative restoration, combining actionable compliance feedback with automated asset repair to shorten the creative production cycle.

AI-Powered Content Moderation

  • For content moderation, Mamoda2.5 uses multimodal understanding to identify creative issues and provide fine-grained rejection reasoning with specific violations and remediation suggestions.It checks visual quality, text rendering accuracy, and semantic appropriateness, identifying issues such as distorted text, inconsistent branding, and low-quality frames before delivery.Source: §7 · p. 27

Automated Creative Restoration

  • For automated creative restoration, Mamoda2.5 achieves a 98% success rate on an internal advertising video editing benchmark, with MoE inference over 12× faster than comparable dense models.Instruction-based editing supports defect correction, frame quality enhancement, and element repair from natural-language instructions.Source: §7 · p. 27

Applications

  • Together, moderation and restoration form a closed loop that identifies non-compliant creatives, repairs problematic assets, and shortens the creative production cycle.The paper frames these complementary capabilities as improving advertiser delivery efficiency.Source: §7 · p. 27

8 Conclusion

Mamoda2.5 unifies visual generation, editing, and understanding in an AR–Diffusion model with a sparse DiT-MoE backbone and a few-step distillation and reinforcement learning framework. It delivers strong benchmark results and has been deployed in advertising scenarios.

8 Conclusion

  • The unified AR–Diffusion framework handles visual generation, editing, and understanding with 128 routed experts and Top-8 sigmoid routing, activating 3 billion parameters in a 25 billion-parameter model.Loss-free Expert Bias balancing supports routing, while warm-start upcycling accelerates MoE convergence by approximately 5× and improves instruction-following performance beyond the dense teacher.Source: §8 · p. 27
  • The 30-step Mamoda2.5 runs over 12× faster than Wan2.2 A14B on a single device, while the distilled 4-step model achieves an editing-inference speedup of approximately 15×.The joint distillation and reinforcement learning post-training framework is combined with MoE-tailored system optimizations and a high-compression VAE.Source: §8 · p. 27
  • Mamoda2.5 achieves top-tier video editing performance, outperforming most evaluated open-source models on OpenVE-Bench, Reco-Bench, and FiVE-Bench, and surpassing proprietary models including Kling O1 on OpenVE-Bench.It also reaches top-tier generation quality on VBench 2.0, excels on image generation and editing benchmarks, and maintains multimodal understanding competitive with its Qwen3-VL backbone.Source: §8 · pp. 27–28
  • Mamoda2.5 has been deployed for content moderation and creative restoration in real-world advertising scenarios, improving ad delivery efficiency.Source: §8 · p. 28

9 Future Work

Mamoda2.5 currently unifies image and video generation and editing, and the paper identifies audio integration and deeper understanding–generation synergy as future directions.

Omni Audio-Video Generation and Editing

  • Integrating audio processing could enable synchronized audio-video generation and editing, including coherent soundtracks, dialogue, and sound effects.This would broaden applicability to real-world content creation scenarios.Source: §9 · p. 28

Deeper Unification of Understanding and Generation

  • Building on GPT-Image-2 and Vision Banana, the authors aim to make understanding and generation mutually reinforce each other within Mamoda2.5.The cited systems illustrate generation as an interface for vision tasks and reasoning as a way to improve generation quality.Source: §9 · p. 28

A Appendix · A.1 Additional Evaluation Results

The appendix reports Reco-Bench evaluation under the original protocol, using Gemini 2.5 Flash as evaluator.

A.1 Additional Evaluation Results

  • Reco-Bench evaluation follows the original protocol and uses Gemini 2.5 Flash as evaluator.The detailed performance comparison is presented in Table 13.Source: §A.1 · p. 34
  • Table 13 marks the best and second-best results, with higher scores indicating better performance.The supplied passage does not include the table’s cell values.Source: Table 13 · p. 34

A.2 Additional Video Editing Visualizations

Mamoda2.5 shows highly competitive qualitative video editing, with results comparable to or surpassing closed-source models in real-world scenarios. The visualizations cover object edits, creative transformations, removals, and style changes while emphasizing temporal consistency and preservation of other video content.

A.2 Additional Video Editing Visualizations

  • Mamoda2.5 produces qualitative video edits comparable to or surpassing closed-source models in real-world scenarios.The comparisons include changing a cap and adding a tracked laptop while retaining the surrounding video content.Source: §A.2 · p. 34, Figure 17 · p. 35, Figure 18 · p. 35
  • The visualizations span imaginative transformations, including turning a paddleboarder into a water guardian and tea objects into miniature ecosystems.They also show snowy and pixel-art style changes, as well as person removal with temporally consistent background reconstruction.Source: Figure 19 · p. 36, Figure 20 · p. 36, Figure 21 · p. 37

Related papers

Loading 2605.02641v1…