PaperlayerView exact revision on arXiv

Source-linked AI summary

Mora: Enabling Generalist Video Generation via A Multi-Agent Framework

Zhengqing Yuan, Yixin Liu, Yihan Cao, Weixiang Sun, Haolong Jia, Ruoxi Chen, Zhaoxu Li, Bin Lin, Li Yuan, Lifang He, Chi Wang, Yanfang Ye, Lichao Sun

arXiv:2403.13248v3cs.CV

TL;DR

Long-form text-to-video generation remains difficult to reproduce in open systems, particularly given Sora’s closed-source nature and challenges in collaboration and training data. Mora combines specialized agents with synthesized, filtered training data; experiments report performance comparable to Sora across six tasks and better than existing open-source methods on VBench.

  • Problem

    Open-source methods struggle to match Sora, whose closed-source nature limits study and replication, while agent collaboration and training-data quality remain challenges.

    Source: §1 · pp. 1–2

  • Method

    Mora coordinates agents through self-modulated fine-tuning, synthesizes training data with large models, and filters it using human-in-the-loop multimodal models.

    Source: §1 · p. 2

  • Results

    Across six video-generation tasks, Mora is comparable to Sora on VBench and outperforms existing open-source methods; text-to-video Video Quality is 0.800 versus Sora’s 0.797.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Mora provides an accessible platform for future research on multi-agent video generation.

    Source: §1 · p. 2

    Mora’s video quality and length fall short of Sora, with noticeable degradation beyond 12 seconds, and it struggles to control motion direction from prompts.

    Source: D DISCUSSION · p. 31

Abstract

from arXiv · show

Text-to-video generation has made significant strides, but replicating the capabilities of advanced systems like OpenAI Sora remains challenging due to their closed-source nature. Existing open-source methods struggle to achieve comparable performance, often hindered by ineffective agent collaboration and inadequate training data quality. In this paper, we introduce Mora, a novel multi-agent framework that leverages existing open-source modules to replicate Sora functionalities. We address these fundamental limitations by proposing three key techniques: (1) multi-agent fine-tuning with a self-modulation factor to enhance inter-agent coordination, (2) a data-free training strategy that uses large models to synthesize training data, and (3) a human-in-the-loop mechanism combined with multimodal large language models for data filtering to ensure high-quality training datasets. Our comprehensive experiments on six video generation tasks demonstrate that Mora achieves performance comparable to Sora on VBench, outperforming existing open-source methods across various tasks. Specifically, in the text-to-video generation task, Mora achieved a Video Quality score of 0.800, surpassing Sora 0.797 and outperforming all other baseline models across six key metrics. Additionally, in the image-to-video generation task, Mora achieved a perfect Dynamic Degree score of 1.00, demonstrating exceptional capability in enhancing motion realism and achieving higher Imaging Quality than Sora. These results highlight the potential of collaborative multi-agent systems and human-in-the-loop mechanisms in advancing text-to-video generation. Our code is available at \url{https://github.com/lichao-sun/Mora}.

1 INTRODUCTION

Long-form text-to-video generation lags image generation, while Sora’s closed-source system is difficult to study and reproduce. Mora combines specialized agents using open-source modules with training-data synthesis and filtering, and reports results comparable to Sora across video-generation tasks.

1 INTRODUCTION

  • Open-source video generation trails image generation, and reproducing Sora is difficult because it is closed-source and costly to train a single model from scratch.Existing multi-agent approaches also struggle with agent collaboration and high-quality training data.Source: §1 · pp. 1–2
  • High-quality video data is costly to collect, scarce, and variable in quality, motivating human-in-the-loop filtering during synthesis.The paper combines human expertise with multimodal large language models to filter synthesized training data.Source: §1 · p. 2
  • Mora uses multi-agent fine-tuning with self-modulation, large-model-synthesized training data, and human-in-the-loop multimodal filtering.The contribution summary also describes human-in-the-loop controls for real-time adjustments.Source: §1 · p. 2
  • Across six video-generation tasks, Mora is comparable to Sora on VBench and outperforms existing open-source methods; text-to-video Video Quality is 0.800 versus Sora’s 0.797.For image-to-video, Mora scores 1.00 versus Sora’s 0.75 on Dynamic Degree and 0.67 versus 0.63 on Imaging Quality; for video editing, it matches Sora at 0.52 Imaging Quality and 0.24 Temporal Style.Source: §1 · p. 2

2 RELATED WORK

Prior video-generation work spans autoregressive and masked-model approaches, diffusion-based synthesis and editing, and multi-agent systems developed for other domains. Mora’s motivation is to apply collaborative agents to text-to-video generation.

2.2 AI AGENTS

  • Video-generation research also includes autoregressive and masked-model approaches, while multi-agent collaboration has been used for tasks such as software development, simulation, translation, and peer review.The paper positions this prior multi-agent work as motivation for applying agents to text-to-video tasks.Source: §2.1 · p. 3, §2.2 · p. 4

2.1 TEXT-TO-VIDEO GENERATION

  • Diffusion models progressed from video adaptations of image generation, including 3D U-Nets and latent diffusion, to methods for video synthesis and editing such as Stable Video Diffusion, Emu Video, Dreamix, and MagicEdit.The section also describes diffusion transformers and systems including Latte and Sora.Source: §2.1 · p. 3

3 MORA: A MULTI-AGENT FRAMEWORK FOR VIDEO GENERATION

Mora organizes video generation as a pipeline of specialized agents, with user control over intermediate outputs. Its training combines self-modulation, dynamically synthesized data, and multimodal-LLM and human review.

3.1 AGENT ARCHITECTURE OF MORA

  • Mora decomposes video generation into specialized collaborating agents, with the goal of producing prompt-aligned, visually high-quality, temporally coherent videos.The model maps text prompts to videos and seeks diverse outputs.Source: §3.1 · p. 4
  • The five agents handle prompt generation, text-to-image, image-to-image, image-to-video, and video connection tasks.The video connection agent uses keyframes and shared elements to create transitions between two input videos.Source: §3.1 · p. 5
  • Users can review and adjust the first frame before video generation, while the video connection agent blends adjacent clips for continuity.Users may request a regenerated frame or send it for image-to-image editing.Source: §3.1 · p. 5

3.2 SELF-MODULATED MULTI-AGENT FINETUNING

  • The training method addresses loss-allocation problems in end-to-end tuning and distribution misalignment in individual-agent tuning by updating agent parameters and modulation embeddings together.Updates minimize loss between the final output and target video.Source: §3.2 · pp. 5–6
  • Self-modulated fine-tuning concatenates a modulation embedding with the enhanced-prompt embedding, allowing agent contributions to adapt to the task and intermediate outputs.The modulation embedding is optimized alongside model parameters.Source: §3.2 · p. 6
  • Multimodal LLMs select videos when they agree; disagreements trigger human review, and candidate sets with no acceptable video are discarded.The selected videos are included in the training set.Source: §3.2 · p. 6
  • Mora synthesizes training data dynamically rather than using open-source datasets directly, addressing inconsistent data quality and missing intermediate outputs.An LLM first generates diverse enhanced prompts.Source: §3.2 · p. 6
  • Selected prompt-video pairs form each iteration’s training set, and updated agent parameters seed the next round of data generation and fine-tuning.The process repeats for N iterations.Source: §3.2 · p. 7

4 EXPERIMENTS

The experiments assess Mora on six video-generation tasks using VBench and self-defined metrics. Results vary by task: Mora leads on multiple text-to-video measures, matches Sora on some other-task measures, and trails on others.

4.1 SETUP

  • The setup covers six generation tasks, uses Sora-report-inspired prompts for text-to-video, compares against Sora and other baselines, and combines VBench with self-defined metrics.Source: §4.1 · p. 8

4.2 TRAINING

  • GPT-4o generated prompts for self-training; multimodal LLMs selected the best video per prompt, yielding 96 training examples.The paper reports that all tasks improved substantially in early iterations, with improvement slowing as iterations continued.Source: §4.2 · pp. 8–9, Figure 3 · p. 8

4.3 RESULTS

  • Finetuned Mora (OSP) outperformed all baselines, including Sora, on multiple benchmarks; it scored 0.98 Object Consistency.The paper identifies 0.98 as a leading Object Consistency score.Source: Table 1 · p. 9, §4.3 · p. 9
  • 1.00 Dynamic Degree and 0.67 Imaging Quality were Mora’s image-to-video scores; Mora and Sora tied at 0.90 VideoTI.Mora also scored 0.98 in Motion Smoothness, comparable to Sora.Source: §4.3 · pp. 9–10, Table 2 · p. 10
  • Mora scored 0.99 TCON and 0.24 Temporal Style versus Sora’s 0.98 and 0.23, and led in Imaging Quality at 0.45.These results are reported for Extend Generated Video.Source: Table 2 · p. 10, §4.3 · p. 10
  • Mora matched Sora in Video-to-Video Editing at 0.52 Imaging Quality and 0.24 Temporal Style.Source: Table 2 · p. 10, §4.3 · p. 10
  • Mora’s best Connect Videos scores were 0.43 Imaging Quality and 0.45 Tmean, below Sora’s 0.52 and 0.64, respectively.Source: Table 2 · p. 10, §4.3 · p. 10
  • In Simulate Digital Worlds, Sora scored 0.62 Imaging Quality to Mora’s 0.52, while both scored 0.23 Appearance Style.Source: Table 2 · p. 10, §4.3 · p. 10

5 CONCLUSION

Mora is presented as a generalist SOP-based framework for video tasks, with reported gains in adaptability, efficiency, and output quality. The conclusion also acknowledges a significant performance gap from Sora and points to research needs in language understanding, accessibility, and computational demands.

5 CONCLUSION

  • Mora combines SOPs in a generalist framework intended to address a broad spectrum of video-related tasks.The conclusion describes advances in adaptability, efficiency, and output quality.Source: §5 · pp. 10–11
  • Mora’s evaluation reports that it matches or surpasses leading models in several respects, yet still has a significant performance gap compared with Sora.Sora’s closed-source nature challenges replication and further innovation.Source: §5 · p. 11
  • Future work includes improving agents’ natural-language understanding, while accessibility and high computational demands remain barriers to broader adoption.The conclusion also calls for more open and collaborative research environments.Source: §5 · p. 11

A IMPLEMENTATION DETAILS

Experiments used eight Tesla H100 GPUs, four AMD EPYC processors, 320 GB of memory, and specified PyTorch and CUDA versions.

A.1 HARDWARE DETAILS

  • Experiments ran on eight Tesla H100 GPUs with 8×80 GB VRAM, four AMD EPYC 7552 48-core processors, and 320 GB memory.Video generation used PyTorch 2.0.2 and CUDA 12.2; evaluation used one Tesla A40 with PyTorch 1.10.2 and CUDA 11.6.Source: §A.1 · p. 18

A.2 METRICS

The appendix defines standard and custom metrics for video quality, consistency, style, and instruction adherence, then describes the workflows, agents, human oversight, and MLLM-assisted selection used across Mora’s video tasks.

A.2 METRICS

  • Object and background consistency, motion smoothness, aesthetics, dynamic degree, imaging quality, and two style metrics assess complementary video properties.The basic metrics use DINO, CLIP, AMT, LAION, RAFT, and MUSIQ; temporal and appearance style use ViCLIP and CLIP, respectively.Source: §A.2 · p. 18
  • Video-Text Integration measures semantic similarity between text representations of the input image plus instruction and the generated video.Temporal Consistency compares input and output video features; Tmean averages correlations with the preceding and following videos.Source: §A.2 · p. 18

A.3 DETAILS OF SOPS

  • Text-to-video starts from a prompt-generated image and extends video from the preceding clip’s last frame; image-to-video adds an initial image to the prompt.Other workflows extend an existing video, edit it by transforming its first frame before video generation, connect two videos using their boundary frames, or apply a digital-world style.Source: §A.3 · pp. 18–19

A.4 ITERATIVE PROGRAMMING WITH HUMAN IN THE LOOP

  • Users supervise each stage and may proceed or request a repeat, with at most three iterations per stage until the output is satisfactory.Figure 5 depicts structured agent communication and a human quality check after prompt or image generation.Source: §A.4 · pp. 19–20, Figure 5 · p. 19

A.5 FUNCTION OF AGENTS

  • The prompt agent extracts key information and actions, the text-to-image agent creates initial images, and the image-to-image agent edits images from instructions.The image-to-video agent turns an initial frame into a temporally stable sequence; the video connection agent uses key frames from two inputs to create a coherent transition.Source: §A.5 · p. 20

A.6 MLLMS TO JUDGE VIDEO QUALITY

  • Two MLLMs rank four candidate videos from random prompts, accepting an agreed top-ranked video and sending disagreements to human reviewers.This selection combines automated ranking with human judgment.Source: §A.6 · p. 20

A.7 DETAILS OF TRAINING PIPELINE

The training pipeline builds a prompt-video dataset and applies self-modulated multi-agent fine-tuning; its agents use different open-source models and architectures for image, video, and connection tasks.

A.7 DETAILS OF TRAINING PIPELINE

  • The training pipeline has two phases: constructing a training dataset and self-modulated fine-tuning.Figure 6 illustrates both phases.Source: §A.7 · p. 20, Figure 6 · p. 21
  • Enhanced prompts and human-vetted candidate videos are paired to form training data for fine-tuning.GPT-4o generates and selects detailed prompts to guide text-to-image generation.Source: §A.7 · p. 21, §A.8 · p. 22
  • During fine-tuning, agents’ text embeddings combine with a modulation embedding, and video-output loss updates model parameters and modulation embeddings.The agents adapt their outputs based on the preceding agent’s state.Source: §A.7 · p. 21

A.8 DETAILS OF AGENTS

  • The SVD configuration uses SDXL for initial images, InstructPix2Pix for instruction-based image editing, and Stable Video Diffusion for video generation.SDXL’s described design includes an enlarged UNet, dual text encoders, and an image-refinement stage; InstructPix2Pix performs edits in one forward pass.Source: §A.8 · pp. 22–23
  • The Open-Sora-Plan configuration uses SD3 for text-to-image and image editing, with rectified-flow sampling for image generation and modification.The passages also describe noise re-weighting and refinement for generated images.Source: §A.8 · p. 22
  • The Open-Sora-Plan video agent uses 3D full attention, temporal convolutions, and CausalVideoVAE to handle spatial-temporal features and refine frames.For video connections, the appendix describes SEINE’s random-mask video diffusion and a specialized architecture combining diffusion, temporal convolution, CausalVideoVAE, and 3D full attention.Source: §A.8 · p. 23

A.9 DETAILS SETTINGS IN TRAINING

Training uses a compact optimization setup, while inter-process communication bridges incompatible Open-Sora and SD3 environments for joint training.

A.9 DETAILS SETTINGS IN TRAINING

  • Training uses AdamW at 1e-5 with a constant schedule, no warmup, batch size 4, bf16, gradient checkpointing, 12 steps, SNR Gamma 5.0, and EMA 0.999.Gradient checkpointing saves memory at the cost of slower backward passes.Source: §A.9 · p. 24
  • Because Open-Sora and SD3 environments are mutually incompatible, inter-process communication supports joint training.The SD3 process generates an image for Open-Sora-Plan; its video loss is returned for SD3 backpropagation.Source: §A.9 · p. 24
  • Open-Sora-Plan generates video from an image and prompts, then computes the video loss against a ground-truth video.The pseudocode sends that loss back to SD3.Source: §A.9 · pp. 24–25

B.1 MORE RESULTS DETAILS

These results detail Mora’s text-to-video metrics and qualitative behavior across video generation, editing, extension, connection, simulation, and image-to-video tasks.

Text-to-Video Generation

  • 0.792 Video Quality places Mora just below Sora’s 0.797; Mora matches Sora’s 0.95 Object Consistency and leads other models on Dynamic Degree (0.70) and Imaging Quality (0.59).Mora scores 0.95 in Background Consistency, 0.99 in Motion Smoothness, and 0.26 in Temporal Style, versus Sora’s 0.96 and 0.35, respectively.Source: §B.1 · p. 25, Figure 7 · p. 25

Demo of Mora on Various Tasks

  • Mora’s demos cover video editing, video extension, and digital-world simulation.Figure 8 presents demonstrations across various tasks.Source: §B.1 · p. 25, Figure 8 · p. 26

Text-conditional Image-to-Video Generation

  • Mora and Sora both preserve the monster illustration and cloud spelling “SORA” while animating the input imagery and text into coherent video.The qualitative comparison says Mora’s results nearly rival Sora’s.Source: §B.1 · pp. 25–26, Figure 9 · p. 27

Text-to-Video Generation

  • 0.848 overall Video Quality is the highest reported for Mora (Open-Sora-Plan) in Table 4, where it also leads in consistency, aesthetic quality, and imaging quality.The reported scores are 0.98 for object and background consistency, 0.70 for aesthetic quality, and 0.72 for imaging quality.Source: §B.1 · p. 25, Table 4 · p. 26

Extend Generated Videos

  • Mora’s extensions preserve narrative flow and visual continuity, including the rider’s motion and the surrounding environment’s blur.The qualitative comparison describes this as near parity with Sora.Source: §B.1 · p. 26, Figure 10 · p. 28

Video-to-Video Editing

  • Mora’s video edit changes a modern setting toward a 1920s style while preserving the car’s red color, though its environment looks less authentic than Sora’s.The sampled result follows the instruction but is described as less refined.Source: §B.1 · pp. 26–27, Figure 11 · p. 28

Connect Videos

  • In video connection, Mora’s blurred intermediate background makes objects indistinguishable, unlike Sora’s synthesis of elements from preceding and subsequent footage.The passage identifies intermediate-image fidelity and sequence consistency as areas for improvement.Source: §B.1 · p. 27, Figure 12 · p. 29

Simulate Digital Worlds

  • Mora evokes Minecraft’s digital-world style, but its samples have more muted colors, less distinct edges, and lower apparent resolution than Sora’s.The comparison characterizes Mora as not yet reaching Sora’s visual-fidelity level.Source: §B.1 · pp. 27–28, Figure 13 · p. 29

B.2 ABLATION STUDY

The ablation results report that the full Mora configuration outperforms variants across most metrics, with distinct effects from removing human guidance or self-modulation.

B.2 ABLATION STUDY

  • 0.800 Video Quality is the full Mora model’s best score among the ablated variants, alongside leading scores in object consistency (0.98), background consistency (0.97), motion smoothness (0.99), and imaging quality (0.70).The passage reports that removing any component causes a noticeable performance drop.Source: §B.2 · pp. 29–30
  • Removing human-in-the-loop guidance lowers Video Quality to 0.785 and Motion Smoothness to 0.95.The authors describe human guidance as important to maintaining video-generation quality and consistency.Source: §B.2 · pp. 29–30
  • Removing self-modulation lowers Video Quality to 0.776 while raising Dynamic Degree to 0.51; random-initial modulation reaches 0.797 Video Quality but remains below full Mora overall.The paper attributes the full model’s advantage over random initialization to learned modulation embeddings.Source: §B.2 · pp. 29–30

C CAPABILITIES ANALYSIS

The capability comparisons describe Mora’s use of SOP-like agent workflows and human control, alongside its coverage of video-generation tasks undertaken by Sora.

C CAPABILITIES ANALYSIS

  • Mora combines role-play expertise, structured communication, streamlined workflows, and human-in-the-loop systems for video generation.The comparison says other frameworks typically apply SOP-like designs only to code generation.Source: C CAPABILITIES ANALYSIS · p. 30
  • The comparison characterizes Sora as lacking fine-grained control and autonomous video generation, capabilities attributed to Mora.Mora’s framework is compared with Sora and autonomous-agent frameworks such as MetaGPT.Source: C CAPABILITIES ANALYSIS · pp. 30–31
  • The task comparison reports that Mora can accomplish video-related tasks that Sora can undertake through collaboration among multiple agents.The passage presents this as evidence of Mora’s adaptability across video-generation challenges.Source: Table 7 · p. 31, C CAPABILITIES ANALYSIS · p. 31

D DISCUSSION

Mora’s open framework supports diverse video-generation applications and offers flexibility through integration of different models. The discussion also identifies constraints in data, video quality and length, and motion control.

D DISCUSSION

  • Mora’s open framework integrates different models for diverse applications, including text-to-video conversion and digital world simulation.The paper presents this openness as a way to democratize access and foster collaboration and innovation.Source: D DISCUSSION · p. 31
  • Future work should improve Mora’s efficiency, reduce computational demands, and explore new agent configurations to enhance performance.Source: D DISCUSSION · p. 31
  • Mora’s video quality and length fall short of Sora, with noticeable quality degradation beyond 12 seconds.The paper also notes difficulty collecting high-quality datasets under copyright restrictions, which contributes to challenges generating lifelike human movements.Source: D DISCUSSION · p. 31
  • Mora struggles to interpret and render motion dynamics from prompts and lacks control over specific motion directions.The paper links the absence of human labeling in video datasets to outputs that may not align with human visual preferences, and calls for datasets adhering to physical laws.Source: D DISCUSSION · p. 31
Loading 2403.13248v3…