Movie Gen: A Cast of Media Foundation Models

Capabilities

• Movie Gen provides high-fidelity image, video, and audio generation alongside video editing and personalization capabilities.Figure 1 illustrates text-to-video generation, reference-image-consistent personalization, instruction-guided precise editing, and synchronized audio.Source: §1 · p. 1, Figure 1 · p. 2

Model development

• Scaling internet-scale data, compute, and Transformer parameters with Flow Matching yields Movie Gen Video at 30B parameters and Movie Gen Audio at 13B parameters.Personalized Movie Gen Video is obtained by post-training the video model, while Movie Gen Edit uses a novel post-training procedure for precise editing.Source: §1 · p. 1
• Movie Gen enables realistic personalized HD videos up to 16 seconds at 16 FPS, 48kHz audio, and editing of real or generated videos.These capabilities span the Movie Gen Video, Personalized Movie Gen Video, Movie Gen Edit, and Movie Gen Audio models.Source: §1 · p. 1

Results

• Movie Gen outperforms prior state-of-the-art systems on text-to-video quality, video personalization, precise video editing, sound effects, music, and audio extension.Comparisons include Runway Gen3, LumaLabs, and OpenAI Sora for text-to-video, and PikaLabs and ElevenLabs for audio; the cited tables report the task-specific evaluations.Source: §1 · pp. 1, 3

Resources

• The paper publicly releases Movie Gen Video Bench and Movie Gen Audio Bench to enable future benchmarking and provides details intended to accelerate media-generation research.The paper also reports model architectures, training, inference, and experimental settings.Source: §1 · p. 3

System Overview

• The overall Movie Gen system combines synchronized audio, personalized characters, and video editing capabilities.These capabilities are illustrated in Figure 1 and added to the foundation video model through post-training procedures.Source: §2 · p. 3

Movie Gen Video

• Movie Gen Video is a 30B-parameter model generating high-quality HD images and videos up to 16 seconds across aspect ratios, resolutions, durations, and text prompts.It is jointly pretrained on O(100)M videos and O(1)B images, then supervised-finetuned on curated high-quality videos and captions.Source: §2 · p. 3

Movie Gen Audio

• Movie Gen Audio is a 13B-parameter model producing synchronized 48kHz cinematic sound effects and music from videos or text, including coherent audio for videos lasting several minutes.It supports diegetic ambient sounds and action-synchronized effects, non-diegetic mood-aligned music, and professional blending, with supervised finetuning targeting higher quality and cinematic styles.Source: §2 · p. 3

Personalization

• Personalization conditions video generation on text and a person’s image while preserving identity and following the prompt, using automatically constructed image-text-to-video training pairs.The capability is added through post-training on a subset of videos containing humans.Source: §2 · p. 3

Precise Editing

• Precise Editing enables textual, imaginative edits to real or generated videos without large-scale supervised editing data, using a novel training approach.Examples are provided in the cited video-editing figure resource.Source: §2 · p. 3

Training Recipe

• Movie Gen Video is trained by text-to-image pretraining followed by joint text-to-image and text-to-video pretraining at increasingly higher resolutions, then finetuned on high-aesthetic, high-motion-quality videos.The recipe also adds personalization and video-to-video editing capabilities.Source: Figure 2 · p. 4

3 Joint Image and Video Generation

• A single model generates both images and videos, using images as single-frame videos and staged joint training followed by high-quality video finetuning and post-training for personalization and editing.The stages progress from 256 px image pretraining to joint low-resolution image-video training, high-resolution joint training, and high-quality video finetuning.Source: §3 · p. 4

3.1.6 Model Scaling and Training Efficiency

• The largest model has 30B parameters, uses up to 6,144 H100 GPUs, and leaves Grouped-Query Attention unexplored while directly generating videos across specified lengths, resolutions, and aspect ratios.It directly generates 4–16 second videos at 768 × 768 px across 1:1, 9:16, and 16:9 aspect ratios; the Spatial Upsampler produces full HD 1080p output.Source: §3 · p. 4, §3.1.6 · p. 10

3.1.1 Temporal Autoencoder (TAE)

• The Temporal Autoencoder compresses images and videos 8× across time and both spatial dimensions, reducing Transformer sequence length and eliminating the need for frame interpolation.Images are encoded as single-frame videos, and variable-length inputs are supported through temporal strided convolutions; the latent channel count is C = 16.Source: §3.1.1 · p. 5, Figure 4 · p. 6
• The TAE’s outlier penalty loss removes spot artifacts caused by high-norm latent dots, using r = 3 and a loss weight of 1e5.The penalty is added to reconstruction, discriminator, and perceptual VAE losses rather than changing the architecture.Source: §3.1.1 · p. 6

3.1.2 Training Objective for Video and Image Generation

• Flow Matching trains the joint generator to predict latent velocity from noisy samples and naturally enforces zero terminal signal-to-noise ratio, outperforming diffusion losses empirically.Inference starts from Gaussian noise and uses an ODE solver to recover the output latent; the implementation uses a first-order Euler solver.Source: §3.1.2 · pp. 7–8

3.1.3 Joint Image and Video Generation Backbone Architecture

• The backbone follows LLaMa3 with 3D patchification, factorized positional embeddings, cross-attention text conditioning, adaptive timestep normalization, and full bidirectional attention.Adding positional embeddings at every Transformer layer reduces temporal distortion and morphing artifacts, while the simple design scales across model sizes and training settings.Source: §3.1.3 · p. 8

3.1.4 Rich Text Embeddings and Visual-text Generation

• UL2, ByT5, and Long-prompt MetaCLIP provide complementary semantic, character-level, and visually aligned text embeddings, concatenated after projection and normalization into 6144 dimensions.ByT5 is used specifically for visual text, while FPS conditioning controls generated video length.Source: §3.1.4 · p. 9

3.1.5 Spatial Upsampling

• A separate 7B Spatial Upsampler converts 768 px videos to full HD 1080p through conditional video-to-video latent generation, reducing computation in the base text-to-video model.MultiDiffusion averages overlapping-window latents during denoising to improve temporal consistency at sliding-window boundaries.Source: §3.1.5 · pp. 9–10, Figure 7 · p. 10

3.1.6 Model Scaling and Training Efficiency

• Movie Gen uses 3D parallelism to scale model-level training across parameter count, input tokens, and dataset size while scaling horizontally to more GPUs.For self-attention, context parallelism performs an all-gather only for K and V projections in the forward pass, followed by a reduce-scatter for their gradients.Source: §3.1.6 · pp. 11–12

3.2.1 Pre-training Data

• The pre-training pipeline filters videos by visual quality, motion, and content, then generates detailed captions and camera-motion labels for diverse clip-prompt pairs.The final data uses clips of 4–16 seconds, supports five aspect-ratio and five duration buckets, and enables 16–32 FPS control through an FPS token.Source: §3.2.1 · pp. 12–13

3.2.2 Training

• Training proceeds from 256 px text-to-image warm-up to joint text-to-image/text-to-video training and progressive scaling to 768 px resolution.Warm-up improves convergence and visual-temporal quality over training joint models from scratch, while validation loss correlates with human-rated visual quality.Source: §3.2.2 · p. 14

3.3 Finetuning

• Supervised finetuning uses manually curated, concept-balanced cinematic videos and captions, then averages models trained with varied data, hyperparameters, and checkpoints.The finetuning set targets strong motion, realness, aesthetics, and camera control, with durations from 10.6 to 16 seconds.Source: §3.3 · p. 15

3.4 Inference

• Inference uses classifier-free guidance, 50-step sampling, and rewritten prompts to bridge the gap between short user inputs and dense training captions.For prior-work comparisons, the text guidance scale is 7.5, and sampling uses the linear-quadratic sampler.Source: §3.4 · pp. 15–16

3.4.1 Inference Prompt Rewrite

• A LLaMa3-based rewrite model standardizes and expands user prompts, while avoiding excessively elaborate motion descriptions that can introduce artifacts.A distilled 8B student model reduces rewrite latency after teacher generation and human evaluation of rewrite pairs.Source: §3.4.1 · p. 16

3.4.2 Improving Inference Efficiency

• A linear-quadratic t-schedule closely approximates N-step video generation with 50 steps, providing up to ∼20× inference speedup without additional training.The schedule preserves the first 25 linear steps and replaces later steps with 25 quadratic steps; in practice, 50 steps emulate N = 250 linear steps.Source: §3.4.2 · p. 17

3.5 Evaluation

• Movie Gen evaluates text-to-video quality with Movie Gen Video Bench and expert human pairwise A/B tests across multiple axes, allowing winners or ties.Evaluators compare videos side-by-side and select the winner for the measured axis or declare a tie when neither is clearly better.Source: §3.5 · p. 17

3.5.1 Evaluation Axes

• The evaluation uses three axes—text-alignment, visual quality, and realness and aesthetics—with motion-specific sub-axes measuring consistency, completeness, naturalness, and overall quality.Text-alignment separates subject and motion match, while realness and aesthetics assess photorealism and visual appeal.Source: §3.5.1 · pp. 17–18, Table 4 · p. 17

3.5.2 Evaluation benchmark

• Movie Gen Video Bench publicly releases 1003 prompts spanning human activity, animals, nature, physics, unusual subjects, and low-, medium-, and high-motion conditions.The benchmark is more than 3× larger than prompt sets used by prior work and includes non-cherry-picked Movie Gen Video outputs for future comparison.Source: §3.5.2 · pp. 18–19

3.5.3 Evaluation Discussion

• Human evaluation is necessary because temporal alignment, motion naturalness, cross-frame consistency, realness, and aesthetics require judgments that automated metrics fail to capture reliably.Reliability is improved through detailed guidelines, reason annotations, broad prompt coverage, majority votes from three or six annotators, and annotation audits.Source: §3.5.3 · pp. 19–20

3.6 Results

• The experiments compare prior methods, ablate design choices, evaluate the TAE and Spatial Upsampler, and compare Movie Gen Video with prior text-to-image methods.For prior-work comparisons, videos are matched in resolution and aspect ratio to reduce annotator bias, and OpenAI Sora comparisons use manually selected Movie Gen Video videos.Source: §3.6 · p. 20, §3.6.1 · p. 20

3.6.1 Comparisons to Prior Work

• Movie Gen Video strongly outperforms Runway Gen3 and LumaLabs on overall quality, moderately beats OpenAI Sora, and is on par with Kling1.5.It also significantly wins on realness against Runway Gen3, LumaLabs, Kling1.5, and Sora, while outperforming or matching competitors on text faithfulness.Source: Table 6 · p. 21, §3.6.1 · p. 21

3.6.2 Ablations

• Flow Matching, video captions, and the LLaMa3-based architecture each improve generation quality relative to their ablation baselines.Video captions improve overall prompt alignment by 10.8%, including +10.7% motion alignment and +16.1% on high-motion prompts; the LLaMa3 architecture beats Diffusion Transformer by 18.6% in quality and 12.6% in text-alignment.Source: §3.6.2 · pp. 24–25
• Compute-optimal Movie Gen model sizes align closely with the LLaMa3 scaling law across 5B, 9B, 17B, and 30B models.The models were evaluated across compute budgets from 1022 to 1.7 × 10^22 FLOPs, suggesting LLaMa3 scaling laws may predict media-generation model sizes and compute budgets.Source: §3.6.2 · p. 25, Table 10 · p. 26

3.6.3 TAE Results

• The temporal autoencoder achieves competitive video reconstruction relative to a frame-wise autoencoder while providing 8× higher temporal compression, and outperforms it on images.Reconstructions preserve visual detail, although high-frequency spatial details and fast motion can be smoothed or degraded.Source: §3.6.3 · p. 26, Table 11 · p. 26, Figure 16 · p. 26

3.6.4 TAE Ablations

• The TAE uses 2.5D rather than 3D attention and convolutions because 3D gives only slightly better reconstruction metrics at substantially higher memory and compute cost.Outlier-penalty-loss finetuning also improves reconstruction quality for both images and videos and removes artifacts.Source: §3.6.4 · p. 27, Table 12 · p. 27, Table 13 · p. 27

3.7 Text-to-Image Generation

• Movie Gen is trained jointly on videos and images, enabling it to generate both modalities.For image generation, the model is continued with an image autoencoder and evaluated against prior text-to-image systems.Source: §3.7 · p. 28

3.7.1 Method

• The text-to-image model replaces the temporal autoencoder with an image autoencoder and is trained to generate 1024 px images.Post-training uses O(1000) artist-created images for quality tuning, with 6k finetuning steps at learning rate 0.00001 and batch size 64.Source: §3.7.1 · p. 28

3.7.2 Results

• The Movie Gen text-to-image model achieves the highest ELO rating among contemporary text-to-image methods in the benchmark.The comparison uses human pairwise evaluations converted into ELO rankings across text faithfulness and visual quality.Source: §3.7.2 · p. 28

4.1 Model

• PT2V extends the 30B Movie Gen Video model with reference-image identity conditioning, concatenating projected vision tokens with text features in a unified architecture.Identity features come from a trainable Long-prompt MetaCLIP vision encoder initialized from Xu et al. (2023), followed by pre-training and high-quality finetuning.Source: §4.1 · pp. 28–29, Figure 20 · p. 30

4.2.1 Pre-training Data

• The pre-training corpus contains O(1)M single-person text-video pairs and O(10)M paired samples, supplemented by real and synthetic cross-paired data to prevent copy-paste shortcuts.Videos contain one person across 4s–16s, while cross-paired references come from different videos of the same subject; synthetic filtering retains ArcFace similarity above 0.7.Source: §4.2.1 · pp. 29–30

4.2.2 Pre-training recipe

• PT2V pre-training uses three stages: short-video identity injection, longer-video recovery for temporal coherence, and cross-paired training for natural expressions, motion, and detailed identity features.Stage I truncates the TAE embedding to 8 latent frames, Stage II increases latent-frame length, and Stage III finetunes the vision encoder.Source: §4.2.2 · p. 31, Figure 21 · p. 31

4.3 Supervised Finetuning

• High-quality supervised finetuning uses aesthetic data and a manually selected set of O(1000) videos featuring diverse human actions with paired and real cross-paired references.The finetuning stage targets highly aesthetic videos with good-quality motion and matches Movie Gen Video’s visual quality.Source: §4.3 · p. 31, §4.3.1 · p. 31

4.4 Evaluation

• Evaluation measures identity preservation, video quality, and video-text alignment using 50 unseen subjects, 330 image-prompt pairs, and best-frame, worst-frame, and face-consistency identity ratings.The evaluation prompts cover varied human activities and facial expressions, with both frontal and side-view reference images.Source: §4.4 · p. 32

4.5 Results

• PT2V significantly outperforms ID-Animator across identity preservation, video quality, and text alignment, while beating or matching unconditioned T2V on selected quality metrics.Against T2V, PT2V wins consistency and performs on par in overall quality, but loses in motion completeness and prompt alignment.Source: §4.5 · p. 32, Table 16 · p. 33

4.5.1 Ablations

• A trainable vision encoder improves identity preservation by 16% relative to a fixed encoder, while cross-paired training improves text alignment by 27.36%, overall quality by 13.68%, and motion naturalness by 26.14%.Cross-paired training slightly reduces identity metrics but produces more diverse head poses and more natural expressions.Source: Table 15 · p. 32, §4.5.1 · p. 33, Table 17 · p. 33
• High-quality finetuning improves video quality, naturalness, and text alignment substantially, while slightly reducing identity preservation because its dataset includes cross-paired references.Qualitative generations follow identities across diverse motions and camera views, and consistently outperform ID-Animator in identity consistency and video quality.Source: §4.5 · pp. 32–33, §4.5.1 · p. 33, Figure 23 · p. 35

5.1 Model

• Movie Gen Edit trains without supervised video-editing data through three stages that progressively reduce train-test discrepancies: single-frame editing, multi-frame editing, and backtranslation.The stages are summarized in Figure 24 and address discrepancies introduced by training on proxy or synthetic tasks.Source: §5 · p. 36, §5.1 · p. 36, Figure 24 · p. 37

5 Instruction-Guided Precise Video Editing

• Human annotators preferred Movie Gen Edit over the previous state-of-the-art more than 74% of the time on TGVE+.The comparison is against Singer et al. (2024) on the TGVE+ benchmark (Wu et al., 2023c; Singer et al., 2024).Source: §5 · p. 36

5.1.1 Model Architecture and Initialization

• The editing architecture conditions on the input video and task identity while zero-initializing new weights to preserve the pretrained text-to-video model’s generation capabilities.Input-video latents are concatenated with noisy output latents, and task embeddings condition both text representations and timestep embeddings.Source: §5.1.1 · p. 37

5.1.2 Stage I: Single-frame Video Editing

• Stage I alternates image editing and text-to-video generation, using image editing as single-frame video editing, but initially produces blurry edited videos because training lacks multi-frame inputs.The joint objective also supports temporal consistency and generation quality, while black-video conditioning adapts text-to-video examples to the editing architecture.Source: §5.1.2 · pp. 37–38, §5.1.3 · p. 38

5.1.3 Stage II: Multi-frame Video Editing

• Stage II adds animated frame editing and instruction-guided video segmentation to expose the model to multi-frame inputs and outputs, mitigating Stage I blurriness.The model is then trained with text-to-video generation; generated elements show less motion than desired and can appear oversaturated.Source: §5.1.3 · pp. 38–39

5.1.4 Stage III: Video Editing via Backtranslation

• Stage III adapts backtranslation by training on weakly supervised pairs that use generated edited videos as inputs and real videos as targets, addressing unnatural motion and oversaturation.Reverse editing instructions are generated to transform an edited video back into its original video, producing training data with real output videos.Source: §5.1.4 · pp. 39–40, Figure 26 · p. 40

5.2.1 Video Editing Benchmarks

• Movie Gen Edit Bench provides a realistic evaluation set with varied resolutions, frame rates, lengths, and aspect ratios, unlike the short, square, low-resolution TGVE+ videos.The benchmark contains 1,152 human-instructed examples spanning six editing tasks, with 64 validation and 128 test videos.Source: §5.2 · p. 40, §5.2.1 · pp. 40–41

5.2.2 Video Editing Measures

• Evaluation measures instruction and input-video fidelity through ViCLIP direction and output similarity metrics, alongside human judgments of text alignment, structure, quality, and overall preference.Human raters compare pairs of edited videos given the input video and editing instruction.Source: §5.2.2 · p. 41

5.3.1 Comparisons to Prior Work

• Movie Gen Edit is preferred over all baselines on both benchmarks, including 74% more frequent preference than EVE on TGVE+ overall human evaluation.It achieves state-of-the-art ViCLIPdir and comparable ViCLIPout performance to EVE, although EVE uses the output caption for ViCLIPout.Source: §5.3.1 · p. 42

5.3.2 Ablations

• The ablations evaluate design choices using Movie Gen Edit Bench human metrics unless otherwise stated, alongside image-editing metrics including L1, DINO, CLIPim, CLIPout, and CLIPdir.Human raters assess text alignment and image faithfulness by comparing edited outputs against the original image and instruction.Source: §5.3.2 · pp. 42–43
• The ControlNet alternative freezes the text-to-video model, trains a cloned model with zero-initialized convolutional layers, and requires separate video-caption and editing-instruction inputs.The adapter is trained for 10K iterations on the Stage I image-editing data and compared with the Stage I model trained for the same duration.Source: §5.3.2 · pp. 42–43
• Stage I multitask training produces significantly better CLIPdir and L1 performance than a ControlNet adapter, while matching it on CLIPim, CLIPout, and DINO.Human evaluation also favors full model training for instruction alignment and faithfulness to the input image; Table 19 compares the Stage I model against the ControlNet adapter.Source: §5.3.2 · p. 43, Table 19 · p. 43
• Stage II training on animated frames is preferred to training on animated images, with over 70% preference for text faithfulness and over 61% preference for quality.Table 20 reports the human-evaluation win rate of Stage II against its animated-image counterpart.Source: Table 20 · p. 43, §5.3.2 · p. 44
• Stage III backtranslation slightly reduces text faithfulness versus standard fine-tuning but substantially improves structure, quality, and overall preference.Table 21 compares Movie Gen Edit using backtranslation with Stage III trained by standard fine-tuning.Source: Table 21 · p. 43, §5.3.2 · p. 44
• Stage II improves over Stage I with human preference exceeding 89%, while Stage III improves over Stage II with preference exceeding 60%.Stage III’s main benefit is improved edited-video quality, with only a very minor trade-off in text faithfulness; Table 22 summarizes stage-by-stage win rates.Source: §5.3.2 · p. 44, Table 22 · p. 44

6 Joint Sound Effect and Music Generation

• The model generates up to 30 seconds of audio in one shot and extends it to arbitrary lengths using masked audio prediction with surrounding context.Context may be empty for generation, before or after the target for extension, or around it for infilling.Source: §6 · p. 44

6.1 Model

• Movie Gen Audio uses flow matching with a diffusion transformer, whose shared modulation MLP reduces parameters without sacrificing performance.Flow matching is selected for training and inference efficiency and performance, while the DiT conditions normalization, attention, and feed-forward outputs.Source: §6.1 · p. 45, §6.1.1 · p. 46, §6.1.2 · p. 46

6.1.3 Audio Representation and Conditioning Modules

• The audio latent representation uses 48kHz DAC-VAE features at 25Hz with 128 channels, improving reconstruction quality over 24kHz, 75Hz Encodec features.Removing residual vector quantization and training with a VAE objective improves reconstruction, especially at the compressed 25Hz frame rate.Source: §6.1.3 · pp. 46–47
• Video, masked audio context, and text captions condition generation through aligned visual features, channel-concatenated audio context, and cross-attention text features.Frame-level Long-prompt MetaCLIP features improve video-audio alignment, while audio context supports coherent infilling and extension.Source: §6.1.3 · p. 47

6.1.4 Inference: One-shot Generation

• Independent dropout of video, audio context, and text enables one model to perform V2A, text-instructed V2A, infilling, and extension.Changing which conditioning inputs are present provides the different generation modes without separate models.Source: §6.1.4 · p. 47

6.1.5 Inference: Audio Extension

• Long-form extension splits videos into overlapping segments and propagates information through context conditioning or trajectory regularization in autoregressive generation, whereas multi-diffusion solves segments in parallel.Triangular window weighting produces smoother overlap transitions than uniform weighting, which can create boundary discontinuities.Source: §6.1.5 · pp. 47–49
• Multi-diffusion achieves slightly better results than segment-level autoregressive generation and is therefore used as the default audio-extension method.Both methods process overlapping video segments and consolidate predictions to generate coherent long-form audio.Source: §6.1.5 · pp. 47–48

6.1.6 Model, Training, and Inference Configurations

• The 13B-parameter DiT has 36 layers with attention/feed-forward dimensions of 4,608/18,432, and is pretrained and fine-tuned on video segments up to 30 seconds.Pretraining uses 500K updates, an effective batch size of 1,536 sequences, 384 GPUs, and 14 days of training.Source: §6.1.6 · p. 49
• Training and inference use classifier-free conditioning dropout, a 64-step midpoint solver, guidance weight 3, eight-candidate reranking, and task-specific quality scores.Fine-tuning runs for 50K updates on 64 GPUs with an EMA checkpoint; audio extension additionally uses dynamic guidance and multi-diffusion.Source: §6.1.6 · pp. 49–50

6.2.1 Audio-Visual Data Categorization

• Movie Gen Audio jointly generates audio classes because correlations exist between video and audio and among diegetic, non-diegetic, vocal, musical, and sound-effect categories.The work focuses primarily on diegetic general sounds, non-diegetic sound effects, and instrumental music, while omitting diegetic and non-diegetic speech under stated conditions.Source: §6.2.1 · pp. 50–51

6.2.2 Pre-training Data

• Pre-training categorizes audio using AudioSet-derived AED tags and CAVTP audio-video similarity, prioritizing diegetic or mixed audio and general sounds while filtering silence and low-quality visuals.Videos are filtered for OCR text, static content, low resolution, unsuitable lengths, and visual duplicates; only diegetic and mixed diegetic/non-diegetic videos are used for pre-training.Source: §6.2.2 · pp. 51–52, Table 25 · p. 52

6.2.3 Finetuning Data

• Fine-tuning mixes professionally produced cinematic clips with high-quality music and sound-effect audio, using a 10:1 cinematic-to-audio batch ratio to improve sound design and audio quality.The high-quality split contains O(10)K hours each of music and sound effects, while cinematic clips are filtered automatically and selected with human annotation.Source: §6.2.3 · p. 52, Table 26 · p. 52

6.2.4 Caption Structure and Synthetic Caption

• Synthetic captions combine audio quality, voice and music presence, general sound descriptions, and music style, with AED probabilities and music captions improving control despite music-caption hallucinations.Audio quality is scored from 1 to 10, and samples are captioned at both 10-second and 30-second chunk lengths.Source: §6.2.4 · pp. 52–53

6.3 Evaluation

• Evaluation prioritizes audio quality and audio-video alignment over text alignment, using subjective Net Win Rate alongside automatic AQual, ImageBind, and CLAP-based measures.Video alignment assesses sound correctness and synchronization for diegetic audio, mood and action scoring for music, while text alignment measures event precision and recall.Source: §6.3 · p. 53, §6.3.1 · p. 54

6.3.2 Audio Generation Benchmarks

• The released Movie Gen Audio Bench contains 527 reviewed synthetic videos spanning 36 audio categories and 434 concepts, enabling broad video-to-audio evaluation.The benchmark covers ambient environments and sound effects, and releases non-cherry-picked Movie Gen Audio generations for fair future comparison.Source: §6.3.2 · pp. 54–55
• The benchmark suite separates real single-shot, generated single-shot, and multi-shot videos to test robustness, generalization, video-music alignment, sound design, and scene-transition handling.The multi-shot MGen set contains 107 15-second segments derived from 26 short films, while single-shot sets include VGGSound, OpenAI Sora, Runway Gen3, and Movie Gen Audio Bench.Source: §6.3.2 · p. 55

6.4.1 Comparisons to Prior Work

• Movie Gen Audio outperforms all baselines across synchronization, correctness, and overall quality, with net-win ranges of 33.8%–72.8%, 27.5%–82.2%, and 31.3%–91.0%, respectively.It wins more strongly against commercial baselines on generated videos and shows larger gains in professionalness than naturalness.Source: Table 30 · p. 57, §6.4.1 · p. 57
• For joint SFX and music, Movie Gen Audio significantly surpasses all baselines, including separately mixed systems and Seeing&Hearing TV2A, especially on alignment and quality.PikaLabs and ElevenLabs were excluded because preliminary testing found they could not jointly generate sound effects and music.Source: §6.4.1 · pp. 57, 59, Table 31 · p. 58
• Movie Gen Audio extension outperforms stitching and Seeing&Hearing, while independent stitching causes abrupt transitions, incoherent music, and worse audio quality and video-music alignment.The comparison probes smooth transitions using 5.5-second segments because long-form subjective evaluation had limited data and high rater variance.Source: §6.4.1 · pp. 59–60, Table 32 · p. 60

6.4.2 Ablations

• Conditioning audio quality improves predicted and subjective quality, with SFX+music alignment increasing from IB 0.28 to 0.32 between quality settings 5.0 and 6.SFX alignment remains nonsignificant, with IB scores between 0.33 and 0.35; subjective quality preferences largely saturate around 6.5–7.0.Source: §6.4.2 · p. 60, Table 33 · p. 60, Figure 33 · p. 61
• Text prompts improve subjective quality and most alignment metrics, enable desired sound-effect selection, and yield CLAP 0.37 versus 0.23 without prompts.A model trained with captions but inferred without them outperforms a model trained without captions, showing that captions facilitate audio-visual correspondence.Source: §6.4.2 · p. 61, Table 34 · p. 61
• Scaling from 300M to 13B parameters generally improves performance across all metrics, while fine-tuning significantly enhances audio quality, video alignment, and cinematic character.The scaling study uses a 300M model following Vyas et al. (2023) and DiT architectures for larger models.Source: §6.4.2 · p. 62, Table 35 · p. 62, Table 36 · p. 62
• Adding high-quality audio-only data significantly improves quality, slightly improves SFX video alignment, and significantly improves video-music alignment for joint generation.Large-scale text-sound-effect and text-music pairs help disentangle audio types, improving alignment and overall generated-sound quality.Source: §6.4.2 · p. 62, Table 37 · p. 62
• Multi-diffusion generally matches or exceeds alternatives: it outperforms most methods on 3B quality, matches one-shot generation at 3B, and is marginally better at 13B.The triangle window improves 3B quality over uniform windowing, while beam search improves autoregressive generation at 3B; these gaps disappear or narrow at 13B.Source: §6.4.2 · pp. 62–63, Table 38 · p. 63

7.1 Text-to-Image Generation

• Text-to-image systems improve efficiency, caption quality, image quality, and multimodal generation through latent diffusion, prompt rewriting, higher-dimensional latents, and flow transformers; MovieGen combines these ideas.MovieGen uses a 16-channel variational autoencoder, flow transformers, and prompt rewriting for visual quality and text alignment.Source: §7.1 · p. 63

7.2 Text-to-Video Generation

• Text-to-video research explores cascaded, latent, image-conditioned, space-time, autoregressive, and transformer architectures, while MovieGen uses a Llama3 transformer with flow matching.Its autoencoder is a continuous convolutional VAE with discriminator loss, an interleaved 2D-1D convolutional encoder, and symmetrical padding for joint image-video encoding.Source: §7.2 · pp. 64–65

7.3 Image and Video Personalization

• Personalized image generation uses identity-specific tuning or shared-base-model vision injection, whereas personalized video must handle complex motion, natural expressions, and higher computational costs.MovieGen follows the more generalizable identity-extraction and control direction rather than identity-specific tuning.Source: §7.3 · pp. 65–66

7.4 Instruction-Guided Video Editing

• Instruction-guided video editing requires precise changes while preserving unspecified content, but supervised data scarcity has favored imprecise training-free and feature-based methods; MovieGen trains the full model from text-to-video initialization and surpasses EVE.Compared with EVE’s adapter training and memory-intensive Factorized Diffusion Distillation, MovieGen avoids backpropagation through multiple generation passes and reports new state-of-the-art editing results.Source: §7.4 · pp. 66–67

7.5 Audio Generation

• Prior audio systems commonly use latent diffusion or token models trained on in-the-wild data, while symbolic music and small model scales limit cinematic quality, alignment, and generality.MovieGen adds quality and fine-grained music control, jointly generates non-diegetic music and sound effects, and uses diffusion to avoid tokenization information loss.Source: §7.5 · pp. 67–68

Contributions

• Movie Gen improves text-to-video generation, video personalization, video editing, and sound-effect and music generation by scaling data, training compute, and model size together.The recipe emphasizes large-scale high-quality pre-training data and smaller, higher-quality finetuning data across image, video, and audio generation.Source: §8 · p. 68
• The video-editing approach combines multi-task image-editing and video-generation training with synthetic multi-frame editing and backtranslation fine-tuning, without supervised video-editing data.The two short fine-tuning stages use synthetic multi-frame editing data followed by video editing via backtranslation.Source: §8 · p. 68

Limitations

• Video generation still produces artifacts around complex geometry, object manipulation, physics, and state transformations, while audio can desynchronize for dense, small, occluded, or finely interpreted motions.The models also do not support voice generation because of design choices.Source: §8 · p. 68

Benchmarking and future work

• Reliable media-generation benchmarking requires model, data, inference, non-cherry-picked generations, and prompt sets, although human evaluations remain vulnerable to personal biases and other factors.The conclusion also identifies jointly generating video and audio as an important research direction because the modalities are currently trained separately.Source: §8 · p. 68

Safety considerations

• Deployment requires addressing learned associations and modality-specific biases, noting that the study uses only English text inputs and plans safety models to reject policy-violating prompts or generations.The models are intended for research and need multiple improvements before deployment.Source: §8 · pp. 68–69

Contributors and Acknowledgements

• Movie Gen lists core contributors who worked on the project for at least two-thirds of its runtime and orders them alphabetically by first name.Contributors worked on Movie Gen for at least 2/3 of the project runtime.Source: Contributors and Acknowledgements · p. 69

Core Contributors

• The core-contributor roster includes Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, and many additional researchers and engineers.The passage provides the full alphabetical list of core contributors.Source: Core Contributors · p. 69

A.1 Text encoders

• Movie Gen uses text embeddings from three text encoders for visual-text generation.The appendix introduces details on these encoders and their use for visual-text generation.Source: §A.1 · p. 80
• The long-prompt MetaCLIP model processes prompts up to 256 text tokens by expanding its position embeddings and finetuning on synthetic and human-annotated captions.The model is CLIP-style and uses synthetic image captions to finetune MetaCLIP; the position embedding was expanded to 256 tokens.Source: §A.1 · p. 80
• Because finetuning all text-encoder parameters overfit and degraded encoding, Movie Gen freezes both encoders and updates only the text position embedding and biases.The authors attribute overfitting partly to the uniform style of synthetic long captions and mix long synthetic, short synthetic, and human-annotated captions.Source: §A.1 · p. 80
• Visual-text generation identifies quoted prompt text for rendering, encodes it with character-level ByT5, and trains on images where visual text covers 10-50% of the area.Prompt rewriting automatically identifies quoted text at inference time, while OCR detection models help curate suitable pre-training data.Source: §A.1 · p. 80

A.2 Model scaling and training efficiency

• Activation checkpointing reduced peak memory but increased computation, so its effectiveness depended on GPU FLOPs, HBM capacity, and interconnect bandwidth.The implementation was written in PyTorch, compiled into CUDAGraphs, and supported variable-sized inputs based on model and system configurations.Source: §A.2 · pp. 80–81
• Custom TP-, SP-, and CP-based model parallelism achieved strong activation-memory scaling, minimized exposed communication, and enabled near-strong scaling across multiple nodes.A custom forward/backward implementation precisely partitioned and overlapped computation and communication while supporting selective recomputation without performance loss.Source: §A.2 · pp. 80–81
• Multiple sharding plans with similar estimated latency and sufficient HBM capacity enabled different end-to-end components and stages to use different parallel strategies.The performance model estimated memory utilization and latency for backbone blocks, text encoders, and TAE execution.Source: §A.2 · p. 81
• Approximately neutral batch-size-to-step-time scaling allowed optimizer-step counts to increase while GPU count and total training-data size remained fixed.This mattered because TP, SP, and CP reduce effective global batch size and affect training-step wall latency; optimizer-step choices can influence final model quality.Source: §A.2 · p. 81
• For the 73K-token, 768px training stage, frozen ByT5 and Long-Prompt MetaCLIP were replicated, while UL2 used FSDP across each TP-rank’s data-parallel group.The final plan treated text encoders differently according to parameter size, memory overhead, and execution latency.Source: §A.2 · p. 81
• High-resolution TAE latents were precomputed and cached because growing intermediate activations and nontrivial latency made efficient TAE parallelization impractical.The TAE had frozen parameters but prohibitively large intermediate activations for increasing input sizes.Source: §A.2 · p. 81
• The backbone used stage-specific sharding: FSDP-only, FSDP+TP, FSDP+TP+SP, and FSDP+TP+SP+CP for components including patchification, context embedding, cross-attention, and self-attention.Additional learned components and changing activation and gradient connections required sharding to vary across backbone stages.Source: §A.2 · p. 81

B Additional Data Details

• Less than 1% of high-resolution-set examples passed the data-curation thresholds.Table 39 reports the volume drop during high-resolution set curation and notes the resulting acceptance rate.Source: Table 39 · p. 82

B.1 Video Data Curation Thresholds

• OCR filtering retained only videos whose word-detection score multiplied by word-recognition score stayed below 0.6 across all sampled frames.The internal OCR model adaptively samples frames, detects words, and recognizes the detected text.Source: §B.1 · p. 82
• Border detection removed videos with borders that could induce black borders in generated outputs, especially for portrait-mode videos.The detector identifies large vertical and horizontal pixel deltas and uses scanning lines to locate borders.Source: §B.1 · p. 82
• Scene-aware clip sampling avoided abrupt scene changes by selecting 1–2 scenes longer than 16 seconds, then extracting one 4–16-second clip per scene.More than 50% of training clips were 15–16 seconds long.Source: §B.1 · p. 82
• Aesthetic filtering removed clips scoring below 4 with the public LAION aesthetics image model applied to each clip’s middle frame.Multi-frame aesthetic scoring did not significantly improve recall of poor-quality clips.Source: §B.1 · p. 82
• Jittery videos were removed when PySceneDetect’s shot-boundary detector found more than 0.85 shots per second, addressing false-positive shots from frequent camera movement.FFmpeg motion scores and vectors were inadequate for detecting this jittery motion pattern.Source: §B.1 · p. 82
• The strictest curation thresholds were analyzed for per-filter data-volume drops when constructing the high-resolution training set.The analysis is reported in Table 39.Source: §B.1 · p. 82

B.2 Camera Motion Control types

• Training captions encode 16 classifier-predicted camera motions, while supervised finetuning adds six camera-motion and position labels.The 16 types include zoom, push, pull, pan, truck, tilt, pedestal, arc, tracking, static, and handheld; the six added labels are wide angle, close-up, aerial, low angle, over the shoulder, and first person view.Source: §B.2 · p. 83

C.1 Annotation Variance Study on Text-to-Video Evaluation

• Annotation variance was estimated by repeating A/B tests four times on 381 prompts and computing each axis’s net-win-rate standard deviation.The estimates support statistical-significance assessment in Section 3.6.1.Source: §C.1 · p. 83, Table 40 · p. 83
• Overall quality has higher annotation variance than text-faithfulness, because it combines multiple subjective evaluation signals.Source: §C.1 · p. 83
• Frame consistency is more variable than other quality axes, while realness is less variable than aesthetics.Distortion judgments are harder than motion judgments, whereas generated-looking content is more objective to assess than universally pleasing aesthetics.Source: §C.1 · p. 83

C.2 T2V comparison to prior work

• Prior-work comparisons document video-generation details, collection dates, and size parameters to support fair evaluation despite changing black-box systems.The study assumes industry models are updated over time and records when videos were collected.Source: §C.2 · p. 83, Table 41 · p. 84
• For OpenAI Sora, 158 publicly released videos were compared with Movie Gen samples selected as the best of five random-seed generations per prompt.The procedure addresses the likelihood that publicly released closed-source samples were cherry-picked.Source: §C.2 · p. 83
• Movie Gen and Sora videos were adaptively matched in spatial resolution and duration, while retaining the original frame rate.Movie Gen videos are 1080p and 10.66s, whereas Sora releases vary in resolution and are mostly 1280×720 and 10s; Sora comparisons used 500 linear sampling steps.Source: §C.2 · pp. 83–84
• Non-cherry-picked Movie Gen videos were released through Movie Gen Video Bench to facilitate future comparisons.Source: §C.2 · p. 84

Correlation analysis

• Approximately 10,000 annotations across 53 evaluations per task assess whether objective metrics predict subjective system-level net win rates and item-level scores.The analysis examines both Video-Audio Alignment and Audio Quality tasks.Source: §C.3 · p. 84
• Pearson correlations reach at least 0.9 among “overall”, “professional” and “naturalness” audio-quality aspects, while Video-SFX correctness and synchronization correlate at 0.76.The analysis therefore focuses on “Overall” audio quality and “Correctness” for Video-SFX alignment.Source: §C.3 · p. 84
• Text-Alignment precision and recall are mapped from 1–5 scores to percentages and combined into F1, but system-level correlations are omitted because few systems were evaluated.The mappings are 20%, 40%, 60%, 80% and 100% for both precision and recall.Source: §C.3 · p. 84

C.3.1 System-level correlations

• Objective metrics are predictive of subjective measures, but model-specific bias means equal objective-score differences can yield different or opposing net win rates.Thus, relying solely on objective-metric differences for superiority claims is risky.Source: §C.3.1 · p. 84
• Movie Gen Audio shows larger audio-quality net win rates than external baselines at the same objective-score difference, suggesting perceived quality beyond the audio quality score.This pattern appears in non-ablation pairwise comparisons.Source: §C.3.1 · p. 84
• Audio quality score correctly predicted overall audio quality preferences in 21 of 24 significant ablation comparisons (87.5%; 95% CI: 71.7–96%).Among 33 ablation model pairs, only statistically significant objective-score differences were considered.Source: §C.3.1 · p. 84
• Audio quality score tends to better predict audio quality preference than ImageBind, while both are comparably predictive of Video-SFX Alignment.Precision estimates are uncertain because of the limited sample, and both metrics are more predictive when comparing Movie Gen Audio with an external model.Source: Figure 37 · p. 86, §C.3.1 · p. 86
• ImageBind correctly predicted overall audio quality preferences in 82.3% of the 17 of 33 comparisons with statistically significant ImageBind differences (95% CI: 63.6%–94.5%).ImageBind differences were statistically significant for fewer model pairs than audio quality score differences.Source: §C.3.1 · p. 86

C.3.2 Item-level correlations

• The analysis estimates subjective-preference log-odds by fitting Bradley-Terry likelihood regressions separately on one objective metric per visualization.Regression coefficients are modeled by item group and objective-metric bins.Source: §C.3.2 · p. 86
• Audio quality and ImageBind scores predict overall audio quality preferences, with the best observed ImageBind score yielding approximately 4–4.5 to 1 odds over the lowest.The odds comparison is based on fitted Bradley-Terry latent-quality differences.Source: §C.3.2 · pp. 86, 88
• Non-diegetic music reduces the predictive value of ImageBind for overall audio-quality preferences.The analysis conditions the regression on whether non-diegetic music is present.Source: §C.3.2 · pp. 86, 88
• Audio-quality scores below 4 slightly reduce odds of stronger Video-SFX alignment, while improvements beyond approximately 4–5 have little further effect.The reported odds range is approximately 1.5 to 1.Source: §C.3.2 · p. 88
• Overall audio-quality preferences saturate above approximately 6.5 without non-diegetic music, but continue increasing beyond 6.5 when it is present.The passage notes that this pattern is also reflected in direct tests in Table 33.Source: §C.3.2 · p. 88
• ImageBind improvements have a stronger relative effect on overall audio quality than on Video-SFX alignment, while Video-SFX correctness increases monotonically without saturation.These relationships are inferred from regression coefficients relating subjective preferences to objective-score bins.Source: Figure 38 · p. 87, §C.3.2 · p. 88
• ImageBind’s correlation with diegetic synchronization is likely spurious because ImageBind is insensitive to audio shifts, while correctness-improving model changes may also improve synchronization.The passage explicitly cautions that the observed relationship is not likely causal.Source: §C.3.2 · p. 88
• CLAP correlates moderately with consensus Text-Audio alignment evaluations: Spearman r is approximately 0.3 for real and 0.35 for generated single-scene videos.The difference is nominally higher for generated videos but is not statistically significant; confidence intervals use bootstrap resampling of items and annotations.Source: §C.3.2 · p. 88, Figure 39 · p. 88

D.1.1 Additional Metrics for Main Results on Sound Effect Generation

• Movie Gen Audio outperforms all baselines supporting text input on text-audio recall, leads most precision comparisons, and is comparable to ElevenLabs on real videos.Recall is the primary metric for text-audio alignment; CLAP rankings are informative mainly when score differences are large.Source: §D.1.1 · p. 89, Table 42 · p. 89
• Movie Gen Audio leads subjective audio-quality comparisons and objective metrics, with PikaLabs and ElevenLabs showing the smallest subjective gaps.AQual correlates reasonably with subjective pairwise judgments.Source: §D.1.1 · p. 89, Table 42 · p. 89
• Movie Gen Audio achieves the highest video-SFX alignment IB score among baselines and also wins subjective evaluations, although IB rankings poorly reflect subjective baseline rankings.Seeing&Hearing, V2A, and TV2A can show IB scores comparable to Movie Gen Audio despite worse subjective video-SFX alignment; Seeing&Hearing directly optimizes IB.Source: §D.1.1 · p. 89
• On the Movie Gen Audio Bench, Movie Gen Audio leads AQual and IB except against Seeing&Hearing, while remaining comparable to baselines on CLAP.These objective results reproduce the trends reported in the other sound-effect evaluations.Source: Table 43 · p. 90, §D.1.1 · p. 90

D.1.2 Control audio quality through text prompts

• Text prompts can control generated audio quality: low-quality prompts produce natural but intentionally degraded audio, including wind noise or broken bass in music.The examples cover both sound-effect generation and joint music-and-sound-effect generation, using a quality score of 5.0.Source: §D.1.2 · p. 90

D.1.3 Control music style through text prompts

• Varying the music caption for the same video provides an example of controlling music style through text prompts.The capability is illustrated in Figure 42.Source: §D.1.3 · p. 90

Additional samples

• Figures 43 and 44 provide additional samples of sound-effect generation and joint sound-effect plus music generation, respectively.These samples are presented as supplementary Movie Gen Audio outputs.Source: §D.1.4 · p. 90, Figure 43 · p. 93, Figure 44 · p. 94

Prompt-controlled audio quality

• Increasing the prompted audio quality score produces cleaner spectrograms with fewer high-frequency noises.Higher quality scores also tend to produce audio without wind noises compared with lower scores.Source: Figure 40 · p. 91, Figure 41 · p. 92

Music description conditioning

• Figure 42 shows generated audio conditioned on different music descriptions in the text prompt.The figure illustrates variation in Movie Gen Audio outputs based on the supplied music description.Source: Figure 42 · p. 92

• Figures 45 and 46 provide further example video generations from Movie Gen Video.The generated videos are linked in the respective figure captions.Source: §D.2 · p. 95, Figure 45 · p. 95, Figure 46 · p. 96
• Figure 45 shows a video prompted by a person harvesting clouds from a field and placing them in a basket.Source: Figure 45 · p. 95, §D.2 · p. 95
• Figure 46 shows a video prompted by a dense jungle pathway illuminated by oversized, bioluminescent mushrooms.Source: Figure 46 · p. 96, §D.2 · p. 96