PaperlayerView exact revision on arXiv

Source-linked AI summary

Movie Gen: A Cast of Media Foundation Models

Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Rambhatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng, Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, Yuming Du

arXiv:2410.13720v2cs.CVcs.AIcs.LGeess.IV

TL;DR

Movie Gen addresses the challenge of equipping AI systems with generative, compositional, and predictive capabilities for media outputs. It presents a cast of foundation models trained by scaling data, model size, and compute, achieving improvements across video and audio generation, personalization, and editing. The models nevertheless retain limitations involving complex visual phenomena, audio synchronization, and voice generation.

  • Problem

    Equipping AI systems to compose media while predicting realistic motion, geometry, physics, and audio remains a core scientific challenge.

  • Method

    Movie Gen combines large foundation models for image, video, and audio with curated pre-training and fine-tuning, including post-training for personalization and editing.

  • Results

    Movie Gen outperforms prior work and commercial systems across text-to-video, video personalization, video editing, sound-effect generation, and music generation.

  • Takeaways & Limitations

    Scaling data, training compute, and model size, together with high-quality data curation and targeted fine-tuning, improves image, video, and audio generation quality.

  • Takeaways & Limitations

    Video generation still exhibits issues with complex geometry, object manipulation, physics, and state transformations, while audio can lose synchronization during dense or visually subtle motions.

Abstract

from arXiv · show

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models. All videos from this paper are available at https://go.fb.me/MovieGenResearchVideos.

1 Introduction

Movie Gen is a cast of media foundation models focused on generating and manipulating image, video, and audio outputs. Scaling data, compute, and model size supports broad capabilities, including high-quality video, personalization, editing, and synchronized audio.

  • Motivation: Movie Gen targets media generation as a core challenge involving composition and prediction across images, video, audio, motion, physics, geometry, and sound.The paper focuses on media as the output space rather than text alone.
  • Foundation models: 30B parameters support Movie Gen Video, while 13B parameters support Movie Gen Audio across foundation generation tasks.Both models are trained on internet-scale media data using Transformer-based models and Flow Matching.
  • Capabilities: Movie Gen provides text-to-video generation, personalized videos conditioned on a person’s face, precise video editing, and synchronized audio.The capabilities include realistic personalized HD videos up to 16 seconds at 16 FPS and 48kHz audio.
  • Results: The models are reported as state-of-the-art across text-to-video, video personalization, video editing, and audio-generation tasks.The reported comparisons include prior work and commercial systems such as Runway Gen3, LumaLabs, OpenAI Sora, PikaLabs, and ElevenLabs.
  • Resources: The paper releases Movie Gen Video Bench and Movie Gen Audio Bench to support future benchmarking.It also provides details on architectures, training, inference, and experimental settings.

2 Overview

Movie Gen combines large-scale video and audio foundation models with post-training procedures for personalization and precise editing. Its training recipe uses joint image-video learning, curated finetuning, and task-specific post-training.

  • Foundation models: Movie Gen uses two foundation models: Movie Gen Video for image and video generation, and Movie Gen Audio for video-to-audio and text-to-audio generation.The models support variable resolutions, durations, aspect ratios, and audio lengths.
  • Movie Gen Video: Movie Gen Video is a 30B-parameter model pretrained jointly on O(100)M videos and O(1)B images.Joint image-video modeling treats images as single-frame videos and supports broader visual learning.
  • Movie Gen Audio: Movie Gen Audio is a 13B-parameter model pretrained on O(1)M hours of audio for synchronized cinematic sound effects and music.It generates diegetic sounds and non-diegetic music aligned with visual scenes and actions.
  • Personalization: Personalization conditions video generation on text and a person’s image while maintaining the person’s identity and following the prompt.Training uses automatically constructed image-text inputs paired with human-containing video outputs.
  • Precise Editing: Precise Editing supports textual edits to real and generated videos without supervised video-editing data.The approach addresses the difficulty of obtaining large-scale supervised editing data.
  • Training recipe: Movie Gen Video is trained through text-to-image pretraining, joint text-to-image and text-to-video pretraining at increasing resolutions, and finetuning on curated videos.Additional post-training adds personalization and video-to-video editing capabilities.

3 Joint Image and Video Generation

Movie Gen combines a compressed latent-space generator, scaling-oriented Transformer design, and human-centered evaluation to support broad image and video generation capabilities. The paper reports strong quality, alignment, and motion results against prior systems while identifying artifacts and metric reliability as key technical concerns.

  • Training objective: Latent dots in the TAE can decode into spot artifacts, motivating an outlier penalty loss that removes them from generated and reconstructed videos.The loss is added to standard variational autoencoder losses, with r = 3 and loss weight 1e5.
  • Backbone and scaling: Movie Gen uses a Transformer architecture designed to scale to long video contexts, with model-parallel techniques applied to its 30B-parameter backbone.The paper notes that its architecture is more stable across model size, learning rate, and batch size than specialized prior blocks.
  • Evaluation: Automated metrics such as FVD and IS do not reliably correlate with human evaluations, so the paper evaluates generated videos with pairwise human A/B tests.The evaluation protocol asks raters to select a winner or declare a tie for each measured axis.
  • Results: Movie Gen Video strongly outperforms Runway Gen3 and LumaLabs overall, moderately beats OpenAI Sora, and performs on par with Kling1.5.Reported breakdowns include 19.27% net win on motion naturalness and 33.1% on frame consistency against Runway Gen3, plus 11.62% realness net win against Sora.
  • Results: Qualitative results show realistic, high-quality videos with natural motion, consistent objects and identities, and behavior that obeys physical laws across natural and fantastical prompts.The paper also reports complex motion maintained over the video duration and improved text faithfulness relative to several prior systems.

4 Video Personalization

Personalized Movie Gen Video extends the 30B Movie Gen Video model with reference-image conditioning to generate videos that preserve a person’s identity while following text prompts. Its training data, staged recipe, and design choices improve the balance among identity preservation, video quality, text alignment, expressions, and natural motion.

  • 4.1 Architecture: Personalized Movie Gen Video conditions the 30B Movie Gen Video model on identity information from a reference image alongside the text prompt.The architecture uses vision-token concatenation, with identity features extracted by a trainable Long-prompt MetaCLIP encoder and aligned to the text feature dimension.
  • 4.2 Data: The training set focuses on videos containing the same person across all frames and produces O(10)M paired samples from five uniformly sampled frames per clip.Faces are cropped and segmented to reduce attention to non-critical background regions.
  • 4.2 Data: Cross-paired data prevents a copy-paste shortcut in which generated videos inherit the reference face’s expression or head pose.Cross-pairs use a reference image from a different video of the same person; the dataset includes O(10)K real and O(1)M synthetic cross-paired samples.
  • 4.2 Training: The three-stage PT2V pre-training recipe targets identity preservation, long personalized videos, and more natural human expressions and motion.Direct long-video training is inefficient because training speed is nearly proportional to the square of the number of latent frames or tokens, while identity injection can be slow.
  • 4.5 Results: Personalized Movie Gen Video significantly outperforms ID-Animator across identity preservation, video quality, and text alignment, while qualitative comparisons show stronger identity consistency and video quality.The model generates videos that follow identities across diverse motion and camera views and compares against Movie Gen Video without visual conditioning on video quality and text alignment.
  • 4.5.1 Ablations: −16% identity preservation occurs with a fixed vision encoder, while cross-paired training improves text alignment by 27.36%, overall quality by 13.68%, and motion naturalness by 26.14%.Cross-paired training decreases identity metrics but improves facial expressions and natural movement; high-quality finetuning similarly trades a slight identity drop for improved video quality and naturalness.

5 Instruction-Guided Precise Video Editing

Movie Gen Edit addresses the scarcity of supervised video-editing data with a multi-stage training approach for precise, instruction-guided edits. It achieves strong human-evaluation results against prior methods while supporting broader video inputs and a dedicated benchmark.

  • 5 Instruction-Guided Precise Video Editing: Movie Gen Edit targets accessible, precise text-guided editing of real and generated videos, addressing the scarcity of supervised video-editing data.The method is designed to process entire video inputs rather than limited proxy features.
  • 5.2 Evaluation: Movie Gen Edit Bench evaluates six video-editing tasks using videos with varied resolutions, frame rates, lengths, and aspect ratios.The benchmark is intended to reflect the broader inputs used by contemporary video-generation systems.
  • 5.1 Model: Training progresses through single-frame editing, multi-frame editing, and video editing via backtranslation to reduce train-test discrepancies.These stages progressively expose the model to editing inputs and improve the realism of generated edits.
  • 5.1.1 Model Architecture and Initialization: The text-to-video backbone is adapted for editing by adding input-video channels that concatenate the latent input video with the noisy output latent video.The architecture also incorporates conditioning on specific editing information.
  • 5.1.3 Stage II: Multi-frame Video Editing: Stage II mitigates blurry edited-video artifacts by training with multi-frame video inputs, though generated elements can still lack motion or appear oversaturated.Stage I did not condition the model on multi-frame video inputs, creating a major train-test discrepancy.
  • 5.1.4 Stage III: Video Editing via Backtranslation: Backtranslation trains the model to denoise the clean input video while conditioning on an edited generated video, avoiding direct imitation of artifact-prone outputs.The approach adapts backtranslation to create training instructions for video editing.
  • 5.3.1 Comparisons to Prior Work: Human raters preferred Movie Gen Edit over EVE more than 74% of the time on TGVE+, while automated results were state-of-the-art on ViCLIPdir and comparable to EVE on ViCLIPout.On Movie Gen Edit Bench, the method was also preferred over Runway Gen3 V2V settings and SDEdit in human evaluation.
  • 5.3.2 Ablations: Stage II was preferred over Stage I in more than 89% of human evaluations, and Stage III was preferred over Stage II in more than 60%.A separate ablation found full model training better aligned edits with instructions and preserved input images better than a ControlNet adapter.

6 Joint Sound Effect and Music Generation

Movie Gen Audio supports synchronized joint sound-effect and music generation, variable-length extension, and text-based control. It outperforms prior baselines across audio quality and video-audio alignment, while ablations show benefits from high-quality data, captions, and conditioning.

  • Capabilities: Movie Gen Audio generates synchronized sound effects and music, supports audio extension for arbitrary lengths, and accepts text prompts for acoustic-event and music-style control.A single model handles audio generation and extension by changing conditioned inputs; text prompts control what sounds to add and the desired music style.
  • Benchmark: 527 videos spanning 36 audio categories and 434 concepts form Movie Gen Audio Bench, a synthetic benchmark for video-to-audio generation.The benchmark covers ambient environments and sound effects, with generated videos reviewed to exclude severe artifacts.
  • Results: 33.8% to 72.8% on synchronization, 27.5% to 82.2% on correctness, and 31.3% to 91.0% on overall quality are reported for Movie Gen Audio across benchmark comparisons.The paper reports that Movie Gen Audio outperforms all baselines by a large margin across these metrics.
  • Results: Movie Gen Audio outperforms baselines across audio quality and video alignment in multi-shot evaluation, with a larger margin over joint-generation S&H TV2A than in sound-effect-only evaluation.Separately generating sound effects and music improves S&H over joint generation but remains behind Movie Gen Audio.
  • Analysis: Separate text-to-music generation falls short especially on alignment because it cannot model sound-effect/music correlation or respond to scene and mood changes.The external API is unaware of the video, limiting its ability to capture visual changes and coordinate music with prominent sound effects.
  • Ablations: Higher conditioned audio quality consistently improves predicted audio quality, while significant video-music alignment gains occur for SFX+music generation from quality 5.0 to 6.For SFX generation, alignment changes are not significant; fine-tuning and high-quality audio data improve quality and, in some settings, alignment.

7 Related work

Prior work develops diffusion-based and latent-space approaches for image and video generation, but Movie Gen extends foundation-model capabilities across generation, personalization, editing, and cinematic audio.

  • Image and video generation: Diffusion models and latent-space autoencoders became central approaches for efficient text-to-image and video generation.Prior systems commonly use variational autoencoders or related encoders to compress media before generation.
  • Movie Gen approach: Movie Gen combines a 16-channel variational autoencoder with flow transformers and prompt rewriting for visual quality and text alignment.
  • Image and video generation: Transformer architectures have demonstrated scalability and strong performance for diffusion-based video generation alongside earlier U-Net-based designs.
  • Video editing: Text-based video editing requires changing instructed elements while preserving the rest, making high-quality training data difficult to collect.
  • Video editing: Movie Gen trains a text-to-video model fully rather than adding adapters, avoiding the substantially higher memory demands of EVE’s supervision procedure.The cited comparison states that EVE’s FDD is an order of magnitude more memory demanding.
  • Audio generation: In-the-wild audio datasets often lack professional quality, post-production, emphasized sound events, and non-diegetic cinematic sounds.These limitations constrain soundtrack generation that combines music and sound effects.
  • Audio generation: Movie Gen adds quality control and fine-grained music control to text control, improving soundtrack design flexibility for mixed-quality data.
  • Audio generation: End-to-end video-to-music generation avoids MIDI’s transcription limits, annotation requirements, and inability to model relationships with speech and sound effects.

8 Conclusion

Movie Gen improves generation, personalization, editing, and audio capabilities through scaled training and curated data, while the authors identify persistent quality, synchronization, benchmarking, and safety limitations.

  • Conclusion: Scaling data, training compute, and model size supports improvements across text-to-video, personalization, editing, sound effects, and music generation.
  • Conclusion: Movie Gen’s video editing approach uses image editing and video generation multitask training followed by synthetic and backtranslation fine-tuning.
  • Limitations: Generated and edited videos still show artifacts around complex geometry, object manipulation, physics, and state transformations.
  • Limitations: Audio can lose synchronization for dense, small, occluded, or fine-grained visual events, and the system does not support voice generation.
  • Limitations: Reliable comparisons remain difficult without transparent models, data, inference details, non-cherry-picked generations, and prompt sets.
  • Safety: The models were developed for research and require multiple improvements before deployment, including attention to learned associations and modality-specific biases.

Contributors and Acknowledgements

The paper acknowledges extensive contributions from Meta researchers and documents selected contributor categories alongside supporting methodological material.

  • Contributors and acknowledgements: Meta lists core contributors who worked on Movie Gen for at least two-thirds of the project runtime and contributors who worked for at least one-third.
  • Contributors and acknowledgements: The paper provides an alphabetical list of core contributors and contributors by first name.
  • Supporting methods: The paper describes text embeddings from three text encoders for visual-text generation.
  • Supporting methods: A CLIP-style encoder is trained for prompts up to 256 text tokens using synthetic captions and expanded position embeddings.
  • Supporting methods: Visual text generation identifies quoted text and uses prompt rewriting to locate such text at inference time.

A.2 Model scaling and training efficiency

Movie Gen Video training uses memory-aware parallelism and sharding for a 73K-token context, combining analytical performance modeling with custom execution strategies and activation checkpointing trade-offs.

  • Memory and efficiency: 73K-token Movie Gen Video contexts make memory capacity a central training constraint, and activation checkpointing trades extra FLOPs for lower peak memory.
  • Parallelism: Analytical modeling identifies duplicated activations and interdependent compute and communication costs across TP, SP, CP, and FSDP.
  • Parallelism: The parallelism implementation targets strong activation-memory scaling and reduced exposed communication across multi-node model-parallel widths.
  • Implementation: The implementation is written in PyTorch, compiles into CUDAGraphs, and supports variable-sized inputs at job start.
  • Sharding: Sharding-plan tools estimate end-to-end memory and latency, generating candidate plans that fit available GPU high-bandwidth memory.
  • Training efficiency: The training system empirically identifies parallelism configurations with approximately neutral batch-size-to-step-time scaling.
  • Sharding: The final 768px training stage processes 73K tokens per sample with components assigned different combinations of FSDP, TP, SP, and CP.
  • Data curation: High-resolution data curation accepts less than 1% of data under the strict thresholds in Table 39.

B.1 Video Data Curation Thresholds

Movie Gen curates video training data with filters targeting readable content, clean borders, coherent scenes, high aesthetics, and stable motion. It also annotates camera motion categories and tracks data loss across filtering stages.

  • Content filtering: Videos are retained only when OCR detection-recognition scores remain below 0.6 across all sampled frames.The OCR model samples frames adaptively, detects words, and recognizes detected text.
  • Content filtering: Border detection removes videos whose borders could induce black borders in generated outputs, especially for portrait videos.The detector uses large vertical and horizontal pixel deltas followed by scanning-line border detection.
  • Temporal filtering: Scene-aware sampling extracts 1–2 scenes longer than 16 seconds instead of randomly cutting clips, reducing abrupt scene changes.Raw videos average 28 seconds and are converted into the model’s required 4–16 second clips.
  • Quality filtering: Clips with aesthetic scores below 4 are removed to retain high-quality training examples.The public LAION aesthetics model is applied to the middle frame, and multi-frame scoring did not significantly improve recall.
  • Motion filtering: Jittery videos are identified using the number of detected shots per second because standard motion measures can miss frequent camera movement.The curation process addresses false-positive shot segmentation from PySceneDetect on jittery videos.
  • Camera motion and curation analysis: A camera-motion classifier supplies 16 motion types as caption prefixes, while strict thresholds quantify data-volume loss during high-resolution curation.The listed controls include zoom, push, pull, pan, truck, and tilt movements.

C.1 Annotation Variance Study on Text-to-Video Evaluation

The study measures annotation reliability and documents evaluation procedures for comparisons with prior text-to-video systems. Repeated A/B tests reveal that variance differs across evaluation axes, while released non-cherry-picked outputs support future comparisons.

  • Annotation variance: 381 prompts were reannotated four times to estimate standard deviations of net win rates for text-faithfulness, quality, and realness-and-aesthetics evaluations.The repeated subset is used to assess annotation variance and significance of winning or losing outcomes.
  • Annotation variance: Overall quality has higher annotation variance than text-faithfulness because it combines more subjective evaluation signals.Frame consistency is the most variable quality axis, while realness is less variable than aesthetics.
  • Comparison protocol: Prior-work comparisons standardize generation specifications and post-processing to improve fairness and reduce annotator bias.The comparison set includes external systems whose released videos differ in resolution and duration.
  • Comparison protocol: OpenAI Sora comparisons use 158 publicly released videos, whose samples may reflect unknown cherry-picking.Movie Gen Video samples are selected and temporally adjusted to match corresponding Sora videos for comparison.
  • Comparison protocol: Non-cherry-picked Movie Gen Video outputs are released through Movie Gen Video Bench to enable future comparisons.The release is intended to facilitate direct comparison with subsequent work.

C.3 Correlations between audio-based objective and subjective metrics

The analysis examines whether objective audio metrics track human preferences at system and item levels using thousands of annotations. It focuses on representative audio-quality and video-SFX alignment dimensions whose related aspects are strongly correlated.

  • Study design: Around 10,000 annotations across 53 evaluations assess how objective metrics predict system-level net win rates and relate to item-level subjective scores.The analysis covers Video-Audio Alignment and Audio Quality tasks.
  • Study design: The analysis focuses on Overall audio quality and Correctness in Video-SFX alignment as representative task aspects.These aspects are selected for concise analysis of the two tasks.
  • Aspect relationships: Overall, professional, and naturalness audio-quality ratings have Pearson correlations of 0.9 or higher.The high correlations support using Overall audio quality as the focused aspect.
  • Aspect relationships: Correctness and synchronization in Video-SFX alignment have an observed correlation of 0.76.The analysis therefore focuses on Correctness for this task.
  • Text alignment: Text-Alignment combines annotated precision and recall into an F1 score, but system-level correlations are omitted because few systems were evaluated.The 1–5 ratings are mapped to 20%, 40%, 60%, 80%, and 100% for both precision and recall.

C.3.1 System-level correlations

Objective metrics predict subjective preferences at the system level, but model-specific bias limits claims based solely on objective-score differences. Audio quality score and ImageBind are useful predictors, with predictive behavior varying across comparison types and tasks.

  • System-level findings: Objective metrics are predictive of subjective measures, but model-specific bias makes objective-score differences alone risky for superiority claims.Model pairs with the same mean objective difference can have different or opposing net win rates.
  • System-level findings: 21 of 24 statistically significant audio-quality-score comparisons correctly predict overall audio-quality preferences, or 87.5% with a 95% CI of 71.7–96%.The analysis considers ablations alone and includes 33 model-pair evaluations overall.
  • System-level findings: 82.3% of statistically significant ImageBind-score comparisons correctly predict overall audio-quality preferences, with a 95% CI of 63.6%–94.5%.Only 17 of 33 model pairs had statistically significant ImageBind-score differences.
  • Metric comparison: Audio-quality score tends to better predict the examined subjective dimensions, while ImageBind and audio-quality score are comparably predictive for Video-SFX alignment.Precision estimates are uncertain because of the limited sample.
  • Metric comparison: Both metrics are more predictive for comparisons between Movie Gen Audio and external models than for the evaluated ablation sample.Figure 36 relates mean item-level objective differences to subjective net win rates, and Figure 37 reports system-level precision with bootstrap confidence intervals.

C.3.2 Item-level correlations

Item-level analyses compare objective metrics with subjective preferences for audio quality, Video-SFX correctness, and Text-Audio alignment. Correlations and preference effects vary by metric and by whether non-diegetic music is present.

  • Both audio quality and ImageBind scores predict overall audio-quality preferences.
  • Text-Audio alignment has moderate item-level correlation with CLAP: Spearman r is ∼0.3 for real and ∼0.35 for generated single-scene videos.Diegetic synchronization correlates with ImageBind, but the passage states this relationship is likely non-causal because ImageBind is insensitive to audio shifts.
  • ∼4–4.5 to 1 odds favor the best observed ImageBind score over the lowest for overall audio quality.
  • Audio quality scores above ∼4–5 add little to Video-SFX correctness, while gains in overall audio quality saturate above ∼6.5 without non-diegetic music.Items with non-diegetic music continue gaining audio-quality preference beyond ∼6.5.
  • ImageBind scores monotonically increase Video-SFX correctness without saturation, but ImageBind becomes less predictive of overall audio-quality preferences when non-diegetic music is present.

D.1.1 Additional Metrics for Main Results on Sound Effect Generation

Additional sound-effect evaluations compare Movie Gen Audio with baselines using subjective and objective measures of text-audio, audio-quality, and video-SFX alignment. Movie Gen Audio generally leads, while objective alignment metrics can disagree with subjective judgments.

  • Movie Gen Audio outperforms all text-input baselines on text-audio recall and most baselines on precision.It is on par with ElevenLabs on precision for real videos.
  • Movie Gen Audio outperforms all baselines in subjective audio-quality tests and has the highest objective score.PikaLabs and ElevenLabs have the smallest subjective gaps and follow Movie Gen Audio on objective metrics.
  • Movie Gen Audio leads in video-SFX alignment on subjective evaluations, although objective-to-subjective correlation is much weaker.Seeing&Hearing, V2A, and TV2A can achieve ImageBind scores comparable to Movie Gen Audio despite worse subjective alignment.
  • On Movie Gen Audio Bench, Movie Gen Audio leads in AQual and ImageBind except against Seeing&Hearing, and is on par with baselines on CLAP.The passage states that similar trends appear in the additional sound-effect evaluation tables.
  • Text prompts can control generated audio quality, with low-quality prompts producing natural audio containing artifacts such as wind noise or broken bass.Figure 42 varies music captions for the same video.

D.1.4 Additional Audio Samples

The additional audio samples demonstrate controllable music descriptions, audio-quality variation, sound-effect generation, and joint sound-effect-plus-music generation.

  • Higher prompted audio-quality scores tend to produce audio with less wind noise.
  • Increasing prompted quality scores produces cleaner spectrograms by reducing high-frequency noise.
  • Additional samples vary music descriptions for the same video and include sound effects and joint sound-effect-plus-music generation.
  • The appendix provides additional sound-effect and sound-effect-plus-music samples.

D.2 Additional Results from Movie Gen Video

Additional Movie Gen Video results provide example generations across imaginative scenes, including cloud harvesting and bioluminescent jungle settings.

  • Figures 45 and 46 contain further example generations from Movie Gen Video.
  • One example prompt depicts a person harvesting clouds from a field and placing them in a basket.
  • Another example prompt depicts a dense jungle pathway illuminated by oversized bioluminescent mushrooms.
Loading 2410.13720v2…