Source-linked AI summary
DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
TL;DR
DreamX-Creator 1.0 targets the limited reciprocal modeling available when audio and video are omitted or generated separately. It introduces a compact native joint system with gated cross-modal generation, specialized data and training pipelines, and 2K refinement; among listed systems with disclosed backbone sizes, it is the smallest combining downloadable weights, native joint generation, and an officially supported 2K-or-higher path.
Problem
Existing video generators often omit audio or synthesize it separately, limiting reciprocal modeling of visual dynamics and acoustic events.
Method
DreamX-Creator 1.0 jointly denoises modality-specialized audio and video streams with Gated Cross-Modal Attention, supported by unified data construction, progressive joint training, multimodal reinforcement learning, and 1-step 2K refinement.
Results
Among listed systems with disclosed total backbone size, DreamX-Creator 1.0 is the smallest model combining downloadable weights, native joint audio-video generation, and an officially supported output path at 2K resolution or above.
Takeaways & Limitations
The publicly released 7B generator and 2K Refiner support reproducible evaluation and downstream adaptation of native audio-video generation.
Takeaways & Limitations
The 7B model remains behind larger open-weight baselines on several metrics, particularly audio aesthetics, cross-modal semantics, and lip synchronization.
Abstract
from arXiv · showhide
Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.
1 Introduction
DreamX-Creator 1.0 addresses the limited reciprocal interaction of separate audio-video generation by combining a compact native joint generator with cross-modal training, reinforcement learning, and 2K refinement. The system is released with downloadable components and is the smallest listed model combining downloadable weights, native joint generation, and an officially supported 2K-or-higher output path.
- Motivation: Separate audio-video pipelines limit reciprocal modeling of jointly determined visual dynamics and acoustic events.The paper highlights speech and mouth motion, visible impacts and sounds, and scene-level ambience, music, and camera-motion relationships.
- Accessibility: Among listed systems with disclosed total backbone size, DreamX-Creator 1.0 is the smallest combining downloadable weights, native joint audio-video generation, and an officially supported output path at 2K resolution or above.The comparison uses these three system-level properties to assess research accessibility and deployment scope.
- Native Joint Generation: The native joint generator independently processes modality-specialized audio and video streams before coupling them through bidirectional Gated Cross-Modal Attention.Token- and head-wise sigmoid gates modulate active attention-head outputs, while temporal coordinates align cross-modal queries and keys.
- System Design: The system combines a unified data system, Progressive Joint Training, Audio-Video Reinforcement Learning, and Autoregressive 1-Step 2K Refinement.The refiner distills a bidirectional multi-step teacher into a 1-step student requiring one denoising evaluation per temporal chunk.
- Accessibility: The authors publicly release the 7B native joint generator and 2K Refiner to support reproducible evaluation and downstream adaptation.The contribution statement identifies the generator as the smallest open-weight native joint model by disclosed total backbone size in Table 1.
2 Audio-Video Data System
The Audio-Video Data System converts heterogeneous raw videos into filtered, temporally aligned clips with structured multimodal annotations and capability-oriented organization. This organization concentrates relevant cross-modal supervision while preserving diversity, with speech and event sounds forming the largest reported content categories.
- Data System: The data system transforms heterogeneous raw videos into training-ready clips through staged quality and alignment filtering, structured annotation, and capability-oriented organization.Its design responds to noisy, weakly aligned, and imbalanced audio-visual content.
- Data Filtering: Filtering assesses visual quality, motion, audio quality, and audio-visual synchronization, with additional speech-specific lip-audio checks.The pipeline uses Q-Align, optical flow, Audiobox Aesthetics, Synchformer, and SyncNet for these assessments.
- Multimodal Annotation: Structured annotations jointly capture visual content, acoustic events, cross-modal relationships, spoken text, and their temporal ordering.Joint multimodal analysis and caption fusion are used instead of independently captioning modalities and merging them afterward.
- Capability Taxonomy: Clips are assigned to four capability-oriented pools according to dominant cross-modal supervision patterns inferred from their annotations.The organization targets supervision needed by different generation objectives rather than uniformly sampling the heterogeneous corpus.
- Capability Taxonomy: 45.0% of the training data is speech and 33.4% is event sounds, while the remaining samples cover music, natural sounds, and mixed content.The distribution provides supervision beyond speech-centric scenarios for general audio-visual generation.
3 Native Joint Audio-Video Generation
DreamX-Creator jointly denoises modality-specialized audio and video streams, keeping them independent early and coupling them later through gated bidirectional cross-modal attention. Its training mixes directional and joint conditioning modes across staged flow-matching optimization.
- Architecture Overview: Given a first frame and text prompt, the system jointly denoises audio and video latent streams with separate token rates, positional encodings, and transformer backbones.Text conditioning comes from a shared text encoder through modality-specific paths.
- Architecture Overview: The first network half processes streams independently, while the latter half adds parallel A2V and V2A cross-attention paths.A2V uses video queries with audio keys and values; V2A reverses those roles, and a direction mask selects one or both paths.
- Gated Cross-Modal Attention: Shared temporal coordinates and temporal rotary position encoding align cross-modal attention without resampling either latent sequence.The encoding is applied to cross-modal queries and keys while modality-specific positional encodings remain unchanged.
- Gated Cross-Modal Attention: Token- and head-wise sigmoid gates use each target hidden state and attention output to modulate active head outputs before projection and residual addition.The gate scales attention-head outputs, whereas the direction mask determines whether a complete directional residual is added.
- Progressive Joint Training: Training uses two audio-video pre-training stages followed by High-Quality Finetuning, mixing A2V, V2A, and Joint configurations throughout.Stage 1 trains latter-half LoRA adapters and cross-modal modules; Stage 2 jointly optimizes both backbones and cross-modal modules, followed by curated full-parameter finetuning.
- Optimization Details: Flow matching uses independently sampled Gaussian noise, with λv = λa = 0.5 during pre-training and λv = 0.5, λa = 0.1 during High-Quality Finetuning.The base timestep uses a discrete 1,000-timestep schedule with shift factor 5.0.
4 Audio-Video Reinforcement Learning
Audio-Visual Reinforcement Learning post-trains the joint generator with decomposed modality-specific and cross-modal feedback. This routing improves perceptual quality, semantic consistency, and temporal synchronization while preserving the base model’s capabilities and diversity.
- Motivation: Flow-matching training captures the joint audio-video distribution but does not directly optimize perceptual quality, prompt adherence, semantic consistency, or temporal correspondence.These human-perception criteria motivate the reinforcement-learning stage.
- Modality-Aware Feedback: Decomposed video, audio, and cross-modal advantages supervise their corresponding streams, while the shared term also supervises bidirectional interaction modules.This avoids allowing improvement in one modality to mask degradation in the other.
- Alignment Control: Synchronization-relevant video regions receive greater importance through cross-modal responses, while shallow-block gradient propagation is attenuated to protect pretrained modality-specific representations.Token weighting begins nearly uniformly and progressively focuses on regions contributing more strongly to synchronization.
- Modality-Aware Feedback: Video-, audio-, and synchronization-specific rewards are routed to the corresponding streams, with synchronization feedback jointly optimizing both streams and their cross-modal interactions.The design retains synchronization as a shared optimization objective.
- Training: Grouped candidate generation, multimodal reward evaluation, and policy optimization form an online loop that periodically refreshes the rollout model.The post-trained model retains the base model’s first-frame-conditioned joint audio-video interface.
5 Autoregressive 1-Step 2K Refinement
The 2K Refiner converts a coherent low-resolution joint-generation output into high-resolution video through autoregressive temporal-chunk refinement. A bidirectional multi-step teacher is adapted and distilled into a one-step student, reducing 2K inference cost while preserving temporal continuity and audio-synchronized timing.
- Overview: The joint generator first produces coherent low-resolution video, which 2K Refiner enhances to 2K while preserving content, motion, and audio-synchronized timing.Direct 2K generation is computationally expensive, whereas lower-resolution generation lacks fine textures and sharp boundaries.
- Inference: At inference, each temporal chunk receives one denoising evaluation conditioned on the low-resolution video and previously refined high-resolution chunks.The audio stream is not modified during refinement.
- Training Pipeline: The training pipeline progresses from a bidirectional multi-step teacher to an autoregressive multi-step refiner and finally a distilled autoregressive one-step student.The deployed refiner is the final student produced with Distribution Matching Distillation.
- Autoregressive Refinement: Autoregressive temporal factorization reduces high-resolution computation while preserving continuity through previously refined context.The deployment procedure refines each new chunk after earlier chunks have been produced.
- Distillation: Self-rollout training prevents exposure bias caused by conditioning the student only on ground-truth past chunks, which could otherwise accumulate errors across time.The cited failure modes include flicker, over-sharpening, and temporal drift.
- Distillation: The student combines distribution matching with perceptual and reconstruction losses computed after decoding generated latents through a frozen VAE.DISTS supplies perceptual supervision and ℓ2 supplies pixel-level consistency.
6 Evaluation
The evaluation measures video, audio, speech, and audio-visual alignment quality on Verse-Bench, alongside comparisons with established and larger open-weight systems. DreamX-Creator performs competitively in human preference tests, while remaining behind larger models on several audio and cross-modal metrics.
- Evaluation Protocol: Verse-Bench evaluates video quality, audio quality, speech generation, and audio-visual alignment across three scenario-specific sets.Set 1 and Set 2 cover general audio-visual events, while Set 3 focuses on speech-centric scenarios with visible speakers.
- Baselines: The quantitative comparisons separate five established research baselines from the larger open-weight systems LTX-2.3 and MiniMax-H3.Table 3 covers NAVA, UniAVGen, Ovi, and DaVinci-MagiHuman variants; Table 4 covers LTX-2.3 and MiniMax-H3.
- Quantitative Results: LTX-2.3 and MiniMax-H3 exceed DreamX-Creator on several audio-aesthetic and cross-modal metrics, while DreamX-Creator remains favorable on video quality and desynchronization.The 7B model does not achieve across-the-board parity with the 22B and 33B baselines.
- Refiner Comparison: The refiner achieves the best overall trade-off, with MUSIQ 0.7073, MANIQA 0.4382, aesthetic score 0.4911, LSE-C and IB best among refinement methods, and DeSync 0.1731.These results indicate improved perceptual quality with stronger semantic consistency and audio-visual alignment.
- Human Preference Study: DreamX-Creator records more wins than losses against Ovi, UniAVGen, NAVA, and DaVinci across every criterion–baseline pair.Video-quality win rates are 64.2%, 73.7%, 61.7%, and 68.2%, respectively.
- Human Preference Study: Against industrial-scale systems, results are competitive on sampled cases but show losses in audio quality and audio-visual alignment, especially against larger models.The study under-represents highly dynamic and compositionally complex scenes, limiting claims of broad parity.
7 Related Work
Related work spans directional and joint audio-video generation, reward-based alignment, and video refinement with few-step distillation. DreamX-Creator addresses reciprocal multimodal interaction and efficient refinement within this broader landscape.
- Joint Audio-Video Generation: Directional audio-video systems generate one modality from the other, but treating one modality as fixed precludes reciprocal co-generation.Joint diffusion models and diffusion transformers instead sample audio and video together.
- Generative Alignment: Video alignment methods extend image alignment objectives with temporal coherence, motion quality, subject consistency, and long-range dynamics.Examples include differentiable rewards, video preference pairs, and reinforcement-learning formulations.
- Multimodal Alignment: Joint audio-video alignment must handle visual quality, audio quality, text consistency, cross-modal semantics, and temporal synchronization simultaneously.Rewards can conflict, global rewards obscure stream-specific attribution, and synchronization often occurs only in local regions.
- Video Refinement: Video restoration and super-resolution methods use recurrent propagation, transformers, or diffusion priors to improve detail and temporal consistency.These approaches include BasicVSR++, RealBasicVSR, VRT, StableVSR, Upscale-A-Video, and VEnhancer.
- Few-Step Distillation: Few-step distillation reduces sampling cost through teacher-student distribution matching, one-step video super-resolution, and autoregressive training that addresses train–test mismatch.DreamX-Creator’s refiner differs by assuming the full low-resolution clip is available before refinement, enabling bidirectional context.
8 Conclusion
The paper presents DreamX-Creator 1.0 as an open framework for joint audio-video generation with shared-timeline bidirectional interaction and planned multimodal post-training and 2K refinement. The authors frame these extensions as designs requiring explicit validation, while releasing the evaluated model and tools to broaden research access.
- Framework: DreamX-Creator 1.0 is an open framework for joint audio-video generation.The paper releases the evaluated model variant, inference stack, and evaluation tools with explicit provenance.
- Generator: The generator preserves dedicated audio and video streams, aligns them on a shared timeline, and exchanges information through gated bidirectional attention.The gate depends on hidden and context information.
- Extensions: Multimodal reinforcement learning and a bidirectional few-step Refiner are presented as next-stage designs targeting user-facing alignment and 2K output.The conclusion states that these stages have explicit validation requirements rather than completed empirical claims.