PaperlayerView exact revision on arXiv

Source-linked AI summary

JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation

Kai Liu, Yanhao Zheng, Kai Wang, Shengqiong Wu, Rongjunchen Zhang, Jiebo Luo, Dimitrios Hatzinakos, Ziwei Liu, Hao Fei, Tat-Seng Chua

arXiv:2602.19163v1cs.CVcs.MMcs.SD

TL;DR

Open-source joint audio-video generation still struggles with quality, synchronization, and preference alignment. JavisDiT++ addresses these gaps with unified modeling and optimization, achieving state-of-the-art performance across quality, consistency, and synchrony using roughly 1M training entries.

  • Problem

    Existing open-source joint audio-video generation methods remain limited in generation quality, temporal synchrony, and alignment with human preferences.

  • Method

    JavisDiT++ combines modality-specific mixture-of-experts, temporally aligned RoPE, and audio-video direct preference optimization for unified JAVG modeling.

  • Results

    JavisDiT++ achieves state-of-the-art performance across audio-video quality, consistency, and synchrony, outperforming existing open-source approaches with roughly 1M training entries.

  • Takeaways & Limitations

    The framework demonstrates that efficient unified modeling and optimization can support high-quality, synchronized, preference-aligned joint audio-video generation.

  • Takeaways & Limitations

    Training on roughly 1M entries may constrain scalability relative to larger proprietary systems and limit generalization and robustness.

Abstract

from arXiv · show

AIGC has rapidly expanded from text-to-image generation toward high-quality multimodal synthesis across video and audio. Within this context, joint audio-video generation (JAVG) has emerged as a fundamental task that produces synchronized and semantically aligned sound and vision from textual descriptions. However, compared with advanced commercial models such as Veo3, existing open-source methods still suffer from limitations in generation quality, temporal synchrony, and alignment with human preferences. To bridge the gap, this paper presents JavisDiT++, a concise yet powerful framework for unified modeling and optimization of JAVG. First, we introduce a modality-specific mixture-of-experts (MS-MoE) design that enables cross-modal interaction efficacy while enhancing single-modal generation quality. Then, we propose a temporal-aligned RoPE (TA-RoPE) strategy to achieve explicit, frame-level synchronization between audio and video tokens. Besides, we develop an audio-video direct preference optimization (AV-DPO) method to align model outputs with human preference across quality, consistency, and synchrony dimensions. Built upon Wan2.1-1.3B-T2V, our model achieves state-of-the-art performance merely with around 1M public training entries, significantly outperforming prior approaches in both qualitative and quantitative evaluations. Comprehensive ablation studies have been conducted to validate the effectiveness of our proposed modules. All the code, model, and dataset are released at https://JavisVerse.github.io/JavisDiT2-page.

1 INTRODUCTION

JavisDiT++ presents a unified approach to joint audio-video generation that targets quality, temporal synchronization, and human-preference alignment. It combines modality-specific modeling, temporal-aligned positional encoding, and preference optimization in a model trained on approximately 1M public data entries.

  • Motivation: JAVG extends multimodal generation by producing synchronized and semantically aligned audio and video from text.The paper situates JAVG within the broader expansion of AIGC from text-to-image generation toward video, audio, and cross-modal synthesis.
  • Model Architecture: MS-MoE improves modality-specific generation by sharing attention for cross-modal exchange while using separate FFNs to aggregate intra-modal information.This design is contrasted with Uniform, which processes aggregated audio-video tokens through a single FFN module.
  • Temporal Synchronization: TA-RoPE aligns audio and video position IDs on a unified temporal axis, enabling direct frame-level temporal synchronization.The strategy explicitly modulates audio-video synchrony against the ST-Prior in JavisDiT and Stitching in Universe-1.
  • Preference Alignment: AV-DPO aligns sounding-video generation with human preferences across quality, consistency, and synchrony.Preference data are curated with diverse reward models and normalized modality-aware ranking.
  • Training and Results: 1M public data entries support a state-of-the-art JAVG model built upon Wan2.1-1.3B-T2V.The model uses 780K diversified audio-text pairs and 360K high-quality sounding videos, supporting 2–5 seconds, 240p–480p resolution, and different aspect ratios.

2 RELATED WORK

Prior JAVG work includes unified-representation methods that share a latent space and intermediate-fusion approaches, while RL-based optimization has aligned generative models with human preferences. Unified representations can lose modality-specific information and offer insufficient temporal control.

  • Joint Audio-Video Generation: Unified JAVG methods project audio and video into a shared latent space, but this constraint can lose modality-specific information and limit temporal control.Examples include CoDi, MM-LDM, and UniForm.
  • RL in Generative Models: RL alignment for generative models has progressed from PPO-based diffusion optimization to DPO-based preference alignment without explicit reward models.The passage cites policy-based PPO approaches and subsequent DPO methods for text-to-image generation.

3 METHODOLOGY

JavisDiT++ uses flow matching within a unified audio-video DiT backbone, with modality-specific experts for cross-modal processing. Temporal-aligned rotary position encoding and preference optimization further target frame-level synchrony, quality, and alignment.

  • Flow Matching: Flow matching trains a neural velocity field along straight-line paths between data and Gaussian-prior samples, then generates samples by solving an ordinary differential equation.The trajectory is x_t = (1 − t)x_0 + tx_1 with target velocity v = x_1 − x_0.
  • Unified Backbone: The unified backbone flattens and concatenates audio and video tokens for full self-attention, while retaining frozen Wan2.1 video and AudioLDM2 audio VAEs.This replaces dual-stream processing with dense cross-modal interaction and modality-specific intra-modal aggregation.
  • Modality-Specific Feed-Forward Network: MS-MoE deterministically assigns audio and video tokens to modality-specific feed-forward networks after cross-modal attention.Unlike conventional dynamically routed MoE architectures, the experts are selected according to token modality.
  • Temporal-Aligned RoPE: TA-RoPE explicitly enforces temporal alignment and synchrony by assigning audio position IDs with an added leading time dimension aligned to video tokens.The method preserves 3D RoPE for video and uses position-ID manipulation to emulate interleaved audio-video timing without physically reordering tokens.
  • Audio-Video Direct Preference Optimization: AV-DPO aligns rectified-flow JAVG models with human preferences using modality-aware audio, video, and audio-video alignment rewards.It uses curated preference pairs and adds flow matching loss for regularization to avoid overfitting.

4 EXPERIMENTS

JavisDiT++ adapts Wan2.1-1.3B-T2V for joint audio-video generation and achieves strong objective and subjective results through unified modality-specific modeling, temporal alignment, and AV-DPO optimization. Ablations and user studies support the effectiveness and efficiency of these designs.

  • Experimental setup: JavisDiT++ progressively adapts Wan2.1-1.3B-T2V through audio pre-training, audio-video SFT, and audio-video DPO.Training uses 780K audio-text pairs, 330K audio-video-text triplets, and an additional 25K DPO samples.
  • Main results: JavisDiT++ significantly outperforms previous JAVG methods across audio-video quality and consistency, including UniVerse-1 with the same Wan2.1-1.3B backbone.The paper attributes this improvement to unified MS-MoE design rather than stitching two pretrained models together.
  • Main results: TA-RoPE provides accurate audio-video temporal alignment, while qualitative results surpass JavisDiT and UniVerse-1 and narrow the gap to Veo3.Inference adds only 1.6% cost over Wan2.1, compared with the higher efficiency burden of two-stream methods.
  • Module analysis: Ablations show that modality-specific MoE enables JAVG, while limited-capacity LoRA adaptation suffers from poor audio quality and consistency.The study also evaluates LoRA placement during audio pretraining and joint audio-video training using normalized metric scores.
  • Module analysis: AV-DPO improves preference alignment, with its reward strategies evaluated through comparisons of win–lose pair selection.Modality-agnostic Average-Micro and Average-Macro strategies fail to achieve consistent improvements across audio-video generation dimensions.
  • User study: Human evaluations show JavisDiT++ surpassing JavisDiT and UniVerse-1 by more than 70% in preference, while AV-DPO yields over 25% more favored videos.Each comparison used blind win–tie–lose judgments from three volunteers per prompt, averaged across annotators.

5 CONCLUSION · A DISCUSSION

JavisDiT++ is presented as a concise, efficient framework for native joint audio-video generation. Its MS-MoE, TA-RoPE, and AV-DPO components support state-of-the-art quality, consistency, and synchrony using 1M training entries on Wan2.1-1.3B-T2V.

  • 5 CONCLUSION: JavisDiT++ is a concise and efficient framework for native joint audio-video generation.
  • 5 CONCLUSION: MS-MoE enhances modality-specific generation quality within the framework.
  • 5 CONCLUSION: TA-RoPE provides explicit temporal alignment between audio and video generation.
  • 5 CONCLUSION: AV-DPO aligns generated outputs with preferences across quality, consistency, and synchrony.
  • 5 CONCLUSION: 1M data entries were used to train JavisDiT++.The model is built upon Wan2.1-1.3B-T2V.
  • 5 CONCLUSION: JavisDiT++ achieves state-of-the-art performance in quality, consistency, and synchrony.
  • A DISCUSSION: The model significantly outperforms existing open-source approaches.

A.1 POTENTIAL LIMITATIONS … B.1 MODEL DETAILS

The paper identifies limitations involving scale, model capacity, tuning, controllability, and broader multimodal tasks, while reporting ethical safeguards and implementation details for its audio, video, text, and DiT components.

  • A.1 POTENTIAL LIMITATIONS: The roughly 1M-entry training set may constrain scalability relative to larger proprietary systems, motivating more diverse and higher-quality audio-video data.Expanding the dataset could improve generalization and robustness.
  • A.1 POTENTIAL LIMITATIONS: The 1.3B-parameter backbone may limit representational capacity for subtle cross-modal temporal and semantic correlations.Larger backbones could provide stronger modeling capacity.
  • A.1 POTENTIAL LIMITATIONS: Parameter-efficient tuning such as LoRA may leave performance gains available from full-parameter finetuning, although at higher computational cost.The current approach relies on parameter-efficient tuning.
  • A.1 POTENTIAL LIMITATIONS: Current experiments target general text-to-audio-video generation rather than fine-grained control of rhythm, pitch, timbre, or lexical content in music and speech.Controllable generation is identified as an important next step.
  • A.1 POTENTIAL LIMITATIONS: Future unified multimodal generation could extend beyond text-conditioned JAVG to audio-to-video, video-to-audio, and audio-image-to-video tasks.The stated goal is a framework that seamlessly performs across these modalities.
  • A.2 ETHICS STATEMENT: All datasets and models are publicly available, contain no private or sensitive information, and include planned model-generated DPO data that does not infringe personal privacy.These statements constitute the paper’s ethics rationale.
  • A.4 LLM USAGE STATEMENT: LLMs were used only for language polishing and presentation refinement, not for conceiving the paper’s core ideas or designs.The statement limits LLM involvement to writing assistance.
  • B.1 MODEL DETAILS: The implementation freezes the AudioLDM2 audio VAE, Wan2.1 video VAE, and Wan2.1 umT5-xxl text encoder while adapting the Wan2.1-1.3B-T2V DiT with newly introduced audio modules and LoRA components.Audio uses 16 kHz signals, 64-bin mel-spectrograms, 8-channel embeddings, and 2×2 patchification; video uses temporal compression by 4 after the first frame, spatial compression by 8×8, 16-channel embeddings, and 2×2 spatial patchification; the text encoder has context length 512, while the DiT has 30 layers and hidden dimension 1536.

B.2 TRAINING DETAILS

JavisDiT++ is trained through a three-stage pipeline that progressively extends Wan2.1-1.3B-T2V to joint audio-video generation. The training uses audio-text and filtered audio-video-text data, with LoRA retained for preference optimization.

  • Three-stage training pipeline: The three stages are Audio Pre-Training, Audio-Video SFT, and Audio-Video DPO, progressively extending Wan2.1-1.3B-T2V to JAVG.Audio pre-training uses 780K audio-text pairs for 50 epochs; Audio-Video SFT uses LoRA on 330K triplets for 1 epoch; DPO retains LoRA parameters.
  • Audio Pre-Training: 780K audio-text pairs train the Audio FFN for 50 epochs at a learning rate of 1e-4.The audio corpus spans public datasets including AudioSet, AudioCaps, VGGSound, WavCaps, Clotho, ESC50, GTZAN, and MACS.
  • Audio-Video SFT: 330K audio-video-text triplets train the model with LoRA for 1 epoch at a learning rate of 1e-4.The triplets are selected from a subset of 1.1 million TAVGBench entries after filtering.
  • Audio-Video SFT: Video data is filtered by removing human-speech videos and excluding lower-quality samples using aesthetic, flow, and OCR scores.FunASR detects human speech, while the remaining filtering follows OpenSora’s procedure.

B.3 EVALUATION DETAILS

Evaluation uses JavisBench to assess joint audio-video generation quality, consistency, and synchrony under a fixed 240P, 4-second output setting. Ablations use a 1,000-prompt subset, while broader comparisons include cascaded open-source pipelines and Veo3.

  • Evaluation Setup: JavisBench contains 10,140 real-world joint audio-video prompts, while JavisBench-mini randomly selects 1,000 prompts for ablation studies.All models generate 240P, 4-second sounding videos for quantitative evaluation.
  • Evaluation Dimensions: JavisBench evaluates audio/video perceptual quality and text consistency using FVD, KVD, Fréchet audio distance, ImageBind, CLIP-Score, and CLAP-Score.FVD and KVD measure distributional differences, with lower values indicating better quality; text-consistency metrics assess semantic matching to the input description.
  • Evaluation Dimensions: Audio-video semantic consistency is measured with ImageBind audio-video cosine similarity and AVHScore, while CAVP-Score is removed because its narrow range cannot distinguish results.A higher AVHScore indicates stronger audio-video semantic consistency.
  • Evaluation Dimensions: Spatio-temporal synchrony is evaluated with JavisScore and DeSync, where lower DeSync indicates better synchronization.DeSync predicts audio-visual temporal offset through a 21-category classification task spanning offsets from -10 to 10, then uses absolute values.
  • Compared Methods: Comparisons include cascaded T2A+A2V and T2V+V2A pipelines, with AudioLDM2 providing preliminary text-to-audio generation for T2A+A2V.For Fig. 2, 100 JavisBench prompts are sampled; open-source models run locally, while Veo3 is queried through its API.

C DETAILED COMPARISON WITH RELATED WORKS · D ADDITIONAL EXPERIMENTS

JavisDiT++ differs from recent JAVG systems through shared attention with modality-specific FFNs and a TA-RoPE strategy that avoids audio-video position overlap. The comparison also notes that AudioGen-Omni cannot be thoroughly analyzed because its RoPE design lacks details and publicly unavailable code prevents evaluation.

  • C DETAILED COMPARISON WITH RELATED WORKS: Figure A3 compares JavisDiT++ with UniForm, JavisDiT, and UniVerse-1, while Figure A4 compares its audio-video frame interleaving with Qwen2.5-Omni’s strategy.
  • C DETAILED COMPARISON WITH RELATED WORKS: UniForm uses one shared attention and FFN parameter set for audio and video tokens, challenging audio extension while preserving pretrained video quality.These limitations are validated in Tab. 2.
  • C DETAILED COMPARISON WITH RELATED WORKS: JavisDiT introduces dual-stream audio-video generation with separate parameters, ST-Prior, and frame-level bidirectional cross-attention.
  • C DETAILED COMPARISON WITH RELATED WORKS: JavisDiT++ instead combines shared attention for cross-modal interaction with modality-specific FFNs for stronger intra-modal modeling.The design is presented as simpler, more effective, efficient, performant, and scalable than prior alternatives.
  • C DETAILED COMPARISON WITH RELATED WORKS: Qwen2.5-Omni aligns audio and video through position IDs but assigns overlapping three-dimensional IDs because audio is treated as a 1D token sequence.Examples include coinciding IDs such as (0, 0, 0), (25, 25, 25), and (50, 50, 50).
  • C DETAILED COMPARISON WITH RELATED WORKS: For generative JAVG, Qwen2.5-Omni’s audio-video position overlaps introduce non-trivial positional issues despite having negligible influence in multimodal understanding.
  • C DETAILED COMPARISON WITH RELATED WORKS: TA-RoPE represents audio as a 2D mel-spectrogram, matches the time-axis position-ID dimension, and adaptively offsets the other two dimensions.The offsets correspond to video width and height and completely avoid cross-modal position overlap.
  • C DETAILED COMPARISON WITH RELATED WORKS: AudioGen-Omni also identifies RoPE as important for video-to-audio temporal alignment but cannot be thoroughly compared because its paper lacks details and its code is unavailable.

D.1 ABLATION ON POSITIONAL ENCODING STRATEGY

The ablation compares four audio positional encoding strategies for audio-video generation and finds that integer, non-overlapping audio-video position IDs best preserve audio quality while improving video quality. Accordingly, the authors adopt Interleave+Offset as the final scheme.

  • Compared strategies: The study evaluates Vanilla, Interpolate, Interleave, and Interleave+Offset using full audio pretraining and audio-video supervised fine-tuning on Wan2.1-1.3B-T2V.The comparison examines effects on both audio and video generation quality.
  • Audio quality: Integer audio position IDs are essential for audio quality, whereas Interpolate performs significantly worse because fractional IDs represent offsets that frozen attention layers cannot learn.Interpolate aligns audio and video temporally by interpolating intermediate position IDs.
  • Final encoding: Interleave+Offset aligns audio with video temporally and shifts the remaining audio dimensions by video width and height to avoid cross-modal position-ID overlap.Its position construction uses offsets of H and W for the remaining dimensions.
  • Video quality: Video quality decreases as overlap between audio and video position IDs increases, following the order Vanilla → Interleave → Interleave+Offset.The comparison is consistent with the positional-overlap analysis in Section C.
  • Final encoding: The authors adopt Interleave+Offset as the final position encoding scheme.This choice follows the ablation findings on audio quality, video quality, and position-ID overlap.

D.2 INVESTIGATION ON TRAINING DATA QUALITY AND DIVERSITY

The study finds that AV-SFT benefits from a balance of training-data quality and diversity: 330K medium-quality samples outperform narrower high-quality or larger low-quality alternatives. Accordingly, the authors adopt the 330K medium-quality dataset as a practical trade-off.

  • Data diversity: 330K medium-quality samples outperform 120K high-quality samples, showing that high quality with low diversity is insufficient for joint audio-video generation.Joint generation requires sufficient data quantity or diversity to acquire new capabilities.
  • Data quality: 720K low-quality samples produce clearly inferior generation quality compared with 330K medium-quality samples.Low-quality data undermine the high-quality priors learned by the Wan2.1 backbone during pretraining.
  • Data quality: A second SFT round with 120K high-quality or 330K medium-quality data cannot fully recover performance lost after training on 720K low-quality data.The result indicates that degradation from low-quality training data is not fully reversible through subsequent SFT.
  • Data selection: The authors adopt the 330K medium-quality dataset for AV-SFT as a reasonable trade-off between data quality and diversity.They expect further improvements in both diversity and quality to yield better scaling properties.

D.3 ABLATION ON HYPER-PARAMETERS OF AV-DPO · D.4 MORE VISUALIZATIONS

The AV-DPO hyperparameter study uses implicit accuracy to compare β and learning-rate settings, revealing modality-specific β preferences and a preferred learning rate. Additional visualizations demonstrate strong joint audio-video generation across multiple dimensions.

  • D.3 ABLATION ON HYPER-PARAMETERS OF AV-DPO: Implicit accuracy serves as a unified proxy for evaluating AV-DPO hyperparameter effects because parameter choices influence loss magnitude unevenly.It measures movement toward chosen-data distributions and away from rejected-data distributions.
  • D.3 ABLATION ON HYPER-PARAMETERS OF AV-DPO: The modality-specific β behavior reflects pretrained video alignment versus a newly trained audio branch that is initially less aligned with human preferences.Larger β values such as 3000 or 5000 keep video closer to the reference model, whereas audio requires a smaller β such as 1000.
  • D.3 ABLATION ON HYPER-PARAMETERS OF AV-DPO: Among learning rates of 1 × 10^-5, 5 × 10^-6, and 1 × 10^-6, 1 × 10^-5 provides the fastest convergence and highest final accuracy.The learning-rate comparison is presented in Fig. A7.
  • D.3 ABLATION ON HYPER-PARAMETERS OF AV-DPO: The selected AV-DPO learning rate is 1 × 10^-5, consistent with using approximately one-tenth of the SFT-stage learning rate during DPO.This setting is adopted based on the reported convergence and accuracy results and prior work.
  • D.4 MORE VISUALIZATIONS: Figures A8 and A9 provide additional joint audio-video generation cases showcasing the model’s strong generative capability across multiple dimensions.Both figures are described as examples of high-quality audio-video generation results.
Loading 2602.19163v1…