Source-linked AI summary
Apollo: Unified Multi-Task Audio-Video Joint Generation
Jun Wang, Chunyu Qiang, Yuxin Guo, Yiran Wang, Xijuan Zeng, Feng Deng
TL;DR
Audio-video generation still faces alignment problems and gaps in cross-modal modeling, training, and densely captioned data. APOLLO combines a unified architecture, multitask training, and automated data construction, and reports strong results across joint and unimodal tasks, including performance comparable to Veo 3.
Problem
Audio-video generation still faces asynchrony, lip-speech mismatches, and unimodal degradation, alongside limited cross-modal modeling, generalization, and high-quality dense-caption data.
Source: §1 · p. 2Method
APOLLO combines a single-tower architecture with Omni-Full Attention, progressive multitask training, and automated data construction for audio-video generation.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors position APOLLO as a unified direction for further research in audio-video generation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Audio-video joint generation has progressed rapidly, yet substantial challenges still remain. Non-commercial approaches still suffer audio-visual asynchrony, poor lip-speech alignment, and unimodal degradation, which can be stemmed from weak audio-visual correspondence modeling, limited generalization, and scarce high-quality dense-caption data. To address these issues, we introduce Apollo and delve into three axes--model architecture, training strategy, and data curation. Architecturally, we adopt a single-tower design with unified DiT blocks and an Omni-Full Attention mechanism, achieving tight audio-visual alignment and strong scalability. Training-wise, we adopt a progressive multitask regime--random modality masking to joint optimization across tasks, and a multistage curriculum, yielding robust representations, strengthening A-V aligned world knowledge, and preventing unimodal collapse. For datasets, we present the first large-scale audio-video dataset with dense captions, and introduce a novel automated data-construction pipeline which annotates and filters millions of diverse, high-quality, strictly aligned audio-video-caption triplets. Building on this, Apollo scales to large datasets, delivering high-fidelity, semantically and temporally aligned, instruction-following generation in both joint and unimodal settings while generalizing robustly to out-of-distribution scenarios. Across tasks, it substantially outperforms prior methods by a large margin and achieves performance comparable to Veo 3, offering a unified, scalable path toward next-generation audio-video synthesis.
1 INTRODUCTION
Audio-video generation still faces synchronization, lip-speech alignment, and unimodal-quality problems, alongside gaps in cross-modal modeling, training strategy, and aligned data. APOLLO addresses these gaps with coordinated architecture, training, and dataset contributions, and reports strong results across joint and unimodal generation.
1 INTRODUCTION
- Existing joint audio-visual models exhibit asynchrony, lip-speech mismatches, and unimodal degradation, amid limited cross-modal interaction, single-task training, and scarce aligned, densely annotated data.The introduction attributes these issues to architecture, learning strategy, and data limitations.Source: §1 · p. 2
- APOLLO combines a unified single-tower architecture with Omni-Full Attention, progressive multitask training, and automated data construction for audio-video generation.The paper reports high-fidelity, aligned joint and unimodal generation, instruction following, and OOD generalization.Source: §1 · p. 2
- APOLLO consistently excels in unimodal and joint audio-video generation, outperforming prior state-of-the-art benchmarks and achieving performance comparable to Veo 3.The work also reports an 81-million-sample dataset with accurate dense captions.Source: §1 · p. 2
2 RELATED WORKS
Related work surveys text-to-video, image-to-video, text-to-audio, and joint audio-video generation. Across these areas, it identifies challenges in data, motion, task complexity, and synchronized speech-video generation.
Text-to-Video Generation (T2V)
- Text-to-video systems commonly encode video with a 3D VAE and denoise spatiotemporal latents with a DiT, while data quality and scale remain important.The review traces a shift from U-Net backbones toward DiT models and lists open-source and closed-source systems.Source: §2 · p. 3
Image-to-Video Generation (I2V)
- Image-to-video methods progressed from single-frame conditioning to cascaded, dual-injection, and adapter-based designs, but curated data, long-range motion, and appearance-motion trade-offs remain challenges.The review also notes work on motion modeling and stable sampling.Source: §2 · p. 3
Text-to-Audio Generation (TTA)
- Text-to-audio methods improve semantic alignment and generation, yet the review identifies short durations and limited flexibility for complex, evolving instructions as open challenges.Examples span latent diffusion, hierarchical diffusion, and flow-matching approaches.Source: §2 · p. 3
Audio-Video Joint Generation (T2AV)
- Joint audio-video work uses coupled or dual backbones, adapters, and unified towers, while synchronized speech and video synthesis remains underexplored.The review notes that many existing models focus on sound effects or music.Source: §2 · p. 3
3 APOLLO
APOLLO generates audio and video from modality-specific latents and conditions, using a unified full-attention architecture and progressive multitask training. Its design combines shared positional encoding, modality masking, and staged training across generation tasks.
3.1 PRELIMINARY
- The model recursively denoises audio and video latents from pure Gaussian noise at t = 0 toward final generation at t = 1, conditioned on text.Its four inputs are video, video-related text, audio-related text, and audio; captions and speech text are encoded alongside the media.Source: §3.1 · pp. 3–4
3.2 SINGLE TOWER WITH FULL ATTENTION
- APOLLO feeds separately encoded video, video text, audio text, and audio into a single MM-DiT tower, which outputs separate video and audio latents for decoding.The model uses a 3D visual encoder and a text embedding encoder for captions.Source: §3.2 · p. 4, §3.1 · p. 4
- MixD-RoPE applies 3D temporal, width, and height positions to video and compatible temporal 1D positions to audio, sharing temporal position IDs across modalities.Audio position numbering begins one position after the video modality's maximum temporal position ID.Source: §3.2 · p. 4
- Omni-Full Attention concatenates normalized hidden states for video, video text, audio text, and audio for joint attention, then splits the attention values back into modality streams.The design unifies all input modalities within joint full-attention.Source: §3.2 · pp. 4–5
3.3 MULTI-TASK PROGRESSIVE TRAINING STRATEGY
- Random modality masking supports T2A, T2V, T2AV, I2V, and I2AV objectives, with staged training progressing from broad pretraining to adaptive specialization and quality refinement.Pretraining targets basic generation capabilities; later stages rebalance data toward weaker tasks and fine-tune on manually curated high-quality data.Source: §3.3 · p. 5
4 DATASET CONSTRUCTION
The dataset construction pipeline filters audio and video, divides clips into audio-type splits, and applies specialist annotations that are merged into dense captions.
4 DATASET CONSTRUCTION
- 27% of automatically annotated samples remain after filtering; the dataset covers single-speaker speech, multi-speaker speech, singing, and natural sound.Source: §4 · p. 5
4.1 DATASET FILTERING
- Videos are screened for quality, naturalness, and safety, then scene-split; audio filtering removes quality defects and uses Synchformer and ImageBind to assess temporal and semantic alignment.Source: §4.1 · p. 6
4.2 AUDIO-GUIDED DATA SPLITTING
- Clips are partitioned into vocal and non-vocal sound splits, then vocal clips are further divided into singing, single-speaker speech, and multi-speaker speech.Source: §4.2 · p. 6
4.3 DENSE ANNOTATION AND INTEGRATION
- Specialist models provide transcripts, audio captions, video captions, and vocal speaker attributes, which are merged into unified dense captions.Source: §4.3 · p. 6
5 EXPERIMENTS
Experiments compare Apollo with cascaded, joint-generation, and task-specialized baselines, then assess qualitative outputs and ablations. Apollo shows stronger cross-task performance and audio–video consistency, while its architecture and training-stage ablations support the reported design choices.
5.1 EXPERIMENTAL SETUP
- The evaluation compares cascaded and joint-generation methods across multiple tasks using video, audio, and audio–video consistency metrics.Metrics include video quality and identity, audio quality and semantic alignment, and synchronization measures.Source: §5 · p. 6, §5.1 · p. 7
5.2 COMPARISON WITH EXISTING METHODS
- In joint generation, Apollo delivers high audio and video quality, exceeding cascaded and joint baselines by 34% and 18%, respectively.It also outperforms specialized T2A and T2V models on their respective tasks.Source: §5.2 · p. 7
- Across TI2AV, TI2V, T2V, and T2A, Apollo matches or surpasses task-specialized state-of-the-art models and improves audio–video consistency and synchronization over prior methods.The main comparison is summarized in Table 1.Source: §5.2 · p. 7, Table 1 · p. 7
5.3 QUALITATIVE RESULTS
- Qualitative examples show phoneme-level mouth–audio alignment for Apollo, while Universe-1 and Ovi display misalignment, delay, and audio–visual mismatch.Figure 4 presents these lip-sync comparisons.Source: §5.3 · pp. 7–8, Figure 4 · p. 8
- Further examples show expressive affect, synchronized singing and rap, aligned overlapping sounds, and identity-preserving image-conditioned generation.The comparisons contrast Apollo with baseline expression distortions, lip-sync failures, poor audio–visual consistency, and identity drift.Source: §5.3 · p. 8
5.4 ABLATION AND ANALYSIS
- The single-tower model outperforms the dual-tower variant in audio quality, video quality, and audio–video consistency.A pretrained audio tower converges faster but performs suboptimally, which the authors attribute to distribution mismatch with video features.Source: Table 2 · p. 8, §5.4 · p. 9
- Multi-task training outperforms T2AV-only training on joint generation and video-only models on T2V and I2V, with transfer to I2AV and I2V.Table 3 reports the multi-task masking ablation.Source: §5.4 · p. 9, Table 3 · p. 9
- Progressive training first establishes basic audio–video generation; the SP stage raises the IB score by 0.1, and high-quality-data post-training adds an overall gain.Removing the progressive schedule causes a significant performance drop.Source: §5.4 · p. 9, Figure 5 · p. 9
6 CONCLUSION
APOLLO combines a unified multi-task framework, a single-tower backbone with Omni-Full Attention, progressive training, and an automated audio–video annotation pipeline. It outperforms prior state-of-the-art on joint and unimodal benchmarks and is comparable to Veo 3.
6 CONCLUSION
- APOLLO outperforms prior state-of-the-art on unimodal and joint audio–video benchmarks and is comparable to Veo 3.Source: §6 · p. 9
- APOLLO combines a single-tower backbone with Omni-Full Attention, progressive training, and an automated pipeline that produces a high-quality annotated audio–video dataset.Source: §6 · p. 9
- The work identifies asynchrony, lip–speech mismatch, and unimodal degradation as failure modes associated with suboptimal architectures, misaligned data, and single-task training.Source: §6 · p. 9