Source-linked AI summary
UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE
Zhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng, Shenyuan Jiang, Xinyu Chen, Haoyuan Shi, Jinchao Li, Qi Wang, Haolan Chen, Fanbo Meng, Mingjun Zhao, Yu Xu, Yancheng He, Baotian Hu, Min Zhang
TL;DR
Speech and music generation remain difficult to unify because of task conflict and data imbalance. UniMoE-Audio combines dynamic-capacity MoE with a three-stage training curriculum; experiments report synergistic cross-domain learning while avoiding degradation seen in naive joint training.
Problem
Speech and music generation have divergent objectives, and music data are less abundant, complicating unified audio generation.
Source: §1 · p. 1Method
UniMoE-Audio combines Top-P dynamic expert routing and a hybrid expert design with a three-stage, data-aware training curriculum.
Source: §1 · p. 2Results
UniMoE-Audio matches or surpasses strong domain-specific baselines and enables synergistic learning across audio domains, avoiding degradation observed in naive joint training.
Source: §7 · p. 9Takeaways & Limitations
The work provides a robust blueprint for building unified generative audio models.
Source: §7 · p. 10Future directions include incorporating broader audio types and optimizing the MoE architecture.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Recent advances in unified multimodal models indicate a clear trend towards comprehensive content generation. However, the auditory domain remains a significant challenge, with music and speech often developed in isolation, hindering progress towards universal audio synthesis. This separation stems from inherent task conflicts and severe data imbalances, which impede the development of a truly unified audio generation model. To address this challenge, we propose UniMoE-Audio, a unified speech and music generation model within a novel Dynamic-Capacity Mixture-of-Experts (MoE) framework. Architecturally, UniMoE-Audio introduces a Top-P routing strategy for dynamic expert number allocation, and a hybrid expert design comprising routed experts for domain-specific knowledge, shared experts for domain-agnostic features, and null experts for adaptive computation skipping. To tackle data imbalance, we introduce a three-stage training curriculum: 1) Independent Specialist Training leverages original datasets to instill domain-specific knowledge into each "proto-expert" without interference; 2) MoE Integration and Warmup incorporates these specialists into the UniMoE-Audio architecture, warming up the gate module and shared expert using a subset of balanced dataset; and 3) Synergistic Joint Training trains the entire model end-to-end on the fully balanced dataset, fostering enhanced cross-domain synergy. Extensive experiments show that UniMoE-Audio not only achieves state-of-the-art performance on major speech and music generation benchmarks, but also demonstrates superior synergistic learning, mitigating the performance degradation typically seen in naive joint training. Our findings highlight the substantial potential of specialized MoE architecture and curated training strategies in advancing the field of universal audio generation. Homepage: https://mukioxun.github.io/Uni-MoE-site/home.html
1 INTRODUCTION
Speech and music generation face conflicting objectives and imbalanced data, making joint training especially damaging to music quality. UniMoE-Audio combines dynamic expert routing with a staged training curriculum to unify the tasks.
1 INTRODUCTION
- Speech and music have divergent objectives, while speech corpora are more abundant, so naive joint training can degrade both tasks, especially music.The introduction frames overcoming task conflict and data imbalance as the central challenge.Source: §1 · p. 1
- UniMoE-Audio yields synergistic gains across speech and music and achieves the best comprehensive performance against leading models across speech and music metrics.Experiments also report state-of-the-art or competitive results on major generation benchmarks.Source: Figure 1 · p. 2, §1 · p. 2
- Top-P routing dynamically allocates experts by token complexity, while routed, shared, and null experts separate domain-specific features, common features, and skipped computation.The architecture is presented as a means of mitigating speech–music task conflict.Source: §1 · pp. 1–2
- A three-stage curriculum trains specialists independently, warms the gate and shared expert on balanced data, then jointly trains the full model on the balanced dataset.Synergistic joint training is described as supporting learning from imbalanced sources without conventional data sampling.Source: §1 · p. 2
2 RELATED WORK
Prior work advanced speech and music generation largely within separate domains, while unified audio models explored broader generation without covering the full speech–music challenge. UniMoE-Audio addresses that gap with an MoE architecture and data-aware training curriculum.
2.1 Domain-Specific Audio Generation Models
- Speech models advanced text-conditioned synthesis and voice capabilities, while music models explored diffusion and autoregressive generation, including long-form and multimodal approaches.Examples include VALL-E, CosyVoice, MusicGen, and MuMuLlama.Source: §2.1 · pp. 2–3
2.2 Unified Audio Generation Models
- UniAudio targets general audio but reportedly struggles with data imbalance, whereas AudioX generates sound effects and music but omits speech; UniMoE-Audio targets both modalities.The paper frames unified speech–music generation as underexplored despite the fields' shared autoregressive discrete-token paradigm.Source: §2.2 · p. 3, §2.1 · p. 3
3 UNIMOE-AUDIO
UniMoE-Audio unifies speech and music generation from multimodal inputs with a Transformer using Dynamic-Capacity MoE layers. Its architecture combines variable per-token routing with routed, shared, and null experts.
3 UNIMOE-AUDIO
- UniMoE-Audio generates speech and music from text, audio, and video inputs using a Dynamic-Capacity MoE architecture.The framework also supports Voice Cloning, TTS, T2M, and V2M.Source: §3 · p. 3
3.1 Input Representation and Tokenization
- DAC tokenizes audio into discrete acoustic tokens, while a multi-head output predicts all channels; a ViT and projector map video images into soft visual tokens.Predicting all audio channels together avoids additional sequential modules, reducing parameter count and computational latency.Source: §3.1 · p. 3
3.2 Dynamic-Capacity MoE
- Top-P routing selects the smallest expert set whose cumulative probability exceeds threshold p, dynamically varying the number of experts allocated to each token.Expert probabilities are sorted in descending order before selection.Source: §3.2 · pp. 3–4
- Always-active shared experts capture common knowledge and offload routed experts, which can specialize in domain-specific patterns.Routed experts are inefficient for common knowledge because inactive experts are excluded from learning.Source: §3.2 · p. 4
- Parameter-free null experts output zero, expanding the routed-expert activation range to [0, ⌈p(Nr + Nn)⌉] and enabling computation skipping.Without null experts, the range is [1, ⌈pNr⌉].Source: §3.2 · p. 4
4 TRAINING
The training strategy pairs large, imbalanced raw datasets for specialist pretraining with a smaller balanced set for unified-model training. Its three stages progress from isolated specialists to calibrated MoE integration and end-to-end joint training.
4.1 Training Data
- Speech data totals approximately 30K hours versus approximately 10K hours of music, while the balanced training set samples 15K high-quality examples from each of four task domains.The resulting 60K-sample set is used for MoE warmup and synergistic joint training.Source: §4.1 · p. 5
4.2 Three-stage Training Curriculum
- Independent Specialist Training uses the full imbalanced raw datasets to train separate dense models, letting each proto-expert learn domain-specific knowledge without interference.These specialists pre-assign intended functions before integration.Source: §4.2 · p. 5
- MoE Integration and Warmup assembles routed experts from specialist FFNs and calibrates gates and shared experts on balanced data before Synergistic Joint Training unfreezes the full model for end-to-end fine-tuning.An auxiliary load-balancing loss is annealed during joint training, first encouraging balanced expert use and later shifting focus toward sequence generation.Source: §4.2 · p. 5
5 EXPERIMENTS
UniMoE-Audio is evaluated against specialist models and a dense joint-training baseline across speech and music tasks. It reports competitive or superior performance in both domains, with distinct strengths in speech synthesis and music aesthetics.
5.1 UniMoE-Audio Setting
- The evaluated models include four 3.1B single-task specialists, a 7.1B dense Unify-Baseline, and a 7.1B UniMoE-Audio model averaging approximately 4.8B activated parameters at inference.The baseline and UniMoE-Audio have comparable total parameter counts for a fair comparison.Source: §5.1 · p. 5, Table 2 · p. 6
5.2 Implementation Details
- Training uses AdamW and a cosine learning-rate schedule, with distinct GPU counts, batch sizes, and learning rates across the three stages.The stages use 48 GPUs for specialist training and 196 GPUs for both MoE warmup and joint training.Source: §5.2 · p. 6
5.3 Evaluation Setting
- Speech evaluation covers English and Mandarin intelligibility, perceptual quality, and speaker similarity, while music evaluation covers T2M and V2M alignment, audio quality, diversity, and aesthetics.V2M-Bench samples are segmented into 10-second clips.Source: §5.3 · p. 6
5.4 Overall Performance
- UniMoE-Audio achieves competitive or superior performance across speech and music benchmarks, with superior music aesthetics and highly competitive semantic alignment and audio quality.It also reports state-of-the-art results on multiple speech metrics.Source: §5.4 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
- UniMoE-Audio reaches SeedTTS-EN UTMOS 4.36 and WER 1.9 using 280K hours of speech data, while speaker similarity remains below the state of the art.The paper reports that the speech-data volume rivals or surpasses dedicated models trained on 8M–10M hours.Source: §5.4 · p. 6, Table 3 · p. 7
- Across T2M and V2M, UniMoE-Audio obtains the highest aesthetic scores on PC, PQ, and CE, while its FAD and KL scores are inferior to reference-based alternatives.The authors also report strong CLAP and CLaMP3 semantic alignment.Source: §5.4 · pp. 6–7, Table 4 · p. 7
- Compared with the similar-size Unify-Baseline, UniMoE-Audio performs significantly better across speech and music, while retaining robust V2M performance on a task comprising only 5% of training data.The authors attribute the data-imbalance result to pretraining task-specific proto-experts and routing inputs to relevant experts.Source: §5.4 · p. 7
6 DISCUSSION
The discussion examines training stability, layer-wise capacity allocation, and task-specific expert use. The analyses report distinct loss patterns across training stages, adaptive routing budgets, and specialization that varies with depth.
6 DISCUSSION
- The discussion analyzes training-loss curves, expert allocation by layer, and the distribution of expert loading across speech and music tasks.The loss plots distinguish warmup from synergistic joint training.Source: §6 · p. 8, Figure 3 · p. 8
6.1 Training Loss Analysis
- Music loss remains higher than speech loss, converging near 60 versus 40, while joint training is more volatile than warmup across both tasks.The authors interpret the loss gap as evidence that music is intrinsically harder to model and describe warmup as stabilizing router calibration before joint training.Source: §6.1 · p. 8, Figure 3 · p. 8
6.2 Analysis of Dynamic Expert Allocation
- Tokens typically activate 1–3 experts in layers 0–3, mostly 4–5 in layers 4–13 with a peak around layer 12, then fewer again in layers 14–17.The authors contrast this learned layer-wise budget with fixed-budget Top-K routing and report that harder tokens receive more experts than easier ones.Source: §6.2 · pp. 8–9, Figure 4 · p. 8
6.3 Expert Routing Visualization
- Routing load is balanced across the eight routed experts, the null expert is consistently engaged, and experts 1–4 prefer speech while experts 5–8 prefer music.The authors report that this utilization avoids expert collapse and that the specialization persists across layers.Source: §6.3 · p. 9, Figure 5 · p. 9
Hierarchical Processing Emerges from Shallow to Deep
- Task preference is more mixed in shallow layers, while deeper layers show stronger specialization; the null expert prunes tokens from both tasks shallowly and is more active for speech in deeper layers.The authors interpret this as shared low-level processing followed by more domain-specific abstraction, with computation skipped for simpler speech tokens at depth.Source: Hierarchical Processing Emerges from… · p. 9
7 CONCLUSION
UniMoE-Audio addresses the task conflict and data imbalance that hinder unified speech and music generation through a dynamic-capacity MoE architecture and a three-stage training curriculum. Across diverse benchmarks, it matches or surpasses strong domain-specific baselines and avoids the performance degradation observed in naive joint training.
7 CONCLUSION
- Across diverse benchmarks, UniMoE-Audio matches or surpasses strong domain-specific baselines and avoids the performance degradation observed in naive joint training.The experiments also show synergistic learning across audio domains.Source: §7 · p. 9
- UniMoE-Audio combines a dynamic-capacity Mixture-of-Experts architecture to mitigate task conflict with a data-aware, three-stage training curriculum to address data imbalance.Source: §7 · p. 9
- The work identifies broader audio types and MoE architecture optimization as future directions.Source: §7 · p. 10