Source-linked AI summary
UniAudio: An Audio Foundation Model Toward Universal Audio Generation
Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Zhou Zhao, Xixin Wu, Helen Meng
TL;DR
Audio generation research has largely used task-specific systems, motivating a unified model for diverse conditional audio tasks. UniAudio tokenizes audio and other modalities into sequences, concatenates conditions with targets, and applies LLM-based prediction with a multi-scale Transformer for long codec sequences. It supports 11 tasks with competitive performance across them and can incorporate unseen tasks through fine-tuning, although its task and modality coverage remains limited and generated samples may contain errors.
Problem
Prior audio-generation systems are commonly task-specific, while diverse and changing demands motivate a unified model covering multiple conditional audio tasks.
Method
UniAudio tokenizes audio and conditioning modalities, formulates tasks as [conditions, target] sequences for LLM prediction, and uses a multi-scale Transformer for long audio-token sequences.
Results
UniAudio supports 11 audio generation tasks, provides competitive performance across them, and adapts to unseen tasks through fine-tuning.
Takeaways & Limitations
The results support UniAudio as a unified model with the potential to serve as a foundation for universal audio generation and continued task integration.
Takeaways & Limitations
UniAudio excludes several audio tasks, does not explore introducing new modalities during fine-tuning, and generated samples are not guaranteed to be error-free.
Abstract
from arXiv · showhide
Large Language models (LLM) have demonstrated the capability to handle a variety of generative tasks. This paper presents the UniAudio system, which, unlike prior task-specific approaches, leverages LLM techniques to generate multiple types of audio (including speech, sounds, music, and singing) with given input conditions. UniAudio 1) first tokenizes all types of target audio along with other condition modalities, 2) concatenates source-target pair as a single sequence, and 3) performs next-token prediction using LLM. Also, a multi-scale Transformer model is proposed to handle the overly long sequences caused by the residual vector quantization based neural codec in tokenization. Training of UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all generative tasks, aiming to obtain sufficient prior knowledge not only in the intrinsic properties of audio but also the inter-relationship between audio and other modalities. Therefore, the trained UniAudio model has the potential to become a foundation model for universal audio generation: it shows strong capability in all trained tasks and can seamlessly support new audio generation tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves state-of-the-art or at least competitive results on most of the 11 tasks. Demo and code are released at https://github.com/yangdongchao/UniAudio
1 INTRODUCTION
UniAudio targets universal audio generation by unifying diverse audio tasks and modalities in one LLM-based system. It combines shared representations and task formulation with multi-task training and fine-tuning to support 11 tasks.
- Motivation: UniAudio addresses the limitations of task-specific audio generators by targeting multiple speech, sound, music, and singing tasks with one model.The motivation includes increasing and varying demands for conditional audio generation.
- Scalability: UniAudio is trained jointly on seven audio generation tasks and fine-tuned to add four unseen tasks.This two-stage process is intended to provide prior knowledge across audio and related modalities while enabling adaptation.
- Contributions: UniAudio is presented as a unified solution for 11 audio generation tasks, with overall performance validated through experiments.The paper also releases demo and code for future research use.
- Contributions: The system introduces unified sequential representations, task formulation, and an audio-generation-specific efficient architecture.These are identified as the paper’s main methodological contributions.
2 UNIAUDIO
UniAudio tokenizes audio and conditioning modalities into sequences, formulates every task as conditional sequence modeling, and uses a multi-scale Transformer for long codec-token sequences. Its task formats encode modality boundaries and task-specific conditions.
- 2.1 Tokenization: Audio is represented with a neural codec and residual vector quantization, producing discrete tokens that can be decoded back into waveforms.The codec compresses audio into discrete representations and reconstructs the waveform through a decoder.
- 2.1 Tokenization: The discrete audio matrix is flattened so every nq tokens belonging to one frame remain consecutive before language-model processing.The experiments use nq = 3 unless otherwise stated.
- 2.1 Tokenization: Phonemes, MIDI, text, and semantic tokens provide discrete or continuous conditioning sequences alongside audio.Text uses pretrained text-LLM embeddings, while semantic tokens come from self-supervised audio representations and K-means clustering.
- 2.2 Unified Task Formulation: All tasks are uniformly formulated by splicing condition subsequences before the target-audio subsequence for sequential modeling.Special tokens mark sequence boundaries, modality boundaries, and task identity.
- 2.2 Unified Task Formulation: UniAudio supports 11 task-specific sequence formats, with preprocessing such as noise, reverberation, or speaker mixing applied to some conditions.The task table distinguishes training versus fine-tuning tasks and records modality types and speaker prompts.
- 2.3 Multi-scale Transformer: The multi-scale Transformer separates inter-frame global modeling from intra-frame local modeling to address the long flattened sequences used by prior approaches.Flattened self-attention has O((T*nq)^2) space complexity, while the proposed design alleviates this to O(T^2).
3 EXPERIMENTS
UniAudio is evaluated across seven jointly trained tasks and four fine-tuned unseen tasks, with additional studies examining unified training and multi-scale Transformer efficiency. Results show competitive task performance, benefits from joint training and scalable fine-tuning, and a quality–efficiency trade-off for the proposed architecture.
- 3.3 THE RESULTS OF 4 GENERATIVE TASKS IN THE FINE-TUNING STAGE: Fine-tuned UniAudio surpasses baselines for audio editing and speech dereverberation and approaches ground-truth quality for instructed TTS.Speech editing also improves considerably over generating the whole sentence.
- 3.4.1 BENEFIT OF BUILDING UNIFIED AUDIO GENERATION MODEL: Joint training consistently outperforms task-specific models, while fine-tuning four new tasks does not affect performance on the original seven tasks.Increasing each task’s training data consistently benefits UniAudio, supporting the scalability of the unified setup.
- 3.4.2 THE EFFECTIVENESS OF MULTI-SCALE TRANSFORMER MODEL: The multi-scale Transformer preserves comparable generation quality to flattening prediction while reducing its time and memory costs.Flattening has O((T*nq)^2) self-attention complexity, whereas the multi-scale design reduces the relevant complexity to O(T^2); it remains less efficient than three concurrent-prediction baselines.
4 RELATED WORKS
Audio generation research has expanded from task-specific systems toward LLM-based models and a more unified paradigm. UniAudio is motivated by the labor and data demands of handling emergent audio tasks case by case.
- Task-specific audio generation: Conventional audio-generation systems are task-specific, domain-dependent, and built around distinctive workflows and generative models.Their designs draw on linguistics, signal processing, auditory perception, and machine learning.
- LLM-based audio generation: LLM-based audio research has shown competitive performance on independent tasks including text-to-speech and music generation.The passage identifies TTS and music generation as established application areas for LLM-based models.
- Motivation for unification: Handling each audio-generation task separately is laborious, particularly under data shortages and emergent, varying demands.The passage presents universal audio generation as a promising and practical alternative.
5 LIMITATION
UniAudio does not cover every known audio-generation task, and its fine-tuning scope is limited to modalities already seen during training. The authors also note that generated samples may contain errors.
- Task coverage: UniAudio excludes tasks such as noise removal, noise speech edit, and speech-to-speech translation.These omissions define a concrete boundary on the set of tasks directly supported by the proposed system.
- Fine-tuning scope: Fine-tuning new tasks with previously unseen modalities remains unexplored.New fine-tuned tasks are formulated using modalities present during training.
- Output quality: UniAudio’s generated samples are not guaranteed to be high quality and may contain errors.
6 CONCLUSION
UniAudio is a unified LLM-based model for 11 audio-generation tasks, designed to address varying audio-generation needs. Experiments report competitive performance across these tasks and continued integration of unseen tasks through fine-tuning.
- Conclusion: UniAudio supports 11 different audio-generation tasks within one unified LM-based generative model.
- Conclusion: UniAudio provides competitive performance on all 11 evaluated tasks.
- Conclusion: Experiments empirically demonstrate that UniAudio can continuously integrate unseen audio-generation tasks.
7 ETHICAL STATEMENT
The paper frames diverse-audio generation as requiring responsible development and broad access. It releases code and uses openly accessible datasets while acknowledging misuse risks and withholding checkpoints.
- Ethical responsibility: The authors emphasize that advances in diverse-audio generation should account for ethical responsibilities and societal benefit.
- Openness: Code is released publicly, and experiments use openly accessible datasets to support access, comparison, and reproduction.
- Misuse risks: The model’s ability to generate speech and music may enable misinformation, deepfake audio, or other harmful content.The authors advocate responsible use respecting privacy and regulations, and state that checkpoints will not be released.
Appendices
The appendices provide detailed descriptions of the experimental setups, including data statistics, model architecture, and optimization strategy.
- The appendix details the experimental setups for UniAudio.It covers data statistics, model architecture, and optimization strategy.
A.1 DATA DESCRIPTION
This section documents the datasets, model configuration, optimization settings, and evaluation conventions used in the experiments.
- Data: Twelve public datasets are used for training, with additional test sets reserved for zero-shot evaluation.Dataset usage by task is specified separately, and some datasets serve multiple tasks.
- Data: The appendix provides dataset statistics and task-specific dataset adoption tables.
- Model configuration: The multi-scale Transformer configuration is specified in a dedicated model-configuration table.
- Optimization: Training and fine-tuning optimization configurations are reported separately.
- Evaluation: The appendix labels evaluations as zero-shot when training and test sets come from different datasets.
B.1 TTS AND VC TASKS
The task-specific experiments evaluate UniAudio across speech, enhancement, singing, sound, editing, instruction-following, and dereverberation settings against prior systems.
- TTS and VC: UniAudio achieves strong TTS results, including better WER and SIM than several prior systems and a 3.56 SMOS zero-shot cloning score.Its zero-shot cloning SIM is 0.708, while objective performance is comparable to VoiceBox.
- TTS and VC: UniAudio obtains better voice-conversion performance than previous work under comparisons involving both small- and large-scale systems.
- Speech enhancement and target speaker extraction: UniAudio obtains the best DNSMOS score for speech enhancement, while PESQ and VISQOL are lower than competing systems.The authors question whether signal-level metrics adequately assess generative methods and emphasize DNSMOS and MOS.
- Singing and text-to-sound: UniAudio produces singing results comparable to Make-an-Voice and DiffSinger and outperforms DiffSound and AudioLDM on text-to-sound generation.
- Audio edit: Fine-tuning pretrained UniAudio improves audio editing over training from scratch and achieves better performance than the previous state-of-the-art model.The audio-edit experiments jointly fine-tune adding, dropping, and super-resolution tasks.
- Speech dereverberation: UniAudio achieves state-of-the-art speech dereverberation performance by DNSMOS with approximately 100 hours of simulated training data.
- Ablations and continual task addition: Multi-task training improves performance across all evaluated tasks compared with single-task training, while adding new tasks does not significantly reduce prior-task performance.
C.3 THE INFLUENCE OF DATA QUANTITY
The data-quantity study examines how dataset scale affects UniAudio, while the discussion relates its gains to unified tokenization, shared audio information, and cross-task augmentation.
- The influence of data quantity: UniAudio is evaluated with all, one-half, and one-quarter of the training data for each task.
- The influence of data quantity: The paper identifies data quantity as a key factor in building a strong audio foundation model.It proposes exploring additional unlabeled data in future work.
- Potential explanations: Unified tokenization maps modalities into a deterministic fixed vocabulary that supports next-token prediction and learning cross-modal relationships.
- Potential explanations: The authors explain multi-task gains partly through shared information across speech, sounds, music, and singing captured by neural audio codecs.
- Potential explanations: Multi-task training is also framed as data augmentation because related tasks can encode overlapping phonetic information.
- Codec design: The universal neural codec replaces an MS-STFT discriminator with a multi-scale Mel-based discriminator using mel- and log-mel-spectrogram features.
F SUBJECTIVE EVALUATION
Subjective evaluations use task-specific criteria for speech, singing, sound, and music generation, with rater instructions designed to isolate the intended quality, identity, style, or relevance dimension. Tests were crowd-sourced from 20 native speakers through Amazon Mechanical Turk, with instructions shown in Figure 6.
- Speech evaluation: TTS and VC evaluations measure speech quality with QMOS and speaker similarity with SMOS.Raters score naturalness on a 1–5 Likert scale for speech quality and compare synthesized utterances with reference speech for speaker identity.
- Speech evaluation: For TTS and VC speaker-similarity tests, raters focus on timbre matching while ignoring content, grammar, and audio-quality differences.Each synthesized utterance is paired with a reference utterance to assess similarity to the target speaker.
- Speech evaluation: SE and TSE evaluations use explicit instructions asking raters to assess the generated speech, with detailed instructions referenced in Figure 6.The figure presents screenshots of the subjective-evaluation instructions.
- Singing evaluation: SVS evaluations measure quality MOS and style-similarity MOS, directing raters to compare timbre, emotion, and prosody with the reference.Raters are instructed to ignore differences in content, grammar, and audio quality when judging style similarity.
- Sound and music evaluation: Sound and music generation are evaluated for overall quality and relevance to the text input, following AudioGen and MusicGen protocols.The subjective tests used 20 native speakers recruited through Amazon Mechanical Turk and cost about $500 in compensation.