Source-linked AI summary
SoulX-Singer: Towards High-Quality Zero-Shot Singing Voice Synthesis
Jiale Qian, Hao Meng, Tian Zheng, Pengcheng Zhu, Haopeng Lin, Yuhang Dai, Hanke Xie, Wenxiao Cao, Ruixuan Shang, Jun Wu, Hongmei Liu, Hanlin Wen, Jian Zhao, Zhonglin Jiang, Yong Chen, Shunshun Yin, Ming Tao, Jianguo Wei, Lei Xie, Xinsheng Wang
TL;DR
Open-source singing voice synthesis still lacks robust zero-shot generalization and flexible control for practical workflows. SoulX-Singer addresses this with over 42,000 hours of aligned multilingual data, a flow-matching architecture supporting score and melody conditioning, and dedicated evaluation resources. Across benchmarks, it consistently outperforms state-of-the-art baselines in pitch accuracy, intelligibility, timbre similarity, and overall singing quality, while zero-shot use raises impersonation and misuse risks.
Problem
High-quality, flexible zero-shot singing voice generation remains largely unavailable, while existing melody-driven systems lack score-only generation and explicit note-duration control.
Method
SoulX-Singer combines over 42,000 hours of aligned multilingual vocals with a flow-matching DiT model supporting both musical-score and melody-conditioned synthesis.
Results
SoulX-Singer consistently outperforms state-of-the-art baselines in pitch accuracy, intelligibility, timbre similarity, and overall singing quality across multiple benchmarks.
Takeaways & Limitations
The system provides a foundation for personalized singing synthesis, music production, and multilingual vocal content generation.
Takeaways & Limitations
Zero-shot generation from short reference prompts may enable voice impersonation and misuse, requiring respect for intellectual property, privacy, and consent.
Abstract
from arXiv · showhide
While recent years have witnessed rapid progress in speech synthesis, open-source singing voice synthesis (SVS) systems still face significant barriers to industrial deployment, particularly in terms of robustness and zero-shot generalization. In this report, we introduce SoulX-Singer, a high-quality open-source SVS system designed with practical deployment considerations in mind. SoulX-Singer supports controllable singing generation conditioned on either symbolic musical scores (MIDI) or melodic representations, enabling flexible and expressive control in real-world production workflows. Trained on more than 42,000 hours of vocal data, the system supports Mandarin Chinese, English, and Cantonese and consistently achieves state-of-the-art synthesis quality across languages under diverse musical conditions. Furthermore, to enable reliable evaluation of zero-shot SVS performance in practical scenarios, we construct SoulX-Singer-Eval, a dedicated benchmark with strict training-test disentanglement, facilitating systematic assessment in zero-shot settings.
1 Introduction
SoulX-Singer targets high-quality zero-shot singing voice synthesis with flexible control from either musical scores or melodies. The paper also contributes large-scale multilingual data processing and a dedicated benchmark for systematic zero-shot evaluation.
- High-quality zero-shot singing generation remains largely unavailable despite progress in speech synthesis and music generation.
- Prior melody-driven systems require extracted melodies and lack explicit note-duration control, limiting score-only generation and syllable-level timing alignment.These limitations constrain practical use in mixing and arrangement workflows.
- Over 42,000 hours of multilingual singing data support SoulX-Singer’s score-based and melody-conditioned generation modes.The design targets symbolic-score workflows and melody-guided generation from existing songs.
- SoulX-Singer unifies music-score and melody inputs while supporting timbre cloning, reference-based style transfer, and editing of scores and lyrics.
- SoulX-Singer-Eval provides 50 unseen Mandarin and English singers with note-level annotations for standardized zero-shot evaluation.The benchmark assesses synthesis quality, controllability, and generalization.
2 Method
The method builds aligned multilingual singing data through vocal extraction and note-level annotation, then synthesizes vocals with a flow-matching model using lyrics and melodic or score cues. Two inference modes and staged training provide flexible control across melody-guided and score-driven workflows.
- Data Processing: The data pipeline separates vocals, transcribes lyrics, detects notes, and extracts F0 to create aligned training inputs.The workflow converts mixed songs into clean vocal recordings paired with lyrics, MIDI, and melodic representations.
- Corpus Overview: The corpus contains approximately 42,000 hours of vocals: about 20k each in Mandarin and English and 2k in Cantonese.Each musical note is represented using its text token, pitch class, and note type.
- Model Architecture: SoulX-Singer uses a non-autoregressive flow-matching decoder built with DiT to predict mel-spectrograms from lyrics and melody cues.A neural vocoder subsequently converts predicted mel-spectrograms into waveforms.
- Feature Representation: The singing content encoder integrates lyrics, musical scores, note types, and F0 into temporally aligned representations for mel-spectrogram generation.
- Feature Integration: A length regulator expands note pitch, note type, and text embeddings according to note durations before decoding.
- Training Strategy: Two-stage training progresses from short non-adjacent prompts to 30–90-second adjacent segments for robust conditional and long-form generation.The first stage uses 2–16-second segments, while the second models longer-range temporal dependencies.
- Inference Modes: Melody-control mode uses lyrics and reference F0, whereas score-control mode uses only MIDI and lyrics to predict constrained acoustic features.The two modes support expressive melody preservation and creative score-driven synthesis, respectively.
3 Performance of SoulX-Singer
SoulX-Singer is evaluated on complementary open-source and zero-shot benchmarks using measures of pitch, timbre, intelligibility, and singing quality. It outperforms baseline systems across control modes, unseen speakers, lyric editing, and cross-lingual synthesis.
- Evaluation metrics: The evaluation measures melodic accuracy, timbre similarity, intelligibility, and overall singing quality using FFE, SIM, WER, SingMOS, and Sheet-SSQA.FFE measures pitch deviations exceeding 20% from ground-truth F0, while WER compares target lyrics with ASR transcriptions.
- GMO-SVS performance: SoulX-Singer outperforms all baselines on GMO-SVS in both Mandarin and English across its evaluated singing synthesis settings.Melody-control mode achieves the lowest FFE, while score-control mode achieves the lowest WER; the model also reaches state-of-the-art SingMOS and SIM.
- GMO-SVS performance: MIDI score-based control preserves pronunciation accuracy and rhythmic consistency better when lyrics are rewritten than melody-based methods.Melody-based systems show a noticeable intelligibility drop under lyric modification, whereas explicit score control accommodates rewritten lyrics.
- Zero-shot and cross-lingual performance: 0.922 SIM for Mandarin and 0.914 SIM for English are the highest scores on SoulX-Singer-Eval for music-score-controlled synthesis.These results are reported for speakers unseen during training and demonstrate robust zero-shot voice cloning.
- Zero-shot and cross-lingual performance: 0.110 WER with 0.898 SIM shows that SoulX-Singer preserves speaker identity while synthesizing across languages.The reported cross-lingual results contrast with Vevosing’s WER of 0.717 and are attributed to separation of language-independent timbre from linguistic content.
4 Conclusions
SoulX-Singer combines over 42,000 hours of note-aligned vocal data with melody- and score-based control for zero-shot singing synthesis. Evaluations report consistent improvements over state-of-the-art baselines across pitch accuracy, intelligibility, timbre similarity, and singing quality.
- Conclusions: SoulX-Singer is a high-fidelity zero-shot singing voice synthesis model trained with over 42,000 hours of note-level aligned data.The system supports flexible control over melody and score while reproducing diverse vocal timbres.
- Conclusions: SoulX-Singer consistently outperforms state-of-the-art baselines in pitch accuracy, intelligibility, timbre similarity, and overall singing quality.The reported evaluations span multiple benchmarks and support applications including personalized singing synthesis, music production, and multilingual vocal content generation.
5 Ethics Statement
SoulX-Singer’s realistic zero-shot singing generation introduces ethical risks, especially voice impersonation and misuse. The authors emphasize respecting intellectual property, privacy, and personal consent.
- Realistic singing voices generated from short reference prompts may enable voice impersonation and misuse.
- Users should respect intellectual property, privacy, and personal consent when generating singing content.
- The system should not impersonate individuals without authorization or produce deceptive or misleading audio.
SoulX-Singer-SVC
SoulX-Singer-SVC extends SoulX-Singer to singing voice conversion when precise MIDI and time-aligned lyrics are unavailable. It uses Whisperbase-based semantic representations while retaining pretrained components for melody control and synthesis quality, and is evaluated across two datasets.
- SoulX-Singer-SVC replaces the original score and text encoders with a frozen Whisperbase encoder for semantic extraction from source singing audio.The F0 encoder and flow-matching decoder are initialized from pretrained SoulX-Singer parameters.
- The model is evaluated on GMO-SVS and SoulX-Singer-Eval against GroundTruth, melody-controlled SoulX-Singer, YingMusic-SVC, and Vevo-SVC.
- FFE is omitted for SoulX-Singer-Eval because its source audio was pitch-shifted, while the model is released as part of the SoulX-Singer project.
- Table 4 reports singing voice conversion performance on two datasets, with larger-is-better and smaller-is-better metric directions indicated.Best and second-best results are distinguished by bold and underlining.