Source-linked AI summary
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, Zhou Zhao
TL;DR
High-fidelity multi-singer singing synthesis is limited by data scarcity, computational cost, and weak unseen-singer generalization. The paper releases OpenSinger and proposes the GAN-based Multi-Singer vocoder; results report improved speed and quality on unseen singers and robustness with FastSpeech 2.
Problem
High-fidelity multi-singer singing synthesis is challenged by scarce high-quality data, computational cost, and degraded generation quality for unseen singers.
Source: §1 · p. 1Method
The paper releases OpenSinger and proposes a GAN vocoder combining a multi-band generator, singer-conditional discriminator and adversarial objective, and singer perceptual loss.
Source: §1 · p. 2, §4.2 · p. 5, §4.5 · p. 6Results
Takeaways & Limitations
OpenSinger provides a large-scale, multi-singer Chinese singing voice corpus intended to accelerate singing voice synthesis research.
Source: §6 · p. 8Speaker embeddings used as additional input can be hard to obtain during inference and increase computation, reducing inference speed.
Source: §4.1.2 · p. 5
Abstract
from arXiv · showhide
High-fidelity multi-singer singing voice synthesis is challenging for neural vocoder due to the singing voice data shortage, limited singer generalization, and large computational cost. Existing open corpora could not meet requirements for high-fidelity singing voice synthesis because of the scale and quality weaknesses. Previous vocoders have difficulty in multi-singer modeling, and a distinct degradation emerges when conducting unseen singer singing voice generation. To accelerate singing voice researches in the community, we release a large-scale, multi-singer Chinese singing voice dataset OpenSinger. To tackle the difficulty in unseen singer modeling, we propose Multi-Singer, a fast multi-singer vocoder with generative adversarial networks. Specifically, 1) Multi-Singer uses a multi-band generator to speed up both training and inference procedure. 2) to capture and rebuild singer identity from the acoustic feature (i.e., mel-spectrogram), Multi-Singer adopts a singer conditional discriminator and conditional adversarial training objective. 3) to supervise the reconstruction of singer identity in the spectrum envelopes in frequency domain, we propose an auxiliary singer perceptual loss. The joint training approach effectively works in GANs for multi-singer voices modeling. Experimental results verify the effectiveness of OpenSinger and show that Multi-Singer improves unseen singer singing voices modeling in both speed and quality over previous methods. The further experiment proves that combined with FastSpeech 2 as the acoustic model, Multi-Singer achieves strong robustness in the multi-singer singing voice synthesis pipeline. Samples are available at https://Multi-Singer.github.io/
1 INTRODUCTION
The introduction frames high-fidelity multi-singer singing synthesis as constrained by limited data, computational cost, and poor generalization to unseen singers. It presents OpenSinger and Multi-Singer as responses, reporting strong quality and speed results.
1 INTRODUCTION
- Multi-singer SVS faces limited access to high-quality open singing data, generation cost constraints, and degraded quality for unseen singers.Differences in timbre, expression, and style complicate modeling across singers.Source: §1 · p. 1
- Existing web-mined or released corpora lack the quality or scale required, while prior multi-band methods do not adapt to frequency-band characteristics.Prior singer-conditioning strategies can also require hard-to-obtain embeddings and slow inference.Source: §1 · pp. 1–2
- The authors introduce OpenSinger, an open-source, large-scale, multi-singer Chinese singing voice corpus.They describe it as the first open-source Chinese singing voice dataset and say the full dataset will be released after publication.Source: §1 · p. 2
- Multi-Singer combines a frequency-adapted multi-band generator, a singer-conditional discriminator with conditional adversarial training, and an auxiliary singer perceptual loss.The perceptual loss uses speaker-encoder embeddings to supervise singer representation reconstruction in the frequency domain.Source: §1 · p. 2
- Multi-Singer achieves the best mean opinion score among existing neural vocoders and runs 125 times faster than real time on one NVIDIA 2080Ti.It maintains comparable quality to an autoregressive counterpart and shows strong robustness with FastSpeech 2 as the acoustic model.Source: §1 · p. 2
2 RELATED WORKS
The related-work section surveys datasets, singing voice synthesis systems, and vocoders, emphasizing gaps in singing-data coverage and unseen-singer generalization.
2.1 Dataset
- Speech and singing synthesis depend on annotated corpora, with speech datasets spanning varied tasks and collection settings.Examples include LSSED, AISHELL-3, LibriSpeech, and the studio-recorded JVS-MuSiC corpus.Source: §2.1 · p. 2, §2.1.1 · p. 2
2.1.2 Singing voices.
- Available singing datasets differ in annotation and suitability, while high recording costs and copyright constraints limit SVS data.MIR-1K lacks word-level segmentation, NUS-48E has 48 songs, and JukeBox's 467 hours of internet audio are unsuitable for high-quality SVS.Source: §2.1.2 · p. 2
- OpenSinger is proposed as an open-source, large-scale, multi-singer corpus because existing data do not meet the authors' quality and quantity requirements.Source: §2.1.2 · p. 2
2.2 Singing voice synthesis
- Earlier SVS systems generate natural singing, but their vocoders are not designed for multiple singers and degrade on unseen voices.The section describes systems using web-mined data, autoregressive spectrogram generation, and multi-scale adversarial training.Source: §2.2 · p. 2
2.3 Vocoder
- Autoregressive vocoders have long been used for singing waveform generation, while non-autoregressive approaches include distillation-based Parallel WaveNet and computationally costly WaveGlow.Source: §2.3 · pp. 2–3
- GAN vocoders avoid sample-by-sample causal inference or teacher models, but existing vocoders still show degradation when generalized to unseen singers.Examples reviewed include MelGAN, HiFi-GAN, and VocGAN.Source: §2.3 · p. 3
3 CONSTRUCTION OF OPENSINGER
OpenSinger is assembled as an open, large-scale Chinese multi-singer corpus from professional recordings, with processing that prepares utterances for analysis. Its statistics characterize singer diversity and differences between singing and speech.
3 CONSTRUCTION OF OPENSINGER
- OpenSinger is presented as an open-source, large-scale, high-quality Chinese multi-singer corpus recorded by professional singers in studios.The corpus description reports 50 hours from 93 singers, while its feature list identifies openness, scale, and recording quality as key attributes.Source: §3 · p. 3
3.1 Data Collection
- Ninety-three professional singers recorded Chinese traditional and pop songs, which a professional team labeled with singer, song, and reference-text information.The corpus account reports 80 hours recorded, and specifies WAV audio sampled at 24 kHz and quantized at 16 bits.Source: §3.1 · p. 3
3.2 Processing
- Processing segments songs into utterances of up to 11 seconds, trims non-vocal silences with manually verified VAD, and aligns lyrics and audio.Montreal Forced Aligner supplies phoneme annotations from manually labeled phone strings; the pipeline is intended to address long clips, silences, and alignment needs.Source: §3.2 · p. 3
3.3 Statistics
- Male and female sentence durations are similar, while phoneme durations for both groups mainly fall between 10 ms and 500 ms.The paper attributes small sentence-duration variations to manual segmentation.Source: §3.3 · pp. 3–4
- Female singing voices show a larger pitch range and more high-frequency content than male singing voices.Pitch distributions are reported to differ greatly between genders, alongside differences in timbre and pronunciation habits.Source: §3.3 · p. 4
- A UMAP projection of 10 utterances from each of 10 singers shows large inter-speaker distances, indicating singer diversity in OpenSinger.The singer representations are 256-dimensional embeddings reduced to two dimensions; colors distinguish singers.Source: §3.3 · p. 4, Figure 1 · p. 4
3.4 Speech and Singing voices
- Compared with CSMSC speech, singing has broader pitch and phoneme-duration distributions, with higher average pitch and phoneme-level duration.The authors state that these speech–singing characteristics add challenges for singing-voice synthesis research and corpus assembly.Source: §3.4 · p. 4, Table 2 · p. 4
4 MULTI-SINGER
Multi-Singer targets faster multi-singer vocoding and stronger unseen-singer modeling by adapting generation across frequency bands and explicitly supervising singer identity. Its generator, singer-conditional discriminator, and combined training losses support this approach.
4.1.1 Multi-band generartion.
- Frequency-adaptive multi-band generation addresses the speed and quality trade-off by modeling frequency bands separately.Earlier multi-band methods focused on speed and could limit singing-voice quality by not accounting for differences among bands.Source: §4.1.1 · p. 4
4.1.2 Multi-singer modeling.
- Multi-Singer aims to capture singer identity in mel-spectrograms without adding inference-time computation for singer embeddings.Prior multi-speaker vocoders showed degradation on unseen speakers, while embedding-based methods could be hard to obtain and slow inference.Source: §4.1.2 · pp. 4–5
4.2 Overview
- The approach combines a multi-band generator, a singer-conditional discriminator, and joint conditional-unconditional adversarial training with singer perceptual loss.The conditional discriminator guides reconstruction of singer identity; the auxiliary loss penalizes singer-identification bias.Source: §4.2 · p. 5
4.3 Generator
- The non-autoregressive generator models four waveform bands separately with frequency-adapted models, then merges their outputs through PQMF.The two models handle high- and low-frequency bands, with receptive fields and WaveNet-block counts adapted to their acoustic characteristics.Source: §4.3 · p. 5
4.4 Discriminator
- The singer-conditional discriminator judges whether generated voices preserve the singer identity, while the unconditional discriminator provides a separate adversarial signal.The architecture includes both conditional and unconditional discriminators.Source: §4.4 · p. 5
4.5 Training Loss
- Training combines conditional and unconditional adversarial losses with singer perceptual and multi-resolution STFT auxiliary losses.Singer perceptual loss uses speaker-encoder hidden states and frequency-domain mel-spectrograms to supervise singer-identity reconstruction; the STFT loss sums losses across analysis settings, and λ is set to 10.Source: §4.5 · pp. 5–6
5 EXPERIMENT
The experiments assess OpenSinger, Multi-Singer’s multi-band design and unseen-singer modeling, component ablations, and a FastSpeech 2 synthesis system. Results support the corpus and report Multi-Singer’s speed, quality, and system robustness.
5.1 Experimental Setup
- The seen-singer test set used 60 utterances from six singers, while the unseen-singer set used five utterances each from five male and five female singers.The experiments also used 60 Chinese speech test utterances from CSMSC.Source: §5.1 · pp. 6–7
5.2 Model training
- The speaker encoder was fine-tuned on OpenSinger; Multi-Singer training used 80-band mel-spectrograms and RAdam, with generator pretraining before joint training.The passage reports 40k training steps and says joint training followed the first 100k generator pretraining steps.Source: §5.2 · p. 7
5.3 Corpus Verification
- OpenSinger supported robust neural-vocoder training, while CSMSC speech pretraining produced a slight performance drop.The authors conclude that additional speech data is not needed for high-quality singing-voice modeling with OpenSinger.Source: §5.3 · p. 7, Table 3 · p. 7
5.4 Multi-band generation
- Multi-band WaveRNN achieved the best performance and naturalness but limited speed; multi-band MelGAN was fast but showed audio-quality degradation.The experiments compare MOS and real-time factor across multi-band generation methods; the authors report that Multi-Singer adjusts speed and quality.Source: §5.4 · p. 7, Table 4 · p. 7
5.5 Multi-singer modeling
- Multi-Singer is reported to improve unseen-singer modeling in both speed and quality, without extra generation computation for singer identity.The evaluation reports MOS, FDSD, and cosine similarity, but does not compare seen and unseen test sets because the added unseen-singer recordings are lower quality.Source: §5.5 · pp. 7–8, Table 5 · p. 8
5.6 Ablation study
- Removing the multi-band generator reduced generation speed and slightly reduced quality; removing the singer conditional discriminator lowered unseen-singer cosine similarity.The authors also state that removing Singer Perceptual Loss could weaken reconstruction of singer representations in the spectrum envelope.Source: §5.6 · p. 8, Table 6 · p. 8
5.7 Singing voice synthesis system
- Multi-Singer combined with FastSpeech 2 generates high-quality singing voices and shows strong robustness in the synthesis system.FastSpeech 2 converts lyrics, F0, duration, and singer embedding into mel-spectrograms, which Multi-Singer converts into singing voices.Source: §5.7 · p. 8, Table 7 · p. 8
6 CONCLUSION
The paper releases OpenSinger and proposes Multi-Singer, a fast vocoder for multi-singer singing-voice generation. Experiments report high-fidelity unseen-singer synthesis and robustness with a singing-voice synthesis system; generalization across emotions remains future work.
6 CONCLUSION
- OpenSinger is a large-scale, multi-singer Chinese singing-voice dataset released to support research.The authors describe it as the first Chinese open dataset directed toward high-fidelity singing-voice synthesis, to their knowledge.Source: §6 · p. 8
- Multi-Singer combines a fast vocoder with a singer conditional discriminator and singer perceptual loss to reconstruct singer identity.The paper reports that it speeds up generation and synthesizes high-fidelity singing voices of unseen singers.Source: §6 · p. 8
- The authors report strong robustness for Multi-Singer in a singing-voice synthesis system and identify generalization across emotions as future work.Source: §6 · p. 8
A STATISTICS
The passage distinguishes singing voices from speech by their expressive variation and spectral characteristics.
A STATISTICS
- Singing voices vary in expression and style.The passage notes that they include richer emotional information.Source: A STATISTICS · p. 10
- Singing voices include richer emotional information than speech.Source: A STATISTICS · p. 10
- Their usually high sampling rate produces a wider frequency-domain spectrogram band with more high-frequency parts.Source: A STATISTICS · p. 10
B.1 Mean Opinion Scores
The appendix describes the study’s subjective MOS procedure and two objective audio metrics: FDSD and cosine similarity.
B.1 Mean Opinion Scores
- MOS ratings were crowdsourced from native speakers, who assessed one sample at a time using the rubric in.Table 8 includes the scoring criteria.Source: §B.1 · p. 10, Table 8 · p. 10
B.2 Fréchet Deep Speech Distances
- FDSD measures synthetic-audio quality by distance to a reference set and is described as a proper scoring rule for generated-waveform spectrogram distributions [13].The passage says FDSD is conceptually similar to FID.Source: §B.2 · p. 10
B.3 Cosine similarity
- Cosine similarity between synthetic and ground-truth speaker embeddings measures speaker similarity; same-speaker utterances have higher similarity than different-speaker utterances.The metric is the cosine of the angle between embedding vectors.Source: §B.3 · p. 10
C SPEAKER ENCODER
The speaker encoder is used to extract representations of long-term speaker identity. It is trained to reduce within-speaker variation and separate different speakers.
C SPEAKER ENCODER
- A trained speaker encoder extracts audio representations that capture long-term speaker identity.Speaker verification checks speaker identity and whether representations come from related speakers.Source: C SPEAKER ENCODER · p. 10
- The encoder projects mel-spectrograms to 256-dimensional representations and uses GE2E for speaker-verification training.GE2E is described as more efficient to train than the earlier TE2E loss.Source: C SPEAKER ENCODER · p. 10
- During training, the encoder reduces intra-speaker variance and separates different speakers.Source: C SPEAKER ENCODER · p. 10
D MULTI-RESOLUTION STFT LOSS DETAILS
The appendix gives details of the multi-resolution STFT loss, the generator’s WaveNet block, and the modified FastSpeech 2-to-vocoder pipeline.
D MULTI-RESOLUTION STFT LOSS DETAILS
- Table 9 summarizes the multi-resolution STFT loss details and specifies a Hanning window before the FFT.Source: Table 9 · p. 10
E WAVENET BLOCK
- In the generator’s WaveNet block, noise passes through dilated convolutions while noise and mel features are split, processed, and passed onward as H and X.The processed features pass through sigmoid-tanh calculations and two fully connected networks before reaching the next dilated convolution layer.Source: E WAVENET BLOCK · p. 10
F MODIFIED FASTSPEECH 2
- Modified FastSpeech 2 converts lyrics, F0, duration, and singer embedding into a mel-spectrogram, which Multi-Singer converts into singing voice.Source: F MODIFIED FASTSPEECH 2 · p. 10