Source-linked AI summary
SingOMD: Singing Oriented Multi-resolution Discrete Representation Construction from Speech Models
Yuxun Tang, Yuning Wu, Jiatong Shi, Qin Jin
TL;DR
Speech SSL models provide useful discrete representations, but singing lacks dedicated SSL models and requires richer, multi-resolution features because of its distinct acoustic characteristics. SingOMD adapts speech SSL features through singing resynthesis, generates multi-resolution features with resampling, and discretizes them by clustering. The resulting representations are robust for singing vocoders and enable singing voice synthesis comparable to mel-spectrogram systems while outperforming a discrete baseline without auxiliary information.
Problem
Speech-derived discrete representations face a domain gap for singing, while singing requires richer multi-resolution representations and lacks dedicated singing SSL models.
Source: §1 · p. 1Method
Results
SingOMD achieves comparable singing voice synthesis results to mel-spectrogram systems and outperforms a discrete baseline using a larger model and pitch predictor without auxiliary information.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Discrete representation has shown advantages in speech generation tasks, wherein discrete tokens are derived by discretizing hidden features from self-supervised learning (SSL) pre-trained models. However, the direct application of speech SSL models to singing generation encounters domain gaps between speech and singing. Furthermore, singing generation necessitates a more refined representation than typical speech. To address these challenges, we introduce SingOMD, a novel method to extract singing-oriented multi-resolution discrete representations from speech SSL models. Specifically, we first adapt the features from speech SSL through a resynthesis task and incorporate multi-resolution modules based on resampling to better serve singing generation. These adapted multi-resolution features are then discretized via clustering. Extensive experiments demonstrate the robustness, efficiency, and effectiveness of these representations in singing vocoders and singing voice synthesis.
1. Introduction
Singing voice synthesis is moving toward discrete representations, but speech-derived units face domain and resolution gaps because singing has richer pitch variation, broader frequencies, and longer durations. SingOMD addresses these gaps by adapting speech SSL features through resynthesis and constructing multi-resolution singing-oriented discrete tokens.
1. Introduction
- Discrete representations offer reduced storage, improved training efficiency, and potential compatibility with other modalities, motivating their exploration in singing voice synthesis.The Interspeech 2024 Challenge introduced an SVS track centered on cascaded discrete systems.Source: §1 · p. 1
- Discrete-unit research is established in speech generation, where intermediate features from speech SSL models are commonly clustered into downstream representations [5–15],.These representations support tasks including TTS, speech-to-speech translation, and speech enhancement.Source: §1 · p. 1
- Singing requires richer and more expressive discrete representations than speech because it contains nuanced pitch variation, broader vocal frequencies, and longer durations.A single resolution is also described as suboptimal for singing-related tasks.Source: §1 · p. 1
- SingOMD adapts speech SSL features for singing through resynthesis, adds multi-resolution processing, and discretizes the resulting features with clustering.The method is designed for singing generation without modifying speech SSL model parameters or structures.Source: §1 · p. 1
- SingOMD achieves comparable singing voice synthesis results to mel-spectrogram systems and outperforms a discrete baseline that uses a larger model and a pitch predictor without auxiliary information.This result is reported as a main contribution of the paper.Source: §1 · p. 2
2. Method
SingOMD converts speech SSL features into singing-adapted multi-resolution representations through resynthesis, resampling, and clustering. Its pipeline combines weighted hidden-layer features, symmetric U-Net-style resolution changes, vocoder reconstruction, and per-resolution K-means discretization.
2. Method
- SingOMD uses a resynthesis task to adapt speech SSL features to singing, then clusters continuous multi-resolution features into singing-oriented discrete representations.The resynthesis process serves as an adapter between speech features and the singing domain.Source: §2 · pp. 2–3
Speech Self-supervised Learning Model
- Weighted sums of intermediate speech SSL layers form the input features, with trainable weights assigned to each hidden-layer feature.The features are represented as s = Pn i=1 wi · si, where wi is the trainable weight for si.Source: §2 · p. 2
Resampling Module
- The resampling module uses a transfer encoder followed by symmetric downsampling and upsampling stages to produce features at multiple resolutions.Residual connections combine upsampled features with corresponding downsampled features during reconstruction of the resolution hierarchy.Source: §2 · p. 2
- Desired sampling-rate sequences determine symmetric resampling ratios; for [20ms, 40ms, 80ms], adjacent ratios are both 2.A 20ms token duration corresponds to 50 features per second in the stated example.Source: §2 · p. 2
Vocoder and Discrete Representations
- A HiFi-GAN vocoder reconstructs singing waveforms from the resampled features, while the training objective combines adversarial, feature-matching, and mel-spectrogram losses.K-means is then applied independently to continuous features at the desired resolutions.Source: §2 · pp. 2–3
3. Experiments
Experiments evaluate SingOMD across singing resynthesis and singing voice synthesis using Opencpop, objective and subjective metrics, and discrete-token baselines. SingOMD benefits from multi-resolution features and achieves the strongest discrete-token SVS results, with performance comparable to mel-spectrogram systems.
3.2. Evaluation Dataset and Metrics
- Evaluation measures generated audio with MCD, F0 RMSE, semitone accuracy, V/UV error, and MOS from 20 professional annotators rating 30 samples per system.Experiments cover singing resynthesis and singing voice synthesis, using Opencpop as the benchmark dataset.Source: §3 · p. 3, §3.2 · p. 3
3.1. SingOMD Training
- SingOMD is trained on ACE-Opencpop (130 hours), OpenSinger (50 hours), M4Singer (29.8 hours), and Opencpop (5.2 hours), totaling approximately 210 hours, using HuBERT-base features, resampling modules, and 1024-cluster K-means discretization.Three resolutions are set at most, and Opencpop serves as the benchmark for evaluating the quality of SingOMD tokens.Source: §3.1 · p. 3, §3.2 · p. 3
3.4. Experiment results on singing resynthesis
- In singing resynthesis, adding multi-resolution information substantially improves all metrics over a single-resolution configuration, whereas simply increasing token streams does not.The comparison uses unit HiFi-GAN vocoders and includes HuBERT-based single- and multi-stream baselines.Source: §3.4 · p. 3, §3.3 · p. 3, Table 1 · p. 4
- Increasing resolutions generally improves or preserves MCD, F0 RMSE, semitone accuracy, and MOS, but yields diminishing gains and a trade-off with token quantity and computational efficiency.V/UV error shows no apparent correlation with resolution.Source: §3.4 · p. 4
3.5. Evaluation setup on singing voice synthesis
- The SVS cascade trains a discrete-token vocoder and an acoustic model that predicts tokens from musical scores, without auxiliary information for discrete-token systems.The comparison includes RNN-based discrete-token models and XiaoiceSing with mel-spectrograms.Source: §3.5 · p. 4, Table 2 · p. 4
3.6. Experiment results on singing voice synthesis
- SingOMD achieves the best grades among discrete-token systems across all SVS metrics and comparable MOS to the mel-spectrogram baseline.Its F0 RMSE improves notably, while MCD deteriorates plausibly because discretization loses information; a gap remains from Ground-Truth.Source: §3.6 · p. 4, Table 2 · p. 4
4. Conclusion
SingOMD constructs singing-oriented multi-resolution discrete representations from speech SSL models. It adapts speech features through singing resynthesis, adds resampling-based multi-resolution features, and discretizes them for singing vocoders and singing voice synthesis.
4. Conclusion
- SingOMD constructs singing-oriented discrete representations from speech SSL models by adapting features through resynthesis and discretizing multi-resolution features with K-means.The representations demonstrate robustness in singing vocoders and improve singing voice synthesis efficiency and effectiveness when integrated with a discrete acoustic model.Source: §4 · p. 4