PaperlayerView exact revision on arXiv

Source-linked AI summary

XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System

Peiling Lu, Jie Wu, Jian Luan, Xu Tan, Li Zhou

arXiv:2006.06261v1eess.AScs.CLcs.LGcs.SD

TL;DR

Singing voice synthesis must model spectrum, F0, and duration to produce quality, expressive, rhythmic singing, yet prior systems often neglected F0 and duration or trained these models independently. XiaoiceSing integrates these predictions in a modified FastSpeech system with musical-score inputs, residual F0 prediction, and syllable-duration loss. It outperforms the baseline on three MOS measures, and listeners prefer its F0 and duration methods in A/B tests.

  • Problem

    Prior SVS efforts often emphasized spectral features while giving less attention to F0 and duration, and independent models ignored consistency and musical-feature dependencies.

    Source: §1 · p. 1

  • Method

    XiaoiceSing modifies FastSpeech to jointly model spectrum, F0, and duration using score features, a residual note-pitch connection, and syllable-duration loss.

    Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2

  • Results

    XiaoiceSing gains 1.44 MOS on sound quality, 1.18 on pronunciation accuracy, and 1.38 on naturalness over baseline; its F0 and duration methods receive 97.3% and 84.3% preference, respectively.

    Source: §3.3 · p. 3, §3.4 · p. 4, §3.5 · p. 4

  • Takeaways & Limitations

    The authors conclude that the integrated model ensures consistency among predicted acoustic features.

    Source: §4 · p. 4

Abstract

from arXiv · show

This paper presents XiaoiceSing, a high-quality singing voice synthesis system which employs an integrated network for spectrum, F0 and duration modeling. We follow the main architecture of FastSpeech while proposing some singing-specific design: 1) Besides phoneme ID and position encoding, features from musical score (e.g.note pitch and length) are also added. 2) To attenuate off-key issues, we add a residual connection in F0 prediction. 3) In addition to the duration loss of each phoneme, the duration of all the phonemes in a musical note is accumulated to calculate the syllable duration loss for rhythm enhancement. Experiment results show that XiaoiceSing outperforms the baseline system of convolutional neural networks by 1.44 MOS on sound quality, 1.18 on pronunciation accuracy and 1.38 on naturalness respectively. In two A/B tests, the proposed F0 and duration modeling methods achieve 97.3% and 84.3% preference rate over baseline respectively, which demonstrates the overwhelming advantages of XiaoiceSing.

1. Introduction

XiaoiceSing adapts FastSpeech for singing by integrating spectrum, F0, and duration modeling with musical-score inputs. It addresses prior systems’ limited treatment of F0 and rhythm and their independently trained models.

1. Introduction

  • SVS aims to generate expressive, rhythmic singing from musical scores, requiring spectrum, F0, and duration models for sound quality, pitch contours, and rhythm.Source: §1 · p. 1
  • Earlier SVS work often focused on spectral features, while F0 and duration’s dynamic and rhythmic effects received less attention.The cited approaches include unit concatenation, statistical parametric synthesis, and deep-learning acoustic models.Source: §1 · p. 1
  • Prior F0 and duration strategies used post-processing, while independently trained spectrum, F0, and duration models left consistency and musical-feature dependencies unmodeled.The introduction notes that smoothing F0 can neglect its potential interactions with spectral features.Source: §1 · p. 1
  • XiaoiceSing jointly trains spectrum, F0, and duration models in a modified FastSpeech system using musical-score inputs, residual F0 prediction, and syllable-duration loss.It predicts WORLD vocoder features and synthesizes singing with WORLD.Source: §1 · p. 1

2. Architecture

XiaoiceSing uses a modified FastSpeech network to turn musical-score features into acoustic features and a WORLD-generated waveform. Its encoder, duration predictor, and decoder share information across the integrated system.

2. Architecture

  • The modified FastSpeech architecture encodes musical scores, predicts phoneme durations to expand the sequence, decodes acoustic features, and uses WORLD to generate waveform.Source: §2 · pp. 1–2

2.1. Musical score encoder

  • Each phoneme’s encoder input combines its phoneme ID, note pitch, note duration, and position encoding after separate embeddings.Lyrics are converted to phonemes; pitch is represented by MIDI-based IDs and note duration by frame counts.Source: §2.1 · p. 2

2.2. Duration predictor

  • The duration predictor shares encoder output with spectrum and F0 prediction, and its loss adds syllable-duration error to phoneme-duration error.Syllable duration compares the ground-truth syllable length with the summed predicted durations of its phonemes; the two losses have corresponding weights.Source: §2.2 · p. 2

2.3. Decoder

  • A residual connection from input note pitch to predicted logF0 makes the decoder predict pitch bias; binary logistic regression handles V/UV decisions.The decoder loss includes weighted logF0 and V/UV decision losses.Source: §2.3 · pp. 2–3

3. Experiments

Experiments compare XiaoiceSing with a CNN-based baseline using independent LSTM F0 and duration models, across MOS, objective metrics, and targeted listening and contour evaluations. XiaoiceSing performs better on the reported overall measures and on the F0 and duration comparisons.

3.1. Dataset

  • The dataset comprises 10,761 segments, about 74 hours, from 2,297 Mandarin pop songs by one female singer; 9,685 segments were selected for training.The remaining 1,076 segments were used for validation and test.Source: §3.1 · p. 3

3.2. System configuration

  • The baseline combines a CNN spectral model with separate LSTM-based F0 and duration models, while XiaoiceSing uses a modified FastSpeech architecture to predict all three together.XiaoiceSing’s encoder and decoder each stack six FFT blocks, and its duration predictor is jointly trained.Source: §3.2 · p. 3

3.3. Overall performance

  • XiaoiceSing outperforms baseline in all three aspects, with MOS gains of 1.18 on pronunciation accuracy, 1.44 on sound quality, and 1.38 on naturalness.Source: §3.3 · p. 3
  • XiaoiceSing achieves lower RMSE and higher CORR than baseline for duration and F0, and lower MCD, BAPD, and V/UV error.The authors conclude it generates more accurate phoneme duration, F0, and spectral features.Source: §3.3 · p. 3, Table 2 · p. 3

3.4. Evaluation on F0 modeling

  • 97.3% supports XiaoiceSing over baseline in the F0 preference test; its predicted contour also better preserves dynamics such as vibratos and is closer to the recording.The test asked listeners to judge pitch similarity against a recording reference.Source: §3.4 · p. 4, Figure 5 · p. 4, Figure 6 · p. 4

3.5. Evaluation on duration modeling

  • 84.3% supports XiaoiceSing over baseline in the duration preference test, and its phoneme durations are closer to the recording than the separate LSTM model’s.The LSTM model predicts a very long duration for the last phoneme and has greater accumulated errors.Source: §3.5 · p. 4, Figure 5 · p. 4, Figure 7 · p. 4

4. Conclusions

XiaoiceSing combines stacked CNN and attention networks to model long-range dependencies and keep predicted acoustic features consistent. It outperforms the baseline in sound quality, pronunciation accuracy, and naturalness, with F0 and duration gains linked to specific modeling changes.

  • XiaoiceSing outperforms the baseline in sound quality, pronunciation accuracy, and naturalness.The authors associate outstanding F0 perception with the residual connection between note pitch and predicted F0, and duration improvements with the syllable-duration constraint.Source: §4 · p. 4
  • Stacked CNN and attention networks better model remote dependencies along the time axis.Source: §4 · p. 4
  • The integrated model ensures consistency among predicted acoustic features.Source: §4 · p. 4
Loading 2006.06261v1…