Source-linked AI summary
Fitting New Speakers Based on a Short Untranscribed Sample
Eliya Nachmani, Adam Polyak, Yaniv Taigman, Lior Wolf
TL;DR
Neural TTS had not yet realized the promise of fitting new speakers from short samples, and earlier fitting required transcripts. The paper trains an audio-to-speaker embedding network jointly with TTS, reporting that dynamic embeddings can at least match learned ones and that training on 85 speakers captures much population variation.
Problem
Method
A jointly trained fitting network maps an audio sample to a speaker embedding, with consistency losses aligning embeddings from reconstructed and target audio.
Source: §1 · p. 1Results
Dynamic embeddings captured on-the-fly at least matched learned embeddings, and training on VCTK85 captured much variation in the general population.
Source: §6 · p. 8Takeaways & Limitations
The authors conclude that identifiable voices can be captured from short samples without transcripts, including approximately from one second of speech with a simplified priming network.
Source: §6 · pp. 7–8The authors omit MOS results for fitted VoiceLoop models on Libri-rest and VoxCeleb because generated-sample quality was unsuitable for MOS experimentation.
Source: §5.2 · p. 6
Abstract
from arXiv · showhide
Learning-based Text To Speech systems have the potential to generalize from one speaker to the next and thus require a relatively short sample of any new voice. However, this promise is currently largely unrealized. We present a method that is designed to capture a new speaker from a short untranscribed audio sample. This is done by employing an additional network that given an audio sample, places the speaker in the embedding space. This network is trained as part of the speech synthesis system using various consistency losses. Our results demonstrate a greatly improved performance on both the dataset speakers, and, more importantly, when fitting new voices, even from very short samples.
1. Introduction
Neural TTS systems had limited ability to fit new speakers from short samples, and prior fitting required transcripts. The paper introduces a jointly trained audio-to-speaker fitting network and separately investigates priming in a single network.
1. Introduction
- VoiceLoop’s new-speaker fitting had lower quality than training-speaker synthesis, depended on fitting-data quantity, and required transcripts.The paper proposes transcript-free fitting from limited data.Source: §1 · p. 1
- The fitting network maps a new speaker’s audio sample to an embedding and is trained jointly with the TTS system.Training uses the sample’s target audio to obtain the embedding used to generate that same audio, with losses promoting reconstruction-embedding similarity and separation.Source: §1 · p. 1
- The experiments focus on VoiceLoop because it was the only multispeaker TTS system with an official implementation available to the authors.The authors describe their approach as applicable to any neural TTS system.Source: §1 · p. 1
- A separate priming investigation uses a single network that does not model individual speakers to mimic multiple voices.The experiments report that this approach can mimic a new speaker without an optimization process, using even a short sample.Source: §1 · p. 1
2. Previous Work
The section reviews limitations in speaker fitting and describes VoiceLoop, the architecture used in the paper. The proposed modification replaces its speaker-embedding lookup table with a network that derives an embedding from audio.
2. Previous Work
- Multispeaker neural TTS research did not focus on de-novo speakers, while single-speaker systems often used hours of carefully transcribed data.The paper evaluates on VoxCeleb, described as lower-quality and more uncontrolled and heterogeneous than other existing corpora.Source: §2 · p. 2
- VoiceLoop uses phonemes, produces World Vocoder features, and shares a shifting-buffer RNN architecture with this work.The authors note that World Vocoder features upper-bound the obtainable quality.Source: §2 · p. 2
- Earlier speaker fitting was demonstrated for VoiceLoop, but required lengthy optimization and was reported as unnecessary and suboptimal by this work.Conventional concatenative and statistical-parametric systems also require clean, transcribed samples, often tens of minutes long.Source: §2 · p. 2
2.1. The VoiceLoop Architecture
- VoiceLoop’s buffer is differentiable memory shared by the model’s networks; its components include three shallow networks, two lookup tables, and two projection matrices.The networks create a representation vector, update attention, and generate vocoder-feature audio frames; the speaker embedding is stored in a lookup table.Source: §2.1 · pp. 2–3
- Each forward pass computes attention context, adds a representation vector to the shifting buffer, and generates the next audio frame.Training minimizes output vocoder-feature MSE and uses teacher forcing with the correct previous frame.Source: §2.1 · p. 3
- The proposed VoiceLoop modification removes the speaker-embedding lookup table and uses an audio-input network to produce the speaker representation.This makes fitting part of the method, enabling fitting from a short sample without transcribed text.Source: §2.1 · p. 3
3. The Voice Constancy Phenomenon
This section examines whether a speaker embedding is necessary for multi-speaker synthesis and whether fitting requires optimization. It describes voice constancy and uses priming to continue generation in a sampled speaker’s voice.
3. The Voice Constancy Phenomenon
- The section tests whether one network can capture a new speaker from a very short sample without speaker conditioning or an optimization step.It is presented as an investigation of voice fitting principles, distinct from the paper’s main fitting method.Source: §3 · p. 3
- Voice constancy is the persistence of a generated voice across subsequent frames when speaker input is absent and prior generated frames are provided during training.The authors attribute this behavior to teacher forcing during training.Source: §3 · p. 3
- Priming feeds a short sample, typically 300 frames or 1,500 milliseconds, then continues generation with the primed buffer and new text.The procedure uses teacher forcing for the sample and does not reset the buffer before continuing.Source: §3 · p. 3
4. The Fitting Sub-Network
The prior priming-based fitting method required transcripts and did not outperform VoiceLoop’s baseline, so the authors introduce a network that maps an audio sample to a speaker embedding. They train it with synthesis and speaker-consistency losses, alongside a two-phase training procedure.
4. The Fitting Sub-Network
- The fitting network Ns maps an audio sample to an L2-normalized speaker embedding used by VoiceLoop; it has five convolutional layers followed by pooling and fully connected layers.The convolutional layers use 3 × 3 filters with 32 channels, and the fully connected layers each have size 256.Source: §4 · pp. 3–4
4.1. The Loss Term
- The MSE loss compares VoiceLoop’s output sequence with the ground-truth vocoder features for an audio sample.With teacher forcing, both sequences have the same length, and the output depends on both the audio-derived embedding and phoneme sequence.Source: §4.1 · p. 4
- A margin-based contrastive loss encourages embeddings of same-speaker samples to be similar and embeddings of different-speaker samples to be separated.Triplets are formed from same-speaker sample pairs and the first sample of the next pair in the batch; the margin is ∆=1 in all experiments.Source: §4.1 · p. 4
- A cycle loss compares the input audio’s embedding with the embedding of the audio generated by VoiceLoop, encouraging speaker constancy across input and output.The overall objective is a weighted combination of the MSE, contrastive, and cycle losses, with α = β = 10 in all experiments.Source: §4.1 · p. 4
4.2. Training Details
- Training proceeds in two phases: first with more noise and short sequences, then with less noise and longer sequences until convergence.The first phase uses noise SD 4.0, length 100, batch size 256, and 90 epochs; the second uses noise SD 2.0, sequences up to 1000 vocoder features, and batch size 30. Length-bucketed mini-batches reduce padding and computation, while an inference-only attention change yields a slight improvement.Source: §4.2 · p. 4
5. Experiments
The experiments compare the fitting method with VoiceLoop and priming, evaluating trained voices, unseen-speaker fitting, and the contribution of individual losses. The reported results favor the method on fitted-voice quality and show similar identifiability on trained voices.
5.1. Evaluation of Trained Voices
- On trained voices, our model produces higher quality than VoiceLoop on MOS and MCD while retaining similar identifiability despite using no explicit per-speaker embedding during training.Source: §5.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6
5.2. Fitting Experiments
- For 16 new VCTK16 speakers, our method shows a significant MOS gap over VoiceLoop when all available training samples are used.The evaluation fits each new speaker individually.Source: §5.2 · p. 6, Table 4 · p. 6
- On LibriSpeech, our method outperforms VoiceLoop in ROC identification; on VoxCeleb, unfiltered AUC is 0.72 for GT, 0.62 for VoiceLoop, and 0.67 for our method.VoxCeleb identification is low even for original clips, so results are also evaluated on speaker subsets stratified by quality.Source: §5.2 · pp. 6–7, Figure 2 · p. 7
- Our method is preferable to VoiceLoop across all fitting-sample lengths; its best quality results use short samples, while priming yields good quality but lower identifiability.VoiceLoop fitting involves a lengthier backpropagation process; the authors suggest the short-sample quality result may reflect the sample lengths used to train Ns.Source: §5.2 · p. 7
5.3. Ablation Analysis
- Removing either Lcycle or Lcontrast significantly reduces fitted-voice identification accuracy, and removing Lcontrast also reduces accuracy for trained voices; Lcycle helps separate unseen voices in embedding space.Source: §5.3 · p. 7
5.4. Audio Samples
- The project webpage provides generated VCTK16 samples grouped by fitting-sample length, plus samples fitted from VoxCeleb and LibriSpeech-rest.Source: §5.4 · p. 7
6. Conclusions
The conclusions report feed-forward voice fitting from very short, uncontrolled samples and findings about speaker representation and training-population size. A priming-based demonstration further suggests that a single network can mimic a speaker from a short sample without optimization.
6. Conclusions
- The method fits voices feed-forward from very short samples in uncontrolled (“in-the-wild”) datasets.The authors say this brings neural TTS closer to its promise, while noting the advancement was shown in a specific architecture.Source: §6 · p. 7
- Identifiable voices can be captured from short samples without transcripts, and a simplified priming network can approximately replicate a speaker from one second of speech.The authors attribute this capability to a priming operator and the voice constancy property.Source: §6 · pp. 7–8
- Training on VCTK85 and fitting on datasets with different characteristics and many more speakers suggests that 85 training speakers capture much of the general population’s variation.Source: §6 · p. 8
- On-the-fly dynamic embeddings can at least match learned embeddings, and the authors’ ablation analysis attributes this result to the incorporated losses.Source: §6 · p. 8
- The Lcycle loss may support semi-supervised TTS training with a mix of transcribed and untranscribed speakers, including across languages without suitable transcribed corpora; this remains future work.Source: §6 · p. 8