PaperlayerView exact revision on arXiv

Source-linked AI summary

VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop

Yaniv Taigman, Lior Wolf, Adam Polyak, Eliya Nachmani

arXiv:1707.06588v3cs.LGcs.CLcs.SD

TL;DR

VoiceLoop addresses voice imitation from unconstrained, in-the-wild recordings, where noise and speaker variation complicate training. It uses a shared shifting buffer for attention, speech generation, and memory updates, with speaker embeddings that can be fitted after training. The authors report robustness to noisy public-speech data and reasonable quality and speaker discrimination for fitted identities.

  • Problem

    Voice imitation from in-the-wild samples is limited by reliance on carefully collected or curated audio, despite the availability of noisy public speeches with multiple speakers and inaccurate transcripts.

    Source: §1 · p. 1

  • Method

    VoiceLoop uses a shared shifting buffer for attention, output generation, and recursive updates, with phoneme-vector inputs and speaker embeddings that can be fitted for new identities.

    Source: §1 · pp. 1–2, §3 · p. 5

  • Results

    VoiceLoop is reported as mostly robust to clapping, laughter, dramatic prosody, and pauses without reproducing background noise, unlike the Char2Wav baseline on the YouTube dataset.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The method demonstrates multi-speaker TTS from unconstrained public-speech samples and post-training fitting of new speakers, including with very limited data.

    Source: §6 · p. 13

Abstract

from arXiv · show

We present a new neural text to speech (TTS) method that is able to transform text to speech in voices that are sampled in the wild. Unlike other systems, our solution is able to deal with unconstrained voice samples and without requiring aligned phonemes or linguistic features. The network architecture is simpler than those in the existing literature and is based on a novel shifting buffer working memory. The same buffer is used for estimating the attention, computing the output audio, and for updating the buffer itself. The input sentence is encoded using a context-free lookup table that contains one entry per character or phoneme. The speakers are similarly represented by a short vector that can also be fitted to new identities, even with only a few samples. Variability in the generated speech is achieved by priming the buffer prior to generating the audio. Experimental results on several datasets demonstrate convincing capabilities, making TTS accessible to a wider range of applications. In order to promote reproducibility, we release our source code and models.

1 INTRODUCTION

VoiceLoop targets voice imitation from noisy, unconstrained recordings, using a shifting buffer and simple phoneme inputs instead of aligned linguistic features. It also supports fitting new speaker identities and generating different intonations by priming the buffer.

1 INTRODUCTION

  • VoiceLoop targets voice imitation from in-the-wild samples, including noisy public speeches with clapping, multiple speakers, and inaccurate transcripts.The method is presented as an alternative to systems based on carefully collected or curated audio.Source: §1 · p. 1
  • The authors provide PyTorch code and sample audio files to support reproducibility.The passage includes a repository address.Source: §1 · p. 1
  • A shifting buffer replaces conventional recurrent networks and is shared across attention, output generation, and buffer updates.The buffer is continually refreshed with new representations, which reuse information already stored there.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Input sentences are represented as phoneme vectors, weighted by monotonic attention and summed into a context without an RNN or triphone context.The phoneme encoding uses a dictionary of 42 entries, and the same buffer and speaker identity feed the output process.Source: §1 · p. 2, Figure 1 · p. 2
  • Priming the buffer before synthesis can produce multiple intonations from an otherwise deterministic output process.The speaker embedding is fitted for new identities while the rest of the network is frozen.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper reports MOS-based TTS evaluation and near-perfect speaker identification on a real validation set, with generated speech used for testing.The supplied introduction passage does not give the MOS or generated-speech identification values.Source: §1 · p. 2

2 PREVIOUS WORK

Previous TTS approaches range from curated-data methods to increasingly complex neural systems, while in-the-wild voice imitation and post-training speaker fitting remain limited. VoiceLoop contrasts with these systems through shared memory and a simpler architecture, while using a vocoder for waveform synthesis.

2 PREVIOUS WORK

  • Prior HMM and concatenative systems rely on carefully collected or clean, well-transcribed samples, while emerging neural methods had not yet realized in-the-wild imitation from limited samples.Baljekar & Black describe filtering noisy samples for in-the-wild use; concatenative methods require tens of minutes of clean samples.Source: §2 · p. 3
  • DV2 uses specialized subsystems with stacked recurrent and fully connected networks, whereas VoiceLoop uses one shared memory, one output process, and shallow fully connected networks.The cited DV2 work is Arik et al. (2017a).Source: §2 · p. 3
  • DV2 models multiple speakers but reports no in-the-wild experiments or post-training fitting, and uses aligned phonemes and linguistic features at inference.VoiceLoop instead learns where to read from its input; the authors identify in-the-wild use and new-speaker fitting as important for modeling non-professional speakers.Source: §2 · p. 3
  • Tacotron uses a multi-stage encoder-decoder and Griffin-Lim reconstruction, while Char2Wav uses recurrent reader and generator networks with Graves positional attention.The same attention mechanism is used in VoiceLoop, but its parameters are based on the shared buffer.Source: §2 · p. 3
  • The reviewed systems generate audio through lower-level generators or spectrogram reconstruction; VoiceLoop uses the WORLD vocoder, which bounds achievable quality.Adding WaveNet or SampleRNN did not improve results on the in-the-wild training data, according to the authors.Source: §2 · pp. 3–4
  • VoiceLoop’s differentiable FIFO buffer is presented as novel; unlike Stack RNN, it does not restrict reading to the top of a stack.The authors note that memory-augmented networks existed, but had few practical applications outside this setting.Source: §2 · p. 4

3 THE ARCHITECTURE

VoiceLoop combines phoneme and speaker embeddings with a shifting buffer used throughout attention, output generation, and buffer updates. The section describes its training, new-speaker fitting, and priming-based speech variation, alongside near-real-time CPU performance.

3 THE ARCHITECTURE

  • VoiceLoop maps dictionary phonemes and speaker identities to embeddings, then uses buffer-conditioned GMM monotonic attention to form context vectors from phoneme embeddings.Source: §3 · p. 5
  • Each buffer update adds a learned, speaker-conditioned vector, shifts older columns, and discards the last; the buffer links attention, output, and updates.The initial buffer's top part repeats the speaker embedding across columns, while its lower part is zero.Source: §3 · pp. 4–6
  • Figure 2 measures buffer-column importance from network weights; the latest columns weigh more, especially for output, while the rightmost retains at least one-third of the leftmost column's contribution.Source: §3 · p. 6, Figure 2 · p. 6

3.1 TRAINING

  • Training uses MSE on aligned 63-dimensional vocoder targets; random-noise perturbations preserve a systematic train/test difference intended to better fit test-time inputs.Source: §3.1 · pp. 6–7
  • The 9.3-million-parameter model runs near real time on a single-core Intel Xeon E5 CPU and 5 times faster on an M40 GPU, including vocoder CPU decoding.Source: §3.1 · p. 7

3.2 FITTING A NEW PERSON

  • For a new speaker, only embedding z is optimized from transcribed voice samples; pitch fitting is stable, but the sample accent should resemble those in training.Source: §3.2 · p. 7

3.3 GENERATING VARIABILITY

  • Priming initializes the buffer from another utterance to vary generated speech, but links between priming content and output remain anecdotal.Source: §3.3 · p. 7

4 EXPERIMENTS

The experiments evaluate single- and multi-speaker synthesis, fitting new speaker identities, and training on public in-the-wild speech. Results report advantages over baselines in several settings, alongside measurable quality and speaker-identification outcomes.

4 EXPERIMENTS

  • The experiments use single-speaker literature datasets, VCTK subsets, and public speeches; MOS evaluation used at least 20 North American raters per experiment.MOS scores were computed with crowdMOS and 95% confidence intervals.Source: §4 · pp. 7–8

4.1 SINGLE SPEAKER EXPERIMENTS

  • Across the three single-speaker datasets, VoiceLoop outperforms Char2Wav and Tacotron on MOS and generally on MCD, while remaining below ground truth.Tacotron has lower MCD on LJ, but is not competitive there on MOS.Source: §4.1 · p. 8, Table 2 · p. 8, Table 3 · p. 8

4.2 MULTI-SPEAKER EXPERIMENTS

  • VCTK models produce distinguishable voices with different dynamic behavior, and their attention patterns show speaker-specific phoneme durations.Figure 3 depicts attention over audio time and phonemes, alongside generated and ground-truth fourth Mel-cepstrum traces.Source: §4.2 · p. 8, Figure 3 · p. 9
  • VoiceLoop has a considerable MOS advantage over Char2Wav across VCTK subsets, with results below ground truth and MCD scores consistent with that comparison.Generated voices also receive higher identification accuracy than the VCTK test split despite using the same text; Char2Wav results are considerably lower.Source: §4.2 · p. 9, Table 4 · p. 9, Table 5 · p. 10

4.3 NEW SPEAKER FITTING EXPERIMENTS

  • New speaker embeddings are fitted by updating only their weights; fitted voices score 3.08 ± 0.95 MOS and 87.6% top-1 identification accuracy.With two sentences—about 10 seconds—per speaker, identification reaches 64.4% top-1; the fitting set averages 23.65 minutes per speaker.Source: §4.3 · pp. 10–11
  • Identification performance depends on fitting-data length, yet the reported two-sentence setting averages only about 10 seconds per speaker.The authors repeated fitting five times at each capped data limit; Figure 4 plots top-1 accuracy against sample-set length.Source: Figure 4 · p. 10, §4.3 · p. 11

4.4 IN THE WILD EXPERIMENTS

  • Training from scratch on 6.2 hours of noisy, automatically transcribed public speeches yields 2.97 ± 1.03 MOS and 95.81% top-1 accuracy.The data comprises 8,000 segments from four public figures.Source: §4.4 · p. 11
  • On the in-the-wild dataset, priming the buffer differently generates multiple intonations for the same speaker and input sentence.Figure 5 shows waveform, spectrogram, and F0 estimates for outputs generated with different initializations.Source: §4.4 · p. 11

5 DISCUSSION

The discussion reports robustness to noisy, mixed in-the-wild training data and fast generation, while describing priming as a way to vary intonation. It also frames the Baddeley connection as an analogy and the buffer architecture as potentially useful beyond speech synthesis.

Discussion

  • VoiceLoop models voices from mixed web audio despite clapping, laughter, dramatic prosody, and pauses, without reproducing background noise in synthesized output.The authors report that Char2Wav instead presented clapping sounds in its YouTube-dataset output.Source: §5 · p. 11
  • The iterative method can generate voice on mobile devices at speeds far exceeding real time, which the authors connect to its shallow networks and lack of future-sample computation.Deep Voice is reported at up to 2.7 times real-time on a multi-threaded Intel Xeon CPU.Source: §5 · p. 11
  • Different initializations of S0 produce different intonations for the same speaker and sentence, including higher and lower tones.Figure 5 also shows differing phoneme durations across priming conditions.Source: Figure 5 · p. 12
  • The Baddeley link is an analogy, not an implementation; the authors define phonological features as a joint sound- and language-based representation in memory.They state that removing the buffer dependency in the update makes the model completely ineffective.Source: §5 · p. 12

6 CONCLUSION

The paper presents a simpler neural TTS architecture that works with unconstrained samples without phoneme–acoustic alignment or linguistic-feature inputs. It also reports multi-speaker TTS from public speeches and post-training fitting to new speakers from very limited samples.

6 CONCLUSION

  • The method supports multi-speaker TTS from unconstrained public-speech samples, which the authors describe as a first to their knowledge.Source: §6 · p. 13
  • The new memory architecture serves as a working-memory module and underlies a neural TTS system less complex than recent literature systems.Source: §6 · p. 13
  • The system requires neither phoneme–acoustic alignment nor linguistic-feature inputs.Source: §6 · p. 13
  • The system can fit new speakers after training, demonstrated even with very limited sample sizes.Source: §6 · p. 13

Related papers

Loading 1707.06588v3…