Can we steal your vocal identity from the Internet?: Initial investigation of cloning Obama's voice using GAN, WaveNet and low-quality found data

1. Introduction

• Near-zero EERs on ASVspoof2015 do not cover newer direct-waveform and GAN-based synthesis methods [3].A prior study found direct-waveform speech as natural as human speech, while GAN-generated Mel-spectra fooled speaker recognition [8] under impractical feature-access assumptions [5].Source: §1 · p. 1
• ASVspoof2015 studies relied on studio-recorded speech, leaving attacks based on unwillingly obtained, non-controlled recordings insufficiently examined.Public recordings are abundant for public personalities such as Barack Obama, but typically contain noise and reverberation.Source: §1 · p. 1
• The study investigates whether speech enhancement can improve low-quality public data enough to train effective state-of-the-art spoofing TTS and VC systems.The authors explicitly evaluate both enhancement quality and the resulting spoofing materials.Source: §1 · p. 1

2.1. SEGAN-based speech enhancement

• SEGAN uses an encoder-decoder generator to map noisy speech to enhanced speech while a discriminator distinguishes enhanced from clean speech.The discriminator’s feedback updates the generator toward outputs closer to clean speech.Source: §2.1 · p. 2
• The authors modify SEGAN because noise variation makes training sensitive and convergence difficult.The first modification computes early generator content loss against baseline-enhanced speech rather than clean speech.Source: §2.1 · p. 2

2. GAN-based speech enhancement

• The method improves SEGAN’s training robustness by combining baseline-guided initialization with residual learning.These changes target the sensitivity and convergence problems identified during training.Source: §2.1 · p. 2

2.1. SEGAN-based speech enhancement

• A skip connection makes the generator produce a residual refinement of noisy input instead of enhanced speech from scratch.This is intended to focus learning on waveform differences between clean and enhanced speech.Source: §2.1 · p. 2

3.1. Corpora for speech enhancement training

• The enhancement corpora use 28-speaker British-accent VCTK speech plus noisy, reverberant, and mixed corrupted variants.These corpora were constructed from publicly available resources to support replicable training.Source: §3.1 · p. 2, §3.1.2 · p. 2

3.1.1. Device-recorded VCTK corpus

• Device-recorded VCTK replays high-quality speech through eight microphones in an office under two noise conditions, producing 16 recording conditions.This trains relationships between high-quality and consumer-device recordings.Source: §3.1.1 · p. 2

3.1.2. Noisy, Reverberant, and Noisy and reverberant VCTK corpora

• The corrupted VCTK variants separately represent noise, reverberation, and their combination, covering multiple degradation types.All variants share the CSTR VCTK speakers and utterances.Source: §3.1.2 · p. 2

3.2. Obama’s found data

• Obama’s found data combine transcribed public interventions from YouTube, interviews, and political meetings recorded under diverse non-studio conditions.Audio was separated from video, segmented on long pauses, and down-sampled to 16 kHz.Source: §3.2 · p. 2, §3 · p. 2
• 17.15 dB mean SNR and 171.22 dB variance characterize the Obama source data, versus 30.19 dB and 23.73 dB for VCTK.The SNR histogram shows that most found-data speech had very low SNR by conventional speech-generation standards.Source: §3.2 · pp. 2–3, Figure 1 · p. 3

4.1. Design of the speech enhancement models

• The system used a modified SEGAN speech-enhancement pipeline trained across seven data-source regimes, including individual disturbance categories and combinations with device-related degradation.Baseline enhancement used sequential Postfish and HRNR processing before SEGAN training.Source: §4.1 · p. 3, §4.2 · p. 3

4.3. Objective evaluation of the speech enhancement

• All enhancement models improved the average SNR of the original Obama voice data, while noisy-data training outperformed reverberation- or device-focused training for this metric.The authors attribute this pattern to SNR targeting noise reduction, which may not improve perceptual quality.Source: §4.3 · p. 3

4.4. Perceptual evaluation of the speech enhancement

• Perceptual evaluation measured speech quality and cleanliness on MOS scales using 129 listeners and 530 utterances presented across repeated evaluation sets.Cleanliness was defined as freedom from noise, reverberation, and artifacts.Source: §4.4 · pp. 3–4

4.4.1. Results

• Enhanced versions improved source cleanliness by varying amounts, with the noisy-reverberant condition reaching the largest reported cleanliness improvement at 3.42.The all-condition enhancement also achieved a similarly high cleanliness result.Source: §4.4.1 · p. 4
• Enhancement consistently degraded perceived speech quality; noisy and noisy-reverberant conditions received quality scores of 2.73 and 3.11, respectively.Thus, the conditions producing the largest SNR improvements were not necessarily best for producing clean speech for further processing.Source: §4.4.1 · p. 4
• The all-condition model balanced the trade-off most effectively, producing cleanliness 3.40 with only 0.17 quality degradation.The authors suggest that varied degradation training improved generalization while preserving more voice naturalness.Source: §4.4.1 · p. 4

5. Generation of the synthetic samples

• CycleGAN converts source-speaker mel-spectrograms toward target-speaker characteristics, while autoregressive TTS maps linguistic features to target-speaker mel-spectrograms before WaveNet waveform synthesis.CycleGAN preserves speech contents during voice conversion; autoregressive TTS uses previous generated acoustic output as feedback.Source: §5 · p. 4, §5.1 · p. 5, §5.2 · p. 5
• Mel-spectrograms were selected because F0 estimation is problematic in noisy and enhanced speech, with 80 mel bands used to better handle corrupted segments.Source: §5 · pp. 4–5

5.1. VC based on CycleGAN

• CycleGAN learns non-parallel voice conversion by combining cycle consistency, which can retain linguistic information, with adversarial learning, which changes speaker individuality.Source: §5.1 · p. 5
• The CycleGAN systems target Obama using original and enhanced speech, with Japanese-to-English, English-to-English, and mixed Japanese-English source configurations.Source: §5.1 · p. 5

5.2. TTS acoustic modeling based on AR neural network

• The autoregressive TTS model converts linguistic features into frame-aligned mel-spectrograms using feedforward layers, bidirectional and unidirectional LSTMs, and previous-output feedback.Source: §5.2 · p. 5, Figure 6 · p. 5

5.3. WaveNet vococder

• WaveNet is trained on enhanced Obama data to generate 16-kHz waveforms, using 40 dilated-convolution blocks and acoustic features containing 80-dimensional mel-spectrograms.Multiple enhanced versions can increase the data available for training the vocoder.Source: §5.3 · p. 6, Figure 7 · p. 6

6.2. Results

• Natural Obama speech received 4.70 MOS for similarity and 4.40 MOS for quality, showing that listeners identified the speaker despite environmental variation.Source: §6.2 · pp. 6–7
• WaveNet copy synthesis achieved only 2.45 MOS for quality, despite prior success with clean training speech, suggesting difficulties modeling the noisy training data.Source: §6.2 · p. 7
• Adding slight reverberation improved TTS quality from 2.45 to 2.63 MOS, but the low similarity scores prevented determining whether listeners identified Obama’s voice.Source: §6.2 · p. 7
• TTS quality stayed near 2.5 MOS across enhancement conditions and remained similar to copy synthesis.Source: §6.2 · p. 7
• VC systems were slightly but significantly better than TTS and copy synthesis in quality, and better than TTS but much worse than copy synthesis in similarity.The VC systems used selected data with estimated SNR above 30 dB and a clean source speaker.Source: §6.2 · p. 7
• Japanese-to-English and English-to-English VC obtained similar quality and similarity scores, while mixed-language VC had slightly higher quality, likely from using twice as much source-speaker training data.Source: §6.2 · p. 7

• The countermeasure uses speaker-independent GMM back ends with CQCC features and separate natural-speech and synthetic-speech training sets.Source: §7 · p. 7
• CQCC-GMM countermeasures trained on ASVspoof2015 or VCC2016 data easily detected both the proposed VC and TTS samples.The evaluation reports equal error rates, with higher EER indicating greater countermeasure confusion.Source: §7 · p. 7, Table 7 · p. 7
• The authors attribute detectability to artifacts introduced by both speech enhancement and the subsequent VC or TTS process.Source: §7 · p. 7

• Speech enhancement significantly improved the SNR of considerably degraded speech collected from publicly available sources.Source: §8 · p. 7
• Perceptual evaluation found cleaner source speech without significant degradation of voice naturalness, with best enhancement from the largest and most varied training set.Source: §8 · p. 7
• TTS and VC generated from noisy public data still had obvious perceptual problems relative to systems trained on very high-quality data.Source: §8 · p. 7
• The authors therefore do not yet recommend using publicly available data to generate next-generation ASVspoof data, while considering the paradigm important to add.Source: §8 · p. 7