Source-linked AI summary
AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, Mark Hasegawa-Johnson
TL;DR
Non-parallel many-to-many and zero-shot voice conversion remain under-explored, while GANs are difficult to train and CVAEs lack guaranteed distribution matching. The paper proposes a bottlenecked autoencoder trained only with self-reconstruction, formally establishes distribution matching, and reports superior non-parallel many-to-many performance plus first zero-shot conversions.
Problem
Non-parallel many-to-many and zero-shot voice conversion remain under-explored, including conversion to unseen speakers from only a few utterances.
Method
The paper proposes a vanilla autoencoder with a carefully designed bottleneck, trained only on self-reconstruction to remove style information and achieve distribution matching.
Results
AUTOVC significantly outperforms existing state-of-the-art non-parallel voice conversion systems and is the first to perform zero-shot conversion.
Takeaways & Limitations
AUTOVC suggests that a simple autoencoder with careful bottleneck tuning can support simpler voice conversion and general style-transfer systems.
Takeaways & Limitations
The theoretical guarantee depends on assumptions including sufficiently large frame count and properly set bottleneck dimension, while the speech model assumes speakers produce the same amount of gross information.
Abstract
from arXiv · showhide
Non-parallel many-to-many voice conversion, as well as zero-shot voice conversion, remain under-explored areas. Deep style transfer algorithms, such as generative adversarial networks (GAN) and conditional variational autoencoder (CVAE), are being applied as new solutions in this field. However, GAN training is sophisticated and difficult, and there is no strong evidence that its generated speech is of good perceptual quality. On the other hand, CVAE training is simple but does not come with the distribution-matching property of a GAN. In this paper, we propose a new style transfer scheme that involves only an autoencoder with a carefully designed bottleneck. We formally show that this scheme can achieve distribution-matching style transfer by training only on a self-reconstruction loss. Based on this scheme, we proposed AUTOVC, which achieves state-of-the-art results in many-to-many voice conversion with non-parallel data, and which is the first to perform zero-shot voice conversion.
1. Introduction
Voice conversion still lacks robust solutions for non-parallel many-to-many and zero-shot settings. AUTOVC addresses these gaps with a carefully bottlenecked autoencoder trained only on self-reconstruction, combining simple training with distribution matching.
- Voice conversion modifies source speech to match a target speaker’s vocal qualities for applications including privacy, identity protection, and creative work.
- Three gaps remain: limited non-parallel training, few many-to-many systems, and no zero-shot conversion to unseen speakers from few utterances.
- GANs offer distribution matching but are difficult to train, while CVAEs are easier to train but lack guaranteed distribution matching and can over-smooth outputs.
- The paper asks whether style transfer can combine GAN-like distribution matching, CVAE-like training simplicity, and better speech performance.
- The proposed scheme uses a vanilla autoencoder with a carefully designed bottleneck and self-reconstruction loss to remove source style information.
- AUTOVC achieves superior many-to-many conversion without parallel data and among the first zero-shot conversions with decent performance using speaker embeddings.
2. Related Works
Prior work applies VAE-based and GAN-based methods to non-parallel many-to-many voice conversion, but conversion quality remains limited. The paper distinguishes AUTOVC by unveiling distribution matching through bottleneck design without relying on text transcriptions.
- Existing non-parallel many-to-many systems use VAE, adversarial training, auxiliary classifiers, or encoder-decoder GAN architectures.
- Conversion quality of these algorithms remains limited despite methods that remove style information from latent codes or add adversarial objectives.
- Some systems use text transcriptions to assist latent-code learning, whereas this paper focuses on voice conversion without transcriptions.
- Earlier autoencoder style-transfer works had not unveiled distribution matching through proper bottleneck design.
3. Style Transfer Autoencoder
The paper formulates voice conversion as transferring speaker style while preserving source content, and proposes an autoencoder whose bottleneck enables distribution-matching conversion using self-reconstruction training. A properly sized bottleneck supports both perfect reconstruction and speaker disentanglement, which theoretically implies ideal conversion.
- Problem Formulation: Speech is modeled as generated from speaker identity, content, and a conditional speech distribution.Content includes phonetic and prosodic information, and the paper works with speech spectrograms.
- Problem Formulation: Ideal conversion preserves source content while matching the target speaker’s conditional speech distribution.The converted speech should sound like the target speaker uttering the source content.
- The Autoencoder Framework: AUTOVC uses content and speaker encoders with a decoder, training on self-reconstruction and content-code reconstruction losses.During conversion, source speech supplies content and target speech supplies speaker information; during training, another utterance from the same speaker is used.
- Theoretical Guarantee: A properly designed bottleneck is the key condition for proving that autoencoder training can achieve ideal distribution-matching voice conversion.The theorem assumes speaker embeddings are consistent within speakers, distinct across speakers, and speech follows a finite-cardinality ergodic stationary Markov process.
- Theoretical Guarantee: For sufficiently large frame length and properly chosen bottleneck dimension, the loss approaches zero and the average KL divergence to the ideal target distribution is zero.The result applies to the global optimizer under the theorem’s assumptions.
- Why Does It Work?: When the bottleneck is neither too wide nor too narrow, reconstruction remains perfect while the content embedding removes source-speaker information.A wide bottleneck retains speaker information, whereas an overly narrow bottleneck loses content information.
4. AUTOVC Architecture
AUTOVC combines a speaker encoder, content encoder, decoder, and spectrogram inverter around a carefully constrained information bottleneck. The architecture uses a speaker-verification embedding for unseen speakers and temporal downsampling to restrict content information flow.
- Architecture: AUTOVC processes N-by-T mel-spectrograms with a speaker encoder, content encoder, decoder, and spectrogram inverter.The inverter converts output mel-spectrograms back to waveform audio.
- The Speaker Encoder: The speaker encoder produces a 256-by-1 embedding and is pretrained with GE2E loss to cluster utterances by speaker while separating different speakers.This embedding is designed to generalize to unseen speakers for zero-shot conversion.
- The Content Encoder: The content encoder combines an 80-dimensional mel-spectrogram with the speaker embedding before convolutional and bidirectional LSTM processing.The bidirectional LSTM has combined output dimension 64 from forward and backward cell dimensions of 32.
- The Content Encoder: Both bidirectional LSTM outputs are downsampled by 32 to construct the information bottleneck.Forward and backward paths retain offset time-step sequences before the decoder restores temporal resolution.
- The Decoder: The decoder upsamples content and speaker embeddings, concatenates them, and predicts an initial converted spectrogram estimate.A postnetwork then constructs fine spectrogram details as a residual signal.
- The Decoder: Although the total loss differs from the theorem’s loss, the added initial reconstruction loss empirically improves convergence without harming performance.Reconstruction loss is applied to both initial and final reconstruction results.
5. Experiments
AUTOVC is evaluated on non-parallel many-to-many and zero-shot voice conversion using subjective tests and bottleneck analyses. It outperforms baselines, generalizes to unseen speakers, and balances reconstruction with speaker disentanglement.
- Evaluation setup: The VCTK evaluation uses 44 hours from 109 speakers, with non-parallel data and speaker-level 9:1 train-test splits.
- Evaluation setup: MTurk evaluates converted-speech naturalness with MOS scores and target-speaker similarity using utterances or paired utterances.
- Traditional many-to-many conversion: AUTOVC’s MOS exceeds 3 for all traditional conversion gender groups, while baseline scores almost all fall below 3.The authors report that AUTOVC approaches the naturalness of parallel conversion systems and outperforms existing non-parallel systems.
- Traditional many-to-many conversion: AUTOVC outperforms baselines in similarity, without their significant same-gender to cross-gender degradation.AUTOVC and AUTOVC-ONE-HOT show no significant similarity difference, indicating the gain does not result from the speaker encoder.
- Zero-shot conversion: For zero-shot conversion, MOS exceeds 3 in most unseen-speaker settings, while unseen-to-unseen similarity remains competitive despite being the most challenging case.Performance among seen speakers is comparable after training on twice as many speakers, indicating scalability to a larger training set.
- Bottleneck dimension analysis: Bottleneck reduction increases reconstruction error and decreases speaker-classification accuracy; normal AUTOVC balances both properties.On the 20-speaker comparison, AUTOVC obtains 14.2% classification accuracy versus 45.1% after adversarial training in Chou et al. (2018).
6. Conclusion
The paper proposes AUTOVC as a simple non-parallel voice conversion algorithm that uses self-reconstruction and bottleneck tuning. It reports superior performance and the first zero-shot conversions, motivating simpler style-transfer designs.
- AUTOVC significantly outperforms existing state-of-the-art non-parallel voice conversion systems and is the first to perform zero-shot conversions.
- Its autoencoder trains only on self-reconstruction while bottleneck tuning balances reconstruction quality and speaker disentanglement.
- The authors conclude that a carefully designed autoencoder bottleneck can provide a strong alternative to increasingly sophisticated style-transfer algorithms.
Appendix: Proving Thm 1
The appendix proves that a properly capacity-limited bottleneck can support the theorem’s ideal conversion property at an asymptotic global optimum. The proof uses coding and information-capacity arguments, then connects self-reconstruction to distributional equivalence.
- The construction uses a set of inputs whose optimal conditional code length is below the bottleneck dimension, padding valid codes and assigning arbitrary codes outside that set.The set is defined by log pX1(x1|U1) ≤ n−1, and the coding argument invokes Shannon’s coding theorem.
- The proof bounds bottleneck information through its number of dimensions and uses assumptions about finite-support, stationary Markov speech processes to establish the required asymptotic behavior.The argument invokes finite support, Markov-process properties, and a central-limit result for ergodic Markov processes.
- The constructed content and speaker codes form a lossless representation of the source variables, allowing an optimum decoder to recover the relevant information.The appendix states that [C1, S1] is a lossless code of [X1, U1] and that [U1, X1] is fully recoverable from it.
- The resulting encoder-decoder pair is an asymptotic global optimizer of the training objective, with the proof connecting probability convergence to mean-squared convergence.The appendix explicitly concludes that [E*c(·), D*(·, ·)] is the asymptotic global optimizer of Eq. (5).
- For sufficiently large T and properly chosen bottleneck dimension n, self-reconstructed converted speech is asymptotically identically distributed to the intended converted speech.The proof bridges a ground-truth converted sample and AUTOVC self-reconstruction, then establishes asymptotic identical distribution.