Source-linked AI summary
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux
TL;DR
The paper addresses uncertainty about whether self-supervised speech representations support synthesis and disentangle content, speaker identity, and F0. It combines separate discrete encoders with a decoder, evaluates reconstruction and controllability, and adapts the representations into a 365 bits per second codec that outperforms baselines subjectively.
Problem
Self-supervised speech representations have mainly been studied for ASR, leaving their suitability for synthesis and encoding of speaker identity and F0 unclear.
Method
The system combines discrete content and F0 encoders with a global speaker identity encoder and a decoder, evaluating multiple representation-learning models.
Results
365 bits per second is achieved for an ultra-lightweight speech codec with subjective quality superior to baseline methods.
Takeaways & Limitations
The representations support controllable speech resynthesis through separate content, prosodic, and speaker information, and can serve as a lightweight codec.
Abstract
from arXiv · showhide
We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (for both resynthesis and voice conversion), recordings' intelligibility, and overall quality using subjective human evaluation. Lastly, we demonstrate how these representations can be used for an ultra-lightweight speech codec. Using the obtained representations, we can get to a rate of 365 bits per second while providing better speech quality than the baseline methods. Audio samples can be found under the following link: speechbot.github.io/resynthesis.
1. Introduction
Self-supervised speech representations have been successful for recognition, but their suitability for synthesis and their encoding of speaker identity and F0 remain unclear. The paper proposes resynthesis using separate discrete representations and evaluates their controllability and codec potential.
- Self-supervised speech representations have mainly been evaluated for ASR, leaving their suitability for speech synthesis unclear.
- The proposed system uses three parallel encoders whose discrete outputs condition a decoder to reconstruct the input signal.
- Separate learned units represent speech content, while quantized F0 and a global speaker embedding provide additional controllable information.
- The study evaluates signal reconstruction, voice conversion, F0 manipulation, and the properties of several encoder models across multiple datasets.
- 365 bits per second is achieved for speech coding while remaining significantly superior to lightweight and heavyweight baseline codecs.
2. Related Work
Unsupervised speech representation learning includes reconstruction-based and self-supervised approaches, with constraints such as discreteness and hierarchy. Prior SSL methods have excelled in recognition-related tasks, while synthesis work has explored low-bitrate units and neural vocoders.
- Unsupervised speech representation learning is broadly divided into reconstruction and self-supervised learning methods.
- Auto-encoding reconstructs speech from a low-dimensional latent representation, with constraints including temporal smoothness, discreteness, and hierarchy.
- SSL methods have achieved strong results for ASR, phoneme segmentation, and GSLM using contrastive or masked prediction objectives.
- Speech synthesis research has used low-bitrate representations, VQ-VAE codebooks, and neural vocoders conditioned on log Mel-spectrograms.
3. Method
The method combines separate discrete content and F0 representations with a global speaker embedding, then uses a modified HiFi-GAN vocoder for speech resynthesis.
- Encoders: Three fixed encoders extract discrete content and F0 representations plus a global speaker-identity representation for the decoder.The content and F0 encoders operate on raw audio or extracted F0, while the speaker encoder outputs a single d-vector.
- Content representation: CPC, HuBERT, and VQ-VAE are evaluated as content encoders, with k-means discretizing the continuous CPC and HuBERT outputs.VQ-VAE representations are already quantized and therefore do not use the same post-processing.
- F0 representation: The F0 encoder uses VQ-VAE quantization, mapping latent vectors to codebook entries whose indices form the discrete pitch sequence.The decoder reconstructs the original F0 signal from embedded codebook vectors during F0-encoder training.
- Decoder: A modified HiFi-GAN decoder receives embedded content and pitch sequences together with the repeated speaker embedding to generate the resynthesized waveform.Look-up tables convert discrete sequences to continuous representations, which are upsampled, concatenated, and passed through the generator.
- Training objective: Training combines adversarial, Mel-spectrogram reconstruction, and discriminator feature-matching losses for the generator and discriminators.The reconstruction term compares input and generated Mel-spectrograms, while feature matching compares discriminator activations.
4. Results
The experiments evaluate reconstruction, disentanglement, F0 preservation, and bitrate-quality trade-offs across datasets and representation methods. HuBERT provides the strongest reported low-bitrate codec result, while VQ-VAE retains more speaker and F0 information in its encoded units.
- Evaluation setup: The study evaluates reconstruction, speaker conversion, F0 manipulation, and subjective speech-codec quality on LJ and VCTK datasets.All audio is resampled to 16 kHz, with objective and subjective metrics used across the experiments.
- Bitrate configuration: 700 bps for CPC, 350 bps for HuBERT, and 800 bps for VQ-VAE are the content-code bitrates used in the implementation.CPC and HuBERT use 100 k-means centroids, whereas VQ-VAE uses 256 discrete tokens.
- Speaker disentanglement: VQ-VAE preserves more speaker information in encoded units, while CPC and HuBERT outperform it on voice conversion for both LJ and VCTK.The paper links this pattern to more artifacts from speaker information retained in VQ-VAE units.
- F0 disentanglement: 5.2 vs 7.03 and 5.59 vs 7.8 show that VQ-VAE nearly matches original-F0 conditioning after F0 flattening, unlike CPC and HuBERT.The comparison uses F0 reconstruction metrics after replacing frame-level F0 units with speakers’ mean voiced-frame F0.
5. Conclusion
The paper applies self-supervised discrete representations to speech resynthesis and evaluates their disentanglement for reconstruction, voice conversion, and F0 manipulation. It also adapts HuBERT into an ultra-lightweight codec with superior subjective results at lower bitrate than baselines.
- Conclusion: The work demonstrates self-supervised discrete representations for speech resynthesis and evaluates their properties across reconstruction, voice conversion, and F0 manipulation.The representations are also adapted for speech-codec evaluation.
- Conclusion: HuBERT provides superior subjective speech-codec results than the baselines while using a lower bitrate.This conclusion summarizes the codec comparison reported in the experiments.