Source-linked AI summary
Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
Hyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Hwan Lee, Hoon Heo, Kyogu Lee
TL;DR
NANSY addresses the reconstruction-quality and controllability challenges of speech analysis and synthesis without relying on labeled text or speaker information. It combines self-supervised wav2vec and Yingram representations with information perturbation and achieves improvements across zero-shot voice conversion, pitch shifting, and time-scale modification.
Problem
Information-bottleneck approaches for controllable synthesis usually trade reconstruction quality for disentanglement, while text-based methods require labeled data and are language-dependent.
Method
NANSY uses wav2vec and Yingram analysis features with information perturbation to selectively control linguistic and pitch information during synthesis.
Results
NANSY achieves significant performance improvements in zero-shot voice conversion, pitch shift, and time-scale modification.
Takeaways & Limitations
A single fully self-supervised model can provide controllable speech manipulation without text or speaker labels and can be extended to unseen languages with test-time self-adaptation.
Takeaways & Limitations
Voice conversion can be misused to harm people or contribute to spreading fake information.
Abstract
from arXiv · showhide
We present a neural analysis and synthesis (NANSY) framework that can manipulate voice, pitch, and speed of an arbitrary speech signal. Most of the previous works have focused on using information bottleneck to disentangle analysis features for controllable synthesis, which usually results in poor reconstruction quality. We address this issue by proposing a novel training strategy based on information perturbation. The idea is to perturb information in the original input signal (e.g., formant, pitch, and frequency response), thereby letting synthesis networks selectively take essential attributes to reconstruct the input signal. Because NANSY does not need any bottleneck structures, it enjoys both high reconstruction quality and controllability. Furthermore, NANSY does not require any labels associated with speech data such as text and speaker information, but rather uses a new set of analysis features, i.e., wav2vec feature and newly proposed pitch feature, Yingram, which allows for fully self-supervised training. Taking advantage of fully self-supervised training, NANSY can be easily extended to a multilingual setting by simply training it with a multilingual dataset. The experiments show that NANSY can achieve significant improvement in performance in several applications such as zero-shot voice conversion, pitch shift, and time-scale modification.
1 Introduction
NANSY decomposes arbitrary speech into interpretable, manipulable analysis features and synthesizes it without text or speaker labels. It uses wav2vec, Yingram, information perturbation, and test-time self-adaptation to support controllable and multilingual speech applications.
- NANSY decomposes speech into features representing pronunciation, timbre, pitch, and energy for manipulation and re-synthesis.
- Text-based voice conversion requires supervised ASR training and paired text-waveform data, while language dependence limits multilingual and low-resource use.
- wav2vec preserves linguistic information without text, while Yingram represents and controls pitch, including cases where fundamental frequency is ill-defined.
- Information perturbation disentangles shared pitch and timbre information so analysis features can control specific speech attributes without an information bottleneck.
- Test-time self-adaptation updates only the input linguistic feature using a single speech sample, modifying mispronounced regions for unseen-language use.
- The framework is fully self-supervised and supports zero-shot voice conversion, formant-preserving pitch shift, and time-scale modification.
2 NANSY: Neural Analysis and Synthesis
NANSY extracts linguistic, speaker, energy, and pitch-related representations, then reconstructs speech through source and filter generators. Its analysis features include XLSR-53 wav2vec representations and Yingram, with pitch controllability supported by a converted time-lag representation.
- Training procedure: The framework perturbs waveform information during training, including formant, pitch, and frequency response, while one perturbation preserves pitch.The overview assigns these perturbed representations to linguistic, pitch, speaker, and energy information flows.
- Analysis features: XLSR-53 wav2vec intermediate features provide language-agnostic linguistic information for speech reconstruction.The model uses the output from the 12th layer of XLSR-53’s 24-layer transformer encoder.
- Analysis features: The analysis pipeline includes speaker and energy features alongside wav2vec and Yingram representations.Speaker embeddings are also derived from XLSR-53, while energy is computed by averaging a log-mel spectrogram along the frequency axis.
- Analysis features: Yingram represents pitch by converting Yin’s time-lag axis to a MIDI-scale axis, enabling pitch control.Yingram is motivated by the difficulty of explicitly estimating f0 for irregularly periodic speech such as creaky voice.
- Synthesis network: NANSY separates synthesis into source and filter generators, with Yingram driving GS and wav2vec driving GF.Energy and speaker features are inputs to both generators, whose outputs are summed to reconstruct a mel spectrogram before vocoding.
3 Training
NANSY trains generators with perturbed analysis features so they selectively reconstruct speech attributes, while adversarial and reconstruction losses improve synthesis quality. TSA adapts only the input linguistic representation at test time to address pronunciation errors.
- Information Perturbation: Formant shifting, pitch randomization, and parametric equalization perturb wav2vec and Yingram inputs to encourage attribute-selective extraction.The wav2vec pathway preserves linguistic information, while the Yingram pathway preserves f0-related information.
- Training Loss: The generators and speaker embedding network use L1 mel-spectrogram loss together with speaker-conditional adversarial training.The adversarial term addresses over-smoothed acoustic features associated with L1 or L2 training alone.
- Training Loss: The discriminator objective contrasts real and generated mel spectrograms using positive and randomly sampled negative speaker embeddings.The formulation uses projection conditioning for both positive and negative pairs.
- Training Loss: The generator objective combines adversarial training with the L1 distance between generated and ground-truth mel spectrograms.This objective jointly targets realism and reconstruction fidelity.
- Test-time Self-Adaptation: TSA updates only a parameterized wav2vec feature using test-sample reconstruction loss, while gradients pass through the filter generator rather than retraining network parameters.The method uses a single test-time sample to modify the input representation for pronunciation correction.
4 Experiments
Experiments evaluate NANSY on English and multilingual reconstruction, voice conversion, pitch shifting, and time-scale modification. TSA is additionally tested for unseen-language pronunciation, while Yingram is compared with f0 for robust reconstruction.
- Evaluation: English and multilingual reconstruction are evaluated using CER, MOS, and DMOS, with lower CER and DMOS and higher MOS indicating better performance.CER is obtained through Google Cloud ASR, while MOS and DMOS are collected from MTurk.
- Reconstruction: Yingram is preferred over f0 in 68.3% of ABX judgments on samples that f0-based reconstruction failed to reproduce faithfully.The result supports Yingram as a more robust pitch feature when f0 cannot be accurately estimated.
- Reconstruction: The reconstruction experiments use VCTK and LibriTTS for English and CSS10 for multilingual speech, with held-out utterances used for testing.CSS10 contains ten speakers using different languages and has no English-speaking speaker.
- Test-time Self-Adaptation: MUL generally achieves better CER than ENG and ENG-TSA, while ENG-TSA sometimes matches or exceeds MUL on the ten-language evaluation.For each configuration, the wav2vec feature is iteratively updated 100 times.
- Voice Conversion: NANSY evaluates zero-shot voice conversion across seen, multilingual, and unseen-language settings using MOS, speaker similarity, and intelligibility criteria.Pitch is aligned between source and target by shifting the Yingram scope.
- Pitch Shift and Time-Scale Modification: NANSY is compared with PSOLA and WORLD vocoder for pitch shifting and time-scale modification using MOS across multiple control values.Pitch shift spans -6, -3, 0, 3, and 6 semitones, while time-scale ratios span 1/2, 1/1.5, 1, 1.5, and 2.
5 Related Works
Related work covers self-supervised speech representations, information-bottleneck approaches to zero-shot voice conversion, neural integration with speech production models, and consistency learning. NANSY’s information perturbation is distinguished by training a generative decoder to select attributes from perturbed representations.
- Self-supervised Representation Learning: Self-supervised speech research learns representations from augmented or masked speech without relying solely on labeled downstream data.The related work discusses noise contrastive estimation and masked language modeling for speech representation learning.
- Zero-shot Voice Conversion: Zero-shot voice conversion has commonly used information bottlenecks or supervised speaker-recognition representations to restrict information flow.These approaches separate speaker and linguistic information through bottleneck design or speaker embeddings.
- Inductive Bias for Audio Generation: Neural speech systems have also integrated source-filter production models for vocoding and acoustic feature generation.This line of work combines neural networks with traditional speech or sound production structure.
- Consistency Learning: Unlike consistency learning for classification, information perturbation targets generative reconstruction by training the decoder to retain essential attributes from perturbed representations.Both approaches perturb data, but NANSY applies the idea to decoder-side selective synthesis.
6 Conclusions and Discussion
NANSY provides one self-supervised model for voice conversion, formant-preserving pitch shift, and time-scale modification, with TSA supporting unseen-language use. Its remaining limitation is that linguistic information is not directly controllable.
- Conclusions: NANSY performs zero-shot voice conversion, formant-preserving pitch shift, and time-scale modification with a single model.The framework is trained without labeled text or speaker information.
- Conclusions: The information perturbation and TSA experiments support controllable speech manipulation and extension to low-resource languages.TSA is evaluated on unseen languages, while the model remains fully self-supervised.
- Limitations and Future Work: NANSY lacks controllability over linguistic information, motivating future integration of text as a side input.The proposed hybrid direction is intended to let users manipulate linguistic content.
Broader Impacts
NANSY's controllable voice conversion can be misused, creating risks for misinformation and biometric security. The authors recommend consent-based restrictions on release.
- Voice conversion may be used by unidentified users to help spread fake news.
- The technology can raise concerns about speech-based biometric security systems.
- The authors recommend not releasing the system without consent to limit malicious use.
Appendix A Information Perturbation
Information perturbation uses formant shifting, pitch randomization, and parametric equalization to alter selected speech properties during training. The appendix specifies their sampling ranges and filter construction.
- Three perturbation functions are used: formant shifting, pitch randomization, and random frequency shaping with a parametric equalizer.
- Formant-shift ratios are sampled from U(1, 1.4), while pitch-shift and pitch-range ratios are sampled from U(1, 2) and U(1, 1.5).Each sampled ratio may also be replaced by its reciprocal.
- The parametric equalizer composes low-shelving, peaking, and high-shelving filters for random frequency shaping.It uses one low-shelving filter, one high-shelving filter, and eight peaking filters.
- Each equalizer component is a second-order IIR filter with frequency, quality-factor, and gain parameters.Shelving cutoffs are fixed at 60Hz and 10kHz; quality factors and gains are randomized.
Appendix B Neural Architectures
The generator uses shared architecture for source and filter branches, with different analysis features supplying each branch. The discriminator uses residual 1D CNN blocks and speaker conditioning.
- Generator: GS and GF share the same architecture but receive different inputs: Yingram for GS and wav2vec for GF.Both branches also use energy and speaker-embedding features.
- Generator: Conditional layer normalization provides speaker conditioning in the generator.
- Discriminator: The discriminator uses 1D-CNN residual blocks and conditions on speaker embeddings from the speaker network.
- Architecture notation: The generator figures define k, d, c, and ic as kernel size, dilation, channel size, and input channel size.The figure specifies generator input channels of 1024 for wav2vec and 984 for Yingram.
- Architecture notation: The discriminator figure defines c+ as a positively paired speaker embedding and c− as a randomly sampled negative embedding.
Appendix C Full Speaker Similarity Evaluation Results
NANSY's full speaker-similarity evaluation reports better voice-conversion performance than baseline models across every gender-to-gender setting. The figure distinguishes female-to-female, female-to-male, male-to-female, and male-to-male conversions.
- NANSY achieves better voice-conversion performance than baseline models in every gender-to-gender setting.
- Figure 9 reports the complete speaker-similarity results using SSIM across these gender combinations.
- The full evaluation includes female-to-female, female-to-male, male-to-female, and male-to-male conversion combinations.
Appendix D Crowdsource Evaluation
The appendix provides MTurk instructions and resource details for four crowdsource evaluations: MOS, DMOS, speaker similarity, and ABX. It also reports HIT assignments, rewards, budgets, and estimated hourly wages for each evaluation.
- Evaluation procedures: The appendix includes MTurk webpage instructions for MOS, DMOS, speaker similarity, and ABX evaluations.These instructions are shown in Figures 10–13, respectively.
- Evaluation procedures: Each evaluation is organized into Human Intelligence Tasks (HITs), the task unit used by MTurk.
- Evaluation costs: MOS used 6,108 HITs with two subjects per HIT, a $0.05 reward, a reported $716 budget, and an estimated $18 hourly wage.
- Evaluation costs: DMOS used 200 HITs, while speaker similarity used 5,160 HITs; both assigned two subjects per HIT and estimated an $18 hourly wage.The reported budgets were $40 for DMOS and $1,032 for speaker similarity, with $0.10 rewards per HIT.
- Evaluation costs: ABX used 150 HITs with five subjects per HIT, a $0.10 reward, a reported $75 budget, and an estimated $12 hourly wage.