Source-linked AI summary
AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization
Yen-Hao Chen, Da-Yi Wu, Tsung-Han Wu, Hung-yi Lee
TL;DR
One-shot voice conversion remains challenging, with prior approaches presenting problems such as redundant encoders and damaged content information. AGAIN-VC combines a single encoder with activation guidance, and human evaluations support the effectiveness of both choices.
Problem
One-shot VC remains challenging because prior systems may use redundant encoders or damage content information through discrete representations.
Source: §1 · p. 1Method
AGAIN-VC uses a single encoder for speaker and content representations and activation guidance as an information bottleneck on continuous content embeddings.
Source: §1 · p. 1Results
Human evaluations support the effectiveness of using a single encoder and activation guidance, and the proposed method outperforms 1-Enc in naturalness and similarity.
Source: §4.3 · p. 4, §5 · p. 4Takeaways & Limitations
The authors report that AGAIN-VC improves the trade-off between synthesis quality and disentangling ability.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Recently, voice conversion (VC) has been widely studied. Many VC systems use disentangle-based learning techniques to separate the speaker and the linguistic content information from a speech signal. Subsequently, they convert the voice by changing the speaker information to that of the target speaker. To prevent the speaker information from leaking into the content embeddings, previous works either reduce the dimension or quantize the content embedding as a strong information bottleneck. These mechanisms somehow hurt the synthesis quality. In this work, we propose AGAIN-VC, an innovative VC system using Activation Guidance and Adaptive Instance Normalization. AGAIN-VC is an auto-encoder-based model, comprising of a single encoder and a decoder. With a proper activation as an information bottleneck on content embeddings, the trade-off between the synthesis quality and the speaker similarity of the converted speech is improved drastically. This one-shot VC system obtains the best performance regardless of the subjective or objective evaluations.
1. INTRODUCTION
One-shot voice conversion remains challenging because existing approaches can use redundant encoders or damage content information. AGAIN-VC proposes a single-encoder system with activation guidance to improve the synthesis-quality/disentangling trade-off.
1. INTRODUCTION
- One-shot VC remains challenging: AdaIN-VC uses two independent encoders, while VQVC+ seriously damages content information through discrete representations.The paper also notes that AutoVC’s speaker encoder is pretrained only for speaker verification, making generation robustness questionable.Source: §1 · p. 1
- AGAIN-VC is proposed to improve the trade-off between synthesis quality and disentangling ability.The method is named Activation Guidance and Adaptive Instance Normalization.Source: §1 · p. 1
- A single encoder is used to disentangle speaker and content information, with the stated aim of improving synthesis quality while reducing model size.Source: §1 · p. 1
- Activation guidance applies an additional activation as a continuous-space information bottleneck on content embeddings to improve the synthesis-quality/disentangling trade-off.Source: §1 · p. 1
2. METHODS
AGAIN-VC uses one encoder to extract speaker and content representations, then transfers style through AdaIN. Activation guidance constrains content embeddings to reduce speaker-information leakage while preserving reconstruction performance.
2.1. System overview
- The single encoder reuses channel-wise IN means and standard deviations as speaker embeddings, alongside an activation-guided content representation.The model is trained with self-reconstruction loss.Source: §2.1 · p. 2
2.2. Style transfer using AdaIN
- AdaIN transfers the style of target representation Z to source representation H while retaining H’s content.Encoder IN layers remove global speaker information; skip connections pass layerwise μ and σ to corresponding decoder AdaIN layers.Source: §2.2 · p. 2
2.3. Activation guidance (AG)
- Activation guidance adds an activation bottleneck to content embeddings, restricting their range to discourage speaker information leakage without the reconstruction harm attributed to harsh activations.The paper states that time-invariant μ and σ cannot represent time-varying content, guiding content information to the content embeddings.Source: §2.3 · pp. 2–3
3. IMPLEMENTATION DETAILS
Implementation uses VCTK speaker splits and 1.5-second mel-spectrogram segments, trains with ADAM for 100k steps, and uses pre-trained MelGAN to convert generated mel-spectrograms into waveforms.
3.1. Dataset and data preprocessing
- VCTK provides 109 English speakers; 80 were randomly selected for training, with 200 utterances per speaker and remaining speakers reserved for evaluation.Audio was resampled to 22050 Hz, trimmed, and converted to 80-bin mel-spectrograms using a 1024-point STFT window and 256-sample hop, following MelGAN configuration.Source: §3.1 · p. 3
3.2. Training details
- Training used ADAM with a 0.0005 initial learning rate, β1 = 0.9, β2 = 0.999, batch size 32, and 100k steps.Source: §3.2 · p. 3
3.3. Vocoder
- Pre-trained MelGAN converts the model’s output mel-spectrograms into waveforms, chosen for waveform-generation quality and inference speed.Source: §3.3 · p. 3
4. EXPERIMENTS
The experiments report that the proposed model performs best across objective metrics and outperforms encoder baselines in subjective evaluations, while examining activation guidance and encoder count.
4.1. Evaluation on activation guidance
- A speaker classifier probes identity from content embeddings to assess disentangling, alongside reconstruction error as a measure of speech quality.Both measures should be low, but the experiments describe a trade-off between them.Source: §4.1 · p. 3, §4.1.1 · p. 3
4.1.1. Effect of activation functions
- With sigmoid activation guidance, most channel-size settings shift toward lower speaker-classification accuracy but slightly higher reconstruction error; models below 70% accuracy are preferred.The authors report that models above 70% speaker-classification accuracy usually fail to convert audio clips.Source: Figure 3 · p. 3, §4.1.1 · p. 3
4.1.2. Different activation functions
- Sigmoid with α = 0.1 is reported to filter speaker information from content embeddings while yielding the lowest reconstruction error; ELU and ReLU impair performance.Speaker-classification accuracy on speaker embeddings exceeds 90% across activation functions, while ReLU has slightly better probing accuracy on content embeddings but more reconstruction error.Source: §4.1.2 · pp. 3–4, Table 1 · p. 4
4.2. Single encoder
- The proposed 1-Enc-sig model obtains the best performance under all evaluation metrics, while single-encoder models are about 30% smaller than two-encoder models and have almost identical performance.Source: §4.2 · p. 4
4.3. Subjective evaluations
- In subjective tests, 1-Enc outperforms 2-Enc on naturalness and similarity, and the proposed method outperforms 1-Enc on both.Naturalness was measured with MOS; the similarity test asked participants to choose which converted speech better matched a target reference.Source: Figure 4 · p. 4, §4.3 · p. 4
5. CONCLUSION
AGAIN-VC is a one-shot voice-conversion system with a single encoder and activation guidance. The conclusion reports that one encoder suffices for feature disentanglement, activation guidance prevents information leakage, and human evaluation supports both choices.
- Human evaluation supports the effectiveness of using a single encoder and activation guidance in AGAIN-VC.Source: §5 · p. 4
- The results indicate that a single encoder is sufficient for feature disentanglement.Source: §5 · p. 4
- Activation guidance acts as an information bottleneck that prevents information leakage.Source: §5 · p. 4