Source-linked AI summary
Shared-State Local Translations for Training-Free Voice Conversion
Yangyang Qu, Michele Panariello, Massimiliano Todisco, Nicholas Evans
TL;DR
In one-shot VC, different linguistic content can make reliable source–reference frame correspondence unavailable. StateVC defines shared local states and applies posterior-weighted shifts; under the LibriSpeech one-shot protocol, it leads evaluated systems on recognition errors and speaker-similarity outcomes.
Problem
In one-shot VC, source and reference utterances may differ in linguistic content, so reliable frame-level correspondence cannot be assumed.
Source: §1 · p. 1Method
StateVC pools source and reference representations to define shared states, estimates within-state mean shifts, and combines them using source-frame posterior weights.
Source: §1 · p. 1Results
Takeaways & Limitations
Controlled comparisons show jointly defined states yield lower recognition errors than separately fitted and matched states at nearly identical SIM.
Source: §1 · p. 1, §5 · p. 4The evaluation is limited to English LibriSpeech under the reported one-shot protocol.
Source: §5 · p. 4
Abstract
from arXiv · showhide
In one-shot training-free voice conversion (VC), the source and reference utterances may contain different linguistic content, so reliable frame-level correspondence between them cannot be assumed. We propose StateVC, which jointly defines a common set of local regions from pooled frame-level WavLM representations of the source and reference utterances; we refer to these regions as states. These shared states are obtained by fitting a pair-specific Gaussian mixture model to the pooled representations, without explicit source--reference frame matching. Within each state, StateVC estimates a source-to-reference mean shift in the original WavLM space. Source-frame posterior probabilities then combine the state-specific shifts so that different frames can receive different local updates. For the LibriSpeech one-shot protocol, StateVC achieves the lowest word error rate (WER) and character error rate (CER) among the evaluated systems, at 8.01% and 3.22%, respectively, with a speaker similarity (SIM) of 0.9512. It also achieves the highest mean perceived speaker similarity among the evaluated systems and the highest mean naturalness among the evaluated training-free systems.
1. INTRODUCTION
When source and reference utterances may differ linguistically, reliable frame-level matching is not assumed. StateVC instead uses shared local states and state-specific shifts, achieving leading recognition and speaker-similarity results in the reported evaluation.
1. INTRODUCTION
- Because source and reference may contain different linguistic content, reliable frame-level correspondence is not assumed; prior approaches leave open estimating local changes without pairing frames.Existing methods include explicit frame relations and single or multiple mappings.Source: §1 · p. 1
- StateVC pools source and reference representations to define shared local regions, then uses state-specific mean shifts and source-frame posterior weights for frame-level updates.The shared states are defined without explicit source–reference frame matching.Source: §1 · p. 1, Figure 1 · p. 2
- Jointly defining shared states yields lower WER and CER than defining states separately and matching them afterward.This is a controlled comparison of the two state-definition approaches.Source: §1 · p. 1
- StateVC achieves the lowest WER/CER among evaluated systems, with a SIM of 0.9512, and leads mean perceived speaker similarity and training-free naturalness.These outcomes are reported for the LibriSpeech one-shot protocol.Source: §1 · p. 1
2. METHOD
StateVC uses pooled source and reference WavLM representations to define shared states without matching individual frames. It estimates local source-to-reference shifts and combines them into frame-specific updates using source-frame posterior probabilities.
2.1. Shared-State Definition and Assignment
- A pooled, pair-specific GMM defines shared states from reduced WavLM dimensions, while statistics and shifts are computed in the original representation space.Candidate mixtures with states below a minimum assigned-frame count are discarded; the remaining mixture with the lowest BIC is selected.Source: §2.1 · p. 2
- Frame-to-state posteriors are temporally smoothed over a window of L = 2J + 1 frames to reduce frame-to-frame variation.Out-of-range indices are clipped to the nearest valid frame, and ε ensures numerical stability.Source: §2.1 · p. 2
2.2. Local Shift Estimation
- Smoothed routing probabilities weight full-dimensional source and reference means within each state; their difference gives the local shift when both states have sufficient mass.If either state lacks the minimum posterior mass, StateVC uses the utterance-level mean difference as a fallback.Source: §2.2 · p. 3
2.3. Posterior-Weighted Frame Update
- Each source frame receives a posterior-weighted combination of state-specific shifts, so different frames can receive different updates.The conversion-strength parameter λ ranges from 0 to 1.Source: §2.3 · p. 3
- A frozen WavLM-conditioned HiFi-GAN vocoder reconstructs the converted waveform from the updated representations.The vocoder was released with kNN-VC.Source: §2.3 · p. 3
3. EXPERIMENTAL SETUP
The experiments compare StateVC with learned zero-shot references and training-free systems on matched LibriSpeech source–reference pairs. Objective and listener-rated outcomes are measured alongside analyses of model settings and design choices.
Experimental protocol
- 7,800 cross-speaker pairs from LibriSpeech test-clean are shared across systems, with FreeVC and FreeVC-S as learned references and six training-free comparators.The training-free methods are kNN-VC, LinearVC, MKL-VC, kDOT, USCF, and SSL-GMMVC.Source: §3 · p. 3
Evaluation
- Twenty listeners rate all systems on the same 40 pairs for naturalness and perceived similarity, while x-vector cosine similarity and Whisper-base WER/CER assess speaker similarity and content preservation.Naturalness uses a 1–5 scale and perceived similarity a 1–4 scale.Source: §3 · p. 3
StateVC configuration
- StateVC uses frozen sixth-layer WavLM-Large features and posterior smoothing, with Kmax = 4, λ = 1, and specified minimum-count and posterior-mass thresholds.The minimum assigned-frame count validates a candidate GMM; the minimum posterior mass governs use of a state-specific shift.Source: §3 · p. 3
Design analyses
- The analyses compare state-specific Mean, Diagonal, and Block transformations, separate-state matching, and hard versus soft routing; Fig. 2 also examines conversion strength, candidate-state limit, and reference duration.Fig. 2 uses SIM and WER on the left and right axes, respectively, and marks selected settings.Source: §3 · p. 3, Figure 2 · p. 4
4. RESULTS
StateVC leads the reported LibriSpeech one-shot comparisons on recognition errors and achieves strong speaker-similarity outcomes. Controlled analyses favor shared-state definition, soft routing, and the simpler Mean transformation for the final system.
Overall results
- 8.01% WER and 3.22% CER are the lowest among evaluated systems, with SIM of 0.9512; StateVC also has the highest mean S-MOS overall and highest mean N-MOS among training-free systems.Against MKL-VC, WER changes from 8.22% to 8.01%, CER from 3.37% to 3.22%, and SIM from 0.9456 to 0.9512.Source: §4 · p. 4
State definition
- With Block fixed, shared-state definition reduces WER from 13.82% to 8.44% and CER from 6.60% to 3.48% versus Hungarian-matched separate states, at nearly identical SIM.SIM is 0.9541 for separate matched states and 0.9540 for shared states.Source: §4 · p. 4
Frame-level routing
- Soft routing reduces WER from 9.14% to 8.44% and CER from 3.86% to 3.48%, while SIM decreases slightly from 0.9556 to 0.9540.Source: §4 · p. 4
Transformation and sensitivity analyses
- Mean is selected for the final system: it captures most of the SIM gain over None with a smaller WER increase than Diagonal or Block.WER rises from 8.01% for Mean to 8.36% for Diagonal and 8.44% for Block; longer reference speech mainly improves SIM.Source: §4 · p. 4
5. CONCLUSIONS
StateVC uses jointly defined shared local states and posterior-weighted state-specific mean shifts. Under the reported LibriSpeech one-shot protocol, it achieves the lowest recognition errors among evaluated systems, while the evaluation remains limited to English LibriSpeech.
Conclusion
- 8.01% WER and 3.22% CER are the lowest among evaluated systems, with SIM of 0.9512 under the LibriSpeech one-shot protocol.Shared states yield lower recognition errors than separately fitted and matched states at nearly identical SIM.Source: §5 · p. 4
- StateVC jointly defines shared local states for source and reference representations and applies state-specific mean shifts through posterior-weighted frame updates.Source: §5 · p. 4
- The reported evaluation is limited to English LibriSpeech under the one-shot protocol.Source: §5 · p. 4