VOSSA: Voiceprint Optimization for Streaming Speech Architectures

1. Introduction

• ASV embeddings suppress within-speaker phonetic and prosodic variation, creating a mismatch with frame-level phoneme-conditioned acoustic generation.Jointly trained speaker encoders can reduce that mismatch but add a separate branch, increasing model complexity and memory footprint.Source: §1 · p. 1
• VOSSA derives speaker embeddings from intermediate content-encoder layers and jointly optimizes them through reconstruction and cross-speaker conversion objectives.Reusing content representations removes the separate speaker encoder while preserving strict causality and low latency.Source: §1 · p. 1
• Across six datasets, VOSSA shows richer pitch variability and better preservation of speaker- and vowel-dependent formant distributions while maintaining comparable NISQA-MOS, WER, and harmonicity.Target-speaker similarity improves under normalized embedding evaluation.Source: §1 · p. 1
• Figure 1 contrasts same-speaker LibriTTS self-reconstruction with non-parallel VoxCeleb-target conversion and depicts embedding extraction from the frozen content encoder.The embedding aggregates selected encoder-layer features using attentive statistics pooling and an MLP.Source: Figure 1 · p. 2

2. Methods

• TVTSyn supplies VOSSA’s streaming backbone, including a causal content encoder and decoder with 80 ms look-ahead.The content encoder produces quantized frame embeddings, while the decoder reconstructs waveforms.Source: §2 · p. 2

2.1. Joint speaker representation learning

• VOSSA extracts features from selected content-encoder layers and uses attentive statistics pooling to form a global speaker embedding.The selected features include the last CNN layer and every other layer of the eight-layer MHSA stack; a projection network maps the pooled mean and variance to the embedding.Source: §2.1 · p. 2

2.2. Dual-path speaker-aligned training

• Training pairs self-reconstruction of a speaker’s utterance with conversion of the same content to a second speaker.The conversion path uses the target speaker’s average embedding as its identity target.Source: §2.2 · p. 2
• Speaker-consistency and supervised contrastive losses complement reconstruction, adversarial, and feature-matching objectives.Gradients pass through the speaker encoder, TVT module, and decoder to align the learned speaker space with streaming VC.Source: §2.2 · pp. 2–3

3.1. Voice conversion protocol and diagnostics

• The voice-conversion evaluation uses LibriTTS source utterances and target speakers from LibriTTS, VoxCeleb, EMIME, ARCTIC, L2-ARCTIC, and VCTK.All systems use identical source–target pairs; the evaluation reports normalized speaker similarity, WER, NISQA-MOS, and HNR.Source: §3.1 · p. 3
• VOSSA matches TVTSyn in WER and has comparable NISQA-MOS, while achieving significantly higher normalized target similarity and higher HNR than most baselines.The comparison also includes GenVC-s, which VOSSA outperforms on NISQA-MOS.Source: §3.1 · pp. 3–4

3.2. Self-reconstruction protocol and diagnostics

• VOSSA has the lowest pitch MAE and highest PCC across systems, with V/UV mismatches close to slt24 and significantly below GenVC.The self-reconstruction evaluation uses LibriTTS and compares input with reconstructed speech.Source: §3.2 · p. 4
• VOSSA achieves the lowest Wasserstein distance for F1 distributions of high, mid, and low vowels across the baselines.Figure 2 illustrates the comparison with TVTSyn for one target speaker; the analysis measures F1 at vowel midpoints.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Perceptual evaluations

• In listening tests, VOSSA receives the highest MOS among systems, while original speech remains substantially higher.The MOS test used 20 listeners and 15 utterances per model.Source: §3.3 · p. 4
• Listeners prefer VOSSA to TVTSyn for speaker similarity, intelligibility, and vocal vibrancy.Speaker similarity was tested with ABX choices; intelligibility and vibrancy were assessed with AB preference tests.Source: §3.3 · p. 4, Table 2 · p. 4

3.4. Model size and streaming efficiency

• VOSSA uses 19% fewer parameters than TVTSyn, while both systems reach approximately 0.25 RTF and about 73 ms end-to-end latency under the reported streaming setup.The measurement used batch size 1, 16 kHz audio, one RTX 5000 Ada GPU, and 60 ms chunks.Source: §3.4 · p. 4

• VOSSA reduces model size by 19% while maintaining comparable streaming efficiency.It has fewer parameters than TVTSyn, with similar real-time factor and end-to-end latency.Source: §4 · p. 4
• It attains the highest normalized target-speaker similarity and is preferred in ABX speaker-similarity tests.Source: §4 · p. 4
• Vowel-level evaluation finds reduced Wasserstein distance to ground-truth F1 distributions across vowel heights.The passage interprets this as evidence of speaker-dependent phonetic acoustics beyond overall timbre.Source: §4 · p. 4
• VOSSA achieves lower pitch error and V/UV mismatch while preserving high contour correlation.The passage describes this pattern as more accurate and dynamically consistent F0 modeling.Source: §4 · p. 4

• LLMs were used only minimally.Source: §6 · p. 5
• Their stated role was improving writing clarity and presentation.Source: §6 · p. 5
• The authors state that they conducted the experimental design, implementation, data analysis, and interpretation of results.Source: §6 · p. 5