Source-linked AI summary
Enhancing Speech Recognition through Text-to-Speech and Voice Conversion Augmentation
Yunus Emre Ozkose, Ali Haznedaroglu
TL;DR
Because large real-world audio datasets can be difficult to obtain, the study evaluates synthetic speech as an ASR training alternative in Turkish call center-banking data. It compares TTS and voice conversion with real audio, finding real audio performs best and Seed-VC is the strongest synthetic approach relative to TTS.
Problem
Extensive real-world audio is resource-intensive to acquire, while evidence on custom-domain data such as Turkish call center speech remains limited.
Source: §1 · p. 1, §3.1 · p. 4Method
The study compares real audio, TTS, and voice-converted speech for ASR, assessing WER and CER and using NISQA to compare speech quality.
Source: §1 · p. 2, Mesauring Generated Data · p. 7Results
Real-world audio achieves the lowest WER and CER, while Seed-VC yields the largest WER and CER reductions among synthetic methods relative to TTS.
Source: §5 · p. 13Takeaways & Limitations
Voice conversion, especially Seed-VC, is a promising way to improve synthetic ASR training data when extensive real data are unavailable.
Source: §1 · p. 2, §5 · p. 13
Abstract
from arXiv · showhide
Given the challenges associated with obtaining large volumes of real-world audio, the advancement of Automatic Speech Recognition (ASR) systems increasingly depends on synthetic data. This study focuses on the call center-banking domain, providing a targeted analysis of ASR models in this context. We evaluate real and synthetic datasets, using Word Error Rate (WER) and Character Error Rate (CER) as metrics, and employ a speech quality model to assess the impact on ASR performance. Our research compares Text-to-Speech (TTS) and advanced voice conversion methods, including KNNVC, Seed-VC, and Vec2Wav2, revealing significant improvements in speech quality and ASR accuracy, particularly with Seed-VC. Additionally, our domain-specific experimentation provides insight into the unique challenges and opportunities that arise when applying ASR technologies to industry-relevant settings. This highlights the potential of voice conversion technologies to enhance ASR systems, guiding future research in diverse linguistic scenarios, and paving the way for the broader application of ASR innovations across various fields.
1 Introduction
Because real-world audio is costly to collect, synthetic data offers an alternative for ASR training, though TTS may not reproduce natural speech nuances. This study compares real audio, TTS, and voice-conversion data, including in the Turkish call center-banking domain.
1 Introduction
- TTS provides a straightforward route to synthetic speech, but may fail to reproduce natural-speech nuances and yield suboptimal ASR performance.Voice conversion methods such as KNNVC, Seed-VC, and Vec2Wav2 aim to make synthetic speech more like real audio.Source: §1 · pp. 1–2
- The study evaluates real-world audio, TTS, and voice-conversion data for ASR training using WER and CER.It also uses a speech-quality model to compare real and generated audio and examine their distributions.Source: §1 · p. 2
- The study aims to inform future synthetic-data research and ASR development in speech technology.Its stated focus is on the practical implications of synthetic-data use.Source: §1 · p. 2
- Experiments focus on the call center-banking domain, offering an industry-specific analysis beyond work centered on Common Voice.The domain presents its own challenges and opportunities for ASR evaluation.Source: §1 · p. 2
- The study compares TTS with three voice-cloning models across speech quality, diversity, and distribution to assess their potential for improving ASR accuracy.The comparison is intended to clarify the strengths and limitations of each approach in replicating real-world audio.Source: §1 · p. 2
2 Related Works
Prior work has advanced neural TTS and voice conversion, while exploring synthetic speech as ASR augmentation. Studies report gains in several settings, but outcomes depend on how speech attributes and real-synthetic mixtures are handled.
2 Related Works
- TTS progressed from two-stage spectrogram-and-vocoder systems to non-autoregressive and end-to-end models, with newer approaches predicting discrete audio tokens from text.The passage cites Tacotron 2, FastSpeech 2 [19], VITS, VALL-E [21], and Spear-TTS.Source: §2 · pp. 2–3
- Voice cloning includes multi-speaker TTS conditioned on a reference-speaker embedding and direct conversion that changes speaker identity while preserving linguistic content.The cited examples include SV2TTS and kNN-VC [3]; AutoVC is described as using an information bottleneck to separate content from speaker identity.Source: §2 · p. 3
- Prior ASR augmentation studies report gains from synthetic speech, while emphasizing speaker or phoneme diversity and warning that naïve real-synthetic mixing can underperform when distributions mismatch.Reported outcomes include significant WER improvements with GST-enhanced synthetic LibriSpeech [11], over 30% absolute WER improvement in Brazilian Portuguese and Russian [4], and up to 35% relative WER reduction on LibriSpeech [15].Source: §2 · pp. 3–4
3 Methodology
The methodology uses Turkish call center data as a real-audio benchmark, alongside speech generated from diverse speakers. It describes ASR architectures, voice-conversion methods and selection criteria, and uses NISQA to compare generated and real speech quality.
3.1 Data
- The study addresses a gap in custom-domain ASR data by using internally collected Turkish call center audio as its real-world benchmark.Common Voice is identified as the dataset most frequently used in ASR augmentation studies.Source: §3.1 · p. 4
- Librispeech, which includes 1,281 speakers, supplies speaker diversity for generated speech samples.The study also generates 20 hours of Azure-TTS speech from authentic-data transcriptions, initially using two randomly selected voices; its test set contains five hours of real speech and no synthetic data.Source: §3.1 · p. 4
3.2 ASR Model
- The ASR-model discussion describes Conformer as combining convolutional layers with transformer attention to capture local features and long-range dependencies.It also characterizes Zipformer as adapting layer complexity to manage computation and memory for real-time and resource-constrained settings.Source: §3.2 · pp. 4–5
3.3 Voice Conversion Models
- KNN-VC converts speaker characteristics by matching source acoustic features to target-speaker neighbors and averaging them while retaining linguistic content.The method is described as data-efficient and not requiring the extensive training typical of deep-learning models.Source: k-Nearest Neighbors Voice Conversion… · p. 5
- Seed-VC is presented as using neural architectures and fine-tuning to reproduce target-speaker attributes from limited input data, while vec2wav2 encodes speech into tokens and resynthesizes it with a BigVGAN vocoder.Selected conversion models are required to be cross-lingual, accurate in speech generation, and capable of working with short reference speech.Source: Seed-VC · pp. 5–6, vec2wav2 · pp. 6–7
- NISQA is used to assess the quality distributions of authentic and generated speech and compare their characteristics.The analysis is intended to inform assessment of synthetic-data fidelity and its potential impact on ASR performance.Source: Mesauring Generated Data · p. 7
4 Experiments
Across Turkish call-center ASR experiments, voice-converted speech more closely matched real-audio quality variability than TTS, and augmentation reduced WER most at lower training durations. The strongest reported 20-hour blend was TTS with Vec2Wav2.
4 Experiments
- Real audio showed greater quality variability and noise, while TTS clustered at high quality and voice-conversion speech more closely matched real-data variability.The figure describes voice-conversion MOS distributions as more naturally distributed than TTS, with Seed-VC and Vec2Wav2 retaining greater variability.Source: Speech Qualities · p. 7, Figure 1 · p. 8
4.1 Real Data and TTS
- At 20 hours, real data scored WER 34.40% and CER 15.33%, versus TTS at WER 97.10% and CER 80.18%; Seed-VC led voice-conversion systems.Seed-VC scored WER 75.53% and CER 43.97%.Source: §4.1 · pp. 8–9, Table 1 · p. 9
4.2 Substituting Real Speech Data with Synthetic Clones via Voice Conversion Models
- Among conversion-only fivefold runs, Seed-VC reached WER 73.99% versus KNNVC’s 82.25%; TTS+Seed-VC reached 74.62%.TTS+Vec2Wav2 reached WER 76.82%.Source: §4.2 · pp. 9–10, Table 2 · p. 11
4.3 Augmenting Real Speech Data with Synthetic Clones
- At 20 hours, TTS+Vec2Wav2 achieved WER 28.65% against the real-data baseline of 34.40%, the strongest reported blended result.The reported relative improvement was approximately 16.74%.Source: §4.3 · p. 10, Table 3 · p. 12
- Other 20-hour blends scored WER 28.98% for TTS+KNNVC, 32.30% for TTS+Seed-VC, and 29.01% for TTS with all clones.Fivefold Vec2Wav2 and KNNVC had less pronounced gains, while fivefold Seed-VC improved its relative figure to 13.63%.Source: §4.3 · p. 11
4.4 Exploring the Impact of Varying Training Durations on Model Performance
- WER gains from augmentation were approximately 19%–21% at 10 hours, 6%–17% at 20 hours, and 1%–3% at 50 hours.At 50 hours, augmented models clustered around 25% WER, close to the real-data baseline.Source: §4.4 · pp. 12–13, Table 4 · p. 13
5 Conclusion
Real-world audio remains the benchmark for ASR performance, while advanced voice conversion—especially Seed-VC—offers a viable way to improve synthetic alternatives. The conclusion highlights this path for resource-constrained environments and calls for further refinement and broader linguistic evaluation.
5 Conclusion
- Real-world audio achieved the lowest WER and CER, establishing it as the benchmark for optimal ASR performance.The authors connect this advantage to authentic speech capturing nuanced variability needed for robust ASR models.Source: §5 · p. 13
- TTS was widely accessible but had higher WER and CER, likely because it inadequately represented natural speech dynamics.Source: §5 · p. 13
- Seed-VC produced the greatest WER and CER reductions among the synthetic approaches relative to TTS, while Vec2Wav2 was effective to a lesser extent.The authors describe Seed-VC as having potential to bridge the performance gap between synthetic and real data.Source: §5 · p. 13
- Sophisticated voice conversion is presented as a viable path to enhancing ASR, particularly in resource-constrained environments.Source: §5 · pp. 13–14
- The authors call for further refinement of synthetic methods and evaluation across diverse linguistic contexts.Source: §5 · p. 14