PaperlayerView exact revision on arXiv

Source-linked AI summary

Building and Evaluating a Synthetic Bengali Speech Resource for Telecom Customer Care

Kawshik Kumar Paul, Md. Nafiul Alam Fuji

arXiv:2608.20346v1cs.CLcs.SDeess.AS

TL;DR

Domain-specific telecom speech systems need reliable coverage of repeated Bengali expressions and intents. This paper constructs and evaluates a synthetic Bengali telecom customer-care speech resource, finding average WER of 2.54% and CER of 0.59% under its automatic evaluation pipeline.

  • Problem

    Telecom speech systems need reliable coverage of domain-specific Bengali expressions, vocabulary, and user intents.

  • Method

    The paper constructs, documents, publicly releases, and evaluates a synthetic Bengali telecom customer-care speech dataset with original and normalized transcripts.

  • Results

    Average WER was 2.54% and average CER was 0.59%, with median WER and CER both 0.00%.

  • Takeaways & Limitations

    The dataset shows strong text-audio consistency under the selected automatic evaluation pipeline.

  • Takeaways & Limitations

    Because the resource is entirely synthetic, models trained only on it may not fully generalize to real-world speech and phone-call conditions.

Abstract

from arXiv · show

Speech systems used in customer-facing applications often require domain-specific language coverage. We present a synthetic Bengali speech dataset for telecom customer-care scenarios. The dataset contains 10,000 audio-text pairs, approximately 26.82 hours of 24 kHz speech, and predefined train, validation, and test splits of 9,000, 500, and 500 examples. It is publicly released on Hugging Face under the CC-BY-4.0 license. The speech was generated with OmniVoice in voice-cloning mode using a real female reference recording and transcript, with bfloat16 precision, 16 diffusion sampling steps, and a speaking-rate control value of 1.0. Along with the original Bengali text, the dataset provides a normalized transcript field designed for ASR/STT training and evaluation. We report an automatic intelligibility check over all 10,000 samples using a domain-adapted Whisper ASR model fine-tuned from bengaliAI/tugstugi_bengaliai-regional-asr_whisper-medium, along with a manual listening check on selected samples. The evaluation gives an average WER of 2.54%, an average CER of 0.59%, and median WER and CER values of 0.00%. These results suggest strong text-audio consistency under the selected automatic evaluation pipeline, while the paper also discusses the limitations of synthetic speech and STT-based evaluation.

1 Introduction

The paper introduces and releases a 10,000-sample synthetic Bengali speech resource for telecom and customer-care scenarios, with text fields supporting TTS and ASR/STT workflows. It documents OmniVoice-based generation, evaluates intelligibility with fine-tuned Whisper, and discusses practical and ethical limitations.

  • Motivation: Telecom customer-care speech systems need reliable handling of repeated domain-specific vocabulary, phrasing, and user intents.Examples include failed recharges, OTP delays, package activation, balance mismatches, blocking, refunds, registration, and mobile banking verification.
  • Dataset contribution: 10,000 synthetic Bengali speech samples focused on telecom and customer-care utterances are publicly released on Hugging Face.The dataset includes original generation text and normalized text intended for ASR/STT training and evaluation.
  • Generation and scope: The contribution is resource construction, release, documentation, and evaluation using OmniVoice rather than a new TTS architecture.The generation setup includes voice-cloning mode, bfloat16 precision, 16 diffusion sampling steps, and 24 kHz output audio.
  • Dataset contribution: Both original and normalized Bengali text fields support TTS and ASR/STT workflows.The normalized field is designed for ASR/STT training and evaluation, while the original text was used for speech generation.
  • Evaluation and limitations: All 10,000 samples are evaluated with a fine-tuned Whisper-based STT intelligibility check reporting WER/CER summaries.The paper also discusses practical limitations and ethical considerations of synthetic voice-cloned speech.

2 Related Work

The work situates its dataset among public and synthetic speech resources while distinguishing it as synthetic, domain-specific Bengali telecom/customer-care speech. It also uses OmniVoice for generation and a domain-adapted Whisper evaluator, while reporting both WER and CER with acknowledged evaluation limitations.

  • Open speech corpora: Common Voice exemplifies how public, crowdsourced multilingual speech corpora support automatic speech recognition research and speech-system development.Its speech collection and validation are crowdsourced.
  • Dataset positioning: The dataset differs from broad-domain human-recorded corpora because it is synthetic and targets Bengali telecom and customer-care speech.These are the two major distinctions identified in the related work.
  • Synthetic speech resources: Like CVSS, the work uses synthetic speech as a research resource, but targets Bengali telecom/customer-care speech rather than speech translation.It provides paired audio, original text, and normalized transcripts for speech experiments.
  • Generation model: OmniVoice provides the generation backend, supporting massively multilingual zero-shot TTS and voice cloning from a reference recording.The model scales to more than 600 languages and maps text to multi-codebook acoustic tokens.
  • Evaluation setup: The automatic evaluator is a domain-adapted Whisper model fine-tuned for Bengali telecom/customer-care speech, initialized from bengaliAI/tugstugi_bengaliai-regional-asr_whisper-medium.WER and CER are both reported because character-level evaluation can help address multilingual tokenization and writing-system differences.
  • Evaluation limitations: The evaluator serves only as an automatic proxy for text-audio consistency, so its WER/CER values are not a fully independent benchmark of the dataset.The evaluator was adapted to the target language and domain.

3 Dataset Overview

The dataset provides synthetic Bengali speech generated from telecom and customer-care prompts, with intended uses spanning Bengali ASR/STT, TTS, preprocessing, and domain adaptation. It is distributed in train, validation, and test splits with streamlined metadata for standard experiments, while evaluation artifacts are reported separately.

  • Dataset purpose and uses: The dataset contains synthetic Bengali speech generated from telecom and customer-care style prompts.Its dataset-card intended uses include Bengali ASR/STT, Bengali TTS, speech-to-text preprocessing, telecom/customer-care domain adaptation, and synthetic speech research.
  • Dataset splits: The dataset is distributed with train, validation, and test splits.The passage indicates that split sizes are documented separately in Table 2.
  • Metadata structure: Each metadata row contains the fields defined in Table 3, with a deliberately simple structure for direct loading.The metadata supports TTS, ASR/STT, and preprocessing experiments.
  • Metadata structure: Per-sample runtime, STT transcripts, WER, CER, and inference-time logs are excluded from the main metadata as evaluation artifacts.Aggregate evaluation results are reported separately in Section 6, while the released metadata retains fields needed for standard TTS and ASR/STT use.

4 Text Preparation and Normalization

The dataset separates original Bengali text from a normalized transcript to serve the differing requirements of TTS and ASR/STT. Rule-based cleanup reduces evaluation mismatches caused by formatting and written-form differences that may not reflect audible differences.

  • Text fields: The dataset provides original text and a cleaner normalized transcript for ASR/STT training and evaluation.The text field contains the Bengali sentence used for speech generation, while text_normalized stores its normalized transcript form.
  • Rationale: Normalization reduces metric inflation from punctuation, spacing, abbreviation, and Bengali spelling differences that may not affect intelligibility.These written-form differences can cause text-matching mismatches even when the audio is understandable.
  • Usage convention: The recommended convention uses text as the TTS input and text_normalized as the ASR/STT target transcript.The paper follows this convention in its evaluation section.
  • Normalization method: Rule-based cleanup standardizes punctuation, spacing, abbreviations, Bengali number-word variants, and common written forms in borrowed telecom terms.The procedure includes collapsing repeated punctuation and removing unnecessary spacing.

5 Synthetic Speech Generation

The dataset consists entirely of synthetic speech generated with OmniVoice in voice-cloning mode using a real female reference recording and transcript, rather than real customer-care or user audio.

  • Generation method: All speech is synthetic and was generated with OmniVoice in voice-cloning mode using a real female voice recording and transcript as the reference.The reference recording was used for voice cloning, not as dataset audio.
  • Data source limitations: The dataset contains no real customer-care recordings, customer conversations, or real user audio.

6 STT-based Intelligibility Evaluation

The study evaluates all 10,000 generated samples with a fine-tuned Bengali telecom-domain Whisper model, using WER and CER as automatic text-audio consistency checks. Results show low error rates, while manual inspection and stated limitations caution against treating them as naturalness or sample-quality ratings.

  • Evaluation procedure: 10,000 samples were evaluated by transcribing each generated audio file and comparing it with the corresponding text_normalized field.The evaluator was a fine-tuned Tugstugi Whisper model adapted to the Bengali telecom/customer-care setting.
  • Interpretation and limitations: WER/CER results are automatic proxy indicators of text-audio consistency under this adapted evaluation pipeline, not direct labels of sample quality or proof of naturalness across deployment conditions.Human listening tests would be needed to measure perceived naturalness, speaker similarity, and listener preference, and full reproducibility would require the fine-tuning data, decoding configuration, and normalization script.
  • Manual inspection: Manual listening of selected samples served as a qualitative sanity check for mismatches, pronunciation problems, incomplete speech, and unnatural repetition, not a formal MOS study.No MOS or listener-preference scores were reported; manual inspection additionally suggested some mismatches reflected written-form differences.
  • Automatic results: 2.54% average WER and 0.59% average CER were reported, with both median WER and CER at 0.00%.WER counts word-level errors, whereas CER counts character-level errors; both use substitutions, insertions, and deletions.
  • Automatic results: 0.00% median WER and CER indicate that at least half of the samples matched exactly after normalization, while non-zero averages reflect mismatches in a smaller subset.The low average CER indicates strong character-level agreement, while WER is higher because word-level mismatches can have a stronger effect.

7 Use Cases

The dataset supports Bengali ASR/STT and TTS experiments through audio-text mappings, with applications in telecom/customer-care domain adaptation and synthetic speech research.

  • ASR and STT: ASR/STT users can train or evaluate Bengali systems using audio →text_normalized, which reduces formatting mismatch in target transcripts.The normalized transcript field is recommended for ASR/STT training and evaluation.
  • Text-to-speech: TTS users can use text →audio for controlled experiments, bootstrapping, or domain adaptation because the speech is synthetic.The dataset provides a direct text-to-audio mapping for TTS experiments.
  • Telecom and customer-care domain adaptation: Telecom/customer-care utterance patterns support domain-adaptation experiments where broad-domain speech lacks sufficient specialized phrasing.The dataset focuses on customer-care style language in the telecom domain.
  • Synthetic speech research: The explicitly synthetic dataset supports research on synthetic speech utility, ASR robustness to generated speech, and synthetic-to-real transfer.A generation disclosure accompanies the synthetic speech resource.

8 Limitations

The synthetic Bengali speech dataset has limitations in real-world generalization, speaker diversity, and the interpretation and reproducibility of automatic STT-based evaluation metrics.

  • Synthetic data: Synthetic data may not generalize fully to real-world speech, noisy phone calls, spontaneous conversation, different microphones, accents, or background noise.The dataset contains no real customer-care recordings, conversations, or user audio; production systems should combine it with real human-recorded Bengali speech.
  • Speaker diversity: Voice cloning from one real female reference recording limits speaker diversity compared with a multi-speaker corpus.Future versions could add multiple reference voices or voice-design variants subject to clear consent and licensing.
  • STT-based evaluation: Automatic WER and CER estimate text-audio consistency but do not measure all audio-quality aspects, including naturalness.Low STT error can coexist with unnatural audio, while understandable samples may receive higher errors because of ASR limitations.
  • Metric reproducibility: Different normalization pipelines can change WER and CER values, making normalization rules and the evaluation script part of reproducibility.Metric values should therefore be interpreted alongside the specific normalization procedure used.

9 Ethical Considerations

The synthetic dataset avoids direct privacy risks associated with releasing real customer-service calls, but its voice-cloning origin requires consent-aware use and safeguards against misuse. Production applications should supplement synthetic data with realistic speech evaluation and user-safety checks.

  • Privacy: Synthetic audio avoids releasing real customer-care recordings, conversations, or user audio, reducing direct privacy risks.The dataset card explicitly states that all samples are synthetic and contain no real customer-care or user recordings.
  • Voice cloning: The cloned voice used a permitted real female reference recording and transcript, requiring attention to consent, speaker rights, impersonation, and misuse.The dataset should not be used for deception, speaker impersonation, fraud, or misrepresentation of a real person.
  • Production use: Production systems should evaluate synthetic speech alongside realistic speech data and appropriate user-safety checks.Synthetic data may support research and development, but production deployment requires additional evaluation safeguards.

10 Conclusion

The paper presents a publicly released synthetic Bengali telecom customer-care speech resource with bilingual transcript formats and strong text-audio consistency under its selected automatic evaluation pipeline. It also identifies future work to broaden evaluation, speaker diversity, generation comparisons, and integration with real speech.

  • Resource contribution: 10,000 synthetic audio-text pairs provide approximately 26.82 hours of 24 kHz Bengali speech for telecom customer-care scenarios.The resource includes original Bengali text and normalized transcripts, supporting TTS and ASR/STT experiments.
  • Release and evaluation: The resource was generated with OmniVoice voice cloning and publicly released on Hugging Face under the CC-BY-4.0 license.All 10,000 samples underwent a fine-tuned Tugstugi Whisper-based intelligibility check.
  • Release and evaluation: 2.54% average WER and 0.59% average CER were obtained, with median WER and CER values of 0.00%.These results came from the selected automatic evaluation pipeline applied to all 10,000 samples.
  • Future work: Future work includes releasing evaluation artifacts, adding speaker diversity, comparing voice-cloning with voice-design generation, conducting human listening tests, and combining synthetic with real Bengali speech.The stated goal is more robust speech system development.
Loading 2608.20346v1…