PaperlayerView exact revision on arXiv

Source-linked AI summary

The TTS-STT Flywheel: Synthetic Entity-Dense Audio Closes the Indic ASR Gap Where Commercial and Open-Source Systems Fail

Venkata Pushpak Teja Menta

arXiv:2605.03073v1cs.CLcs.SD

TL;DR

Niche-domain Indic ASR lacks coverage for entity-dense content such as numbers, currency, addresses, brands, and code-mix. The paper uses a self-contained TTS–STT flywheel to synthesize multilingual entity-dense audio and LoRA-fine-tune Indic ASR. The approach improves entity recovery over open-source baselines across Telugu, Hindi, and Tamil, but remains below preregistered targets and underperforms commercial Hindi ASR.

  • Problem

    Indic ASR systems under-cover entity-dense speech containing digit strings, currency amounts, addresses, brand names, and English/Indic code-mix.

  • Method

    A self-contained flywheel generates approximately 22,000 multilingual entity-dense utterances with open-source and commercial TTS, then applies LoRA adaptation to vasista22 models.

  • Results

    The flywheel beats vasista22 across Telugu, Hindi, and Tamil, reaching EHR 0.473 in Telugu, 0.337 in Hindi, and 0.543 in Tamil; commercial Deepgram remains stronger on Hindi.

  • Takeaways & Limitations

    Targeted synthetic entity-dense data can improve Indic ASR where commercial entity coverage is weak, while gains are language-dependent.

  • Takeaways & Limitations

    The native-speech validation is limited to 20 Telugu utterances from one speaker, leaving cross-speaker and cross-environment generalisation untested.

Abstract

from arXiv · show

Niche-domain Indic ASR -- digit strings, currency amounts, addresses, brand names, English/Indic codemix -- is under-served by both open-source SOTA and commercial systems. On a synthesised entity-dense Telugu test set (held-out by synthesis system), vasista22/whisper-telugu-large-v2 (open SOTA) achieves Entity-Hit-Rate (EHR) 0.027 and Deepgram Nova-3 (commercial) 0.16. We close this gap with a self-contained TTS<->STT flywheel: an open-source Indic TTS pipeline synthesises ~22,000 entity-dense Indic-English code-mix utterances at <$50 marginal cost, and a LoRA fine-tune on top of vasista22 achieves EHR 0.473 on the held-out test (17x over open SOTA, 3x over commercial), with read-prose regression bounded to +6.6 pp WER on FLEURS-Te. Cross-language: beta-Hi 0.337 (7x vs vasista22) and beta-Ta 0.543 (22x vs vasista22, 22x vs Deepgram); on Hindi where Deepgram has substantial entity coverage, the flywheel underperforms commercial. All three beta models fall below pre-registered EHR targets (0.75 for Te, 0.65 for Hi/Ta); we report honestly. A native-human-recorded sanity check (n=20 Telugu) confirms transfer to real speech (beta-Te EHR 0.516 on native vs 0.473 on synth). An EDSA-isolation ablation (LoRA on FLEURS-Te alone) yields EHR 0.020 on the same held-out, attributing ~100% of the gain to the EDSA corpus. We additionally report a language-conditional finding: vanilla Whisper-large-v3 has Telugu-specific Script Collapse (SFR 0.46-0.71) that a per-language LoRA corrects (SFR 0.81-0.97), but the recipe is contraindicated on Hindi and Tamil where vanilla SFR >= 0.98. Code, holdouts, predictions, EDSA corpus, and entity dictionaries are released open-source.

I. Introduction

Entity-dense Indic audio contains phones, pincodes, currency, addresses, brands, and code-mix that read-prose ASR under-covers. A TTS–STT flywheel targets this gap with synthetic data, entity-aware evaluation, and language-conditional script-fidelity adaptation.

  • Motivation: 0.027 EHR for vasista22 and 0.16 for Deepgram Nova-3 reveal a large entity-dense Telugu gap despite read-prose WERs of 0.33 and 0.37.The held-out test is synthesised entity-dense Telugu audio.
  • Contribution: ∼22,000 entity-dense utterances across Telugu, Hindi, and Tamil support a self-contained TTS↔STT flywheel with LoRA adaptation.The pipeline uses open-source TTS for data generation and reports EHR gains over open-source SOTA.
  • Contribution: EHR evaluates entity recovery with class-specific semantic normalisation, addressing mismatches that conventional WER treats as errors.The implementation is deterministic and has 19/19 unit tests passing.
  • Results: The flywheel reaches EHR 0.473 in Telugu, 0.337 in Hindi, and 0.543 in Tamil, beating open-source SOTA by 7–22× and commercial Deepgram in two languages.Hindi is the exception against commercial Deepgram.
  • Results: Per-language LoRA corrects Telugu Script Collapse from SFR 0.46–0.71, but the same recipe is contraindicated on Hindi and Tamil where vanilla SFR ≥0.98.The finding is language-conditional rather than a universal recipe.

III. Method

The method constructs Entity-Dense Synthetic Audio from curated multilingual entity classes, generated carrier utterances, diverse TTS systems, and filtering. Cartesia audio is withheld by synthesis system to evaluate transfer beyond training acoustics.

  • EDSA corpus: Six entity classes cover digits, currency, addresses, brands, code-mix, and proper nouns, with approximately 500 seed entities per language-class cell.Seeds come from Wikidata, AI4Bharat lexicons, and native-speaker curation.
  • EDSA corpus: 22,193 rows survive generation, deduplication, and script-purity filtering across Telugu, Tamil, and Hindi at an Anthropic spend of $13.95.Utterances are conditioned on language, class, and seed entity.
  • EDSA corpus: Digit runs are rewritten into language-specific spelled-out forms after an audit found that synthetic audio could realise labels such as “OTP 54235” as a currency phrase.Approximately 5,174 rows are affected across digit, pincode, and house-or-plot classes.
  • Multi-system synthesis routing: Five TTS systems provide synthesis diversity because a naive single-TTS pipeline can overfit STT to one voice’s acoustic distribution.Routing uses Praxy, Chatterbox, IndicF5, ElevenLabs, and Cartesia.
  • Filtering: ∼10–15% of clips are rejected by a per-class CER filter, leaving ∼19,500 clips and ∼22 audio-hours.The filter uses the same vasista22 models used as experimental baselines.
  • Held-out evaluation: ∼1,270 Cartesia rows per language are withheld, and class-balanced Cartesia subsets of n = 86–102 form the entity-dense evaluation sets.Other synthesis systems remain in the training mix.

C. LoRA fine-tuning recipe

The paper applies language-specific LoRA fine-tuning to Whisper-based models, using different base checkpoints and conservative settings for the already fine-tuned vasista22 models. Training combines real Indic speech with a smaller synthetic EDSA component while excluding Cartesia holdouts.

  • Whisper-large-v3 recipe: Whisper-large-v3 models are fine-tuned per language with rank-16 LoRA adapters targeting attention projections and language-specific decoder prefixes.The recipe uses 6,000 steps and a peak learning rate of 8 · 10^-5.
  • vasista22 recipe: The vasista22 recipe uses 4,000 steps and a peak learning rate of 4 · 10^-5 to reduce catastrophic forgetting of read-prose competence.Cartesia rows are excluded from the training manifest.
  • Training data: Training mixes approximately 40 hours of IndicVoices, 5–30 hours of Common Voice, 10 hours of FLEURS, and 22 hours of EDSA per language.The resulting mix is approximately 70–80% real and 20–30% synthetic, depending on language.

D. Entity-Hit-Rate (EHR) metric

EHR measures reference entity-token recovery with class-specific normalisation, unlike WER’s surface-form sensitivity. The headline macro-EHR weights entity classes equally, while strict matching remains a stated limitation.

  • Definition: EHR measures the fraction of reference entity tokens correctly recovered with class-specific normalisation.This avoids treating semantically equivalent currency forms as automatically different under ordinary WER.
  • Class normalisation: Digit runs and pincodes require NFKC-normalised exact matches, with pincodes additionally requiring length six.These rules define class-specific entity recovery.
  • Class normalisation: Currency matching allows numeric values within ±0.5% after parsing Latin numerals and Indic word multipliers.Brand, proper-noun, spelled-digit, and house-or-plot classes use separate matching rules.
  • Aggregation: Macro-EHR averages per-class EHRs equally, preventing classes with more tokens from dominating headline results.Micro-EHR instead pools entity tokens across classes.
  • Caveat: EHR conservatively misses cross-form equivalents when the hypothesis and reference use different surface renderings, including Latin digits versus Telugu number words.The paper leaves bidirectional currency parsing for a future version.

E. Script Fidelity Rate (SFR)

SFR measures how closely transcript letters remain in the expected language script, complementing WER by exposing script-collapsed outputs. The evaluation spans real and synthesised holdouts, including a Cartesia-held-out entity-dense set.

  • SFR is the fraction of transcript letter characters falling within the expected language script’s Unicode block, excluding whitespace, digits, and punctuation.
  • The evaluation uses FLEURS read-prose utterances, Common Voice volunteer recordings, IndicVoices-General conversational speech, and a Cartesia-held-out entity-dense set.
  • The entity-dense holdout contains 86–102 utterances per language and excludes Cartesia synthesis rows from training to isolate entity-dense capability from synth-system-specific acoustics.

B. Systems benchmarked

The benchmark compares zero-shot, open-source, commercial, and two paper systems, with Praxy-STT-rb evaluated as the headline entity-adapted model. Its Telugu result substantially improves EHR while retaining high script fidelity, although the headline score remains below the preregistered target.

  • Five systems are benchmarked: vanilla Whisper-large-v3, vasista22 open-source SOTA, Deepgram Nova-3, Praxy-STT-r2, and headline Praxy-STT-rb.
  • The headline EHR of 0.473 falls below the preregistered target of ≥0.75, so entity-dense Indic ASR remains substantially open.
  • “—” denotes unrun scorecards or zero-count holdout classes rather than a system failure, depending on the table.
  • 17× and 3× are the reported gap closures over open-source SOTA and commercial systems, respectively, on the Telugu held-out set.
  • 0.473 EHR is achieved by Praxy-STT-rb on Telugu while preserving SFR at 0.928, compared with vasista22 at 0.027 EHR and Deepgram at 0.160.

B. Native human-recorded sanity check

A 20-utterance native-human Telugu sanity check tests whether the synthetic-data gain transfers to natural speech. The reported EHR, WER, and SFR remain comparable or better on native audio, while broader cross-language and read-prose evaluations define the surrounding scope.

  • The native sanity check uses 20 class-balanced Telugu utterances read naturally by one native speaker with a consumer microphone.
  • 0.516 EHR on native speech exceeds the 0.473 EHR on synthesised speech for β-Te, with no reported degradation.
  • Native WER is 0.358 versus 0.324 on synthesis, while SFR is 0.881 versus 0.928, indicating broadly stable transfer across these metrics.
  • 7–22× EHR lifts over vasista22 occur across Telugu, Hindi, and Tamil, while Hindi trails Deepgram’s 0.485 EHR.
  • +6.6 pp absolute WER is the Telugu FLEURS regression, while Hindi and Tamil exceed the preregistered +7 pp threshold on specified read-prose holdouts.

E. Language-conditional Script Collapse fix

The per-language LoRA fixes Telugu Script Collapse but harms Hindi and Tamil, where vanilla Whisper-v3 already preserves script fidelity. The open-source vasista22 baseline remains near-perfect across all three languages.

  • Language-conditional Script Collapse fix: SFR rises from 0.46–0.71 to 0.81–0.97 on Telugu with the per-language LoRA, while WER falls by 1.5×–3.9×.Vanilla Telugu errors include Kannada or Devanagari script emissions, although WER remains above 0.8 on all three holdouts.
  • Language-conditional Script Collapse fix: SFR is ≥0.98 for vanilla Whisper-v3 on every Hindi and Tamil holdout, leaving no Script Collapse to fix.The same LoRA recipe instead regresses these languages.
  • Language-conditional Script Collapse fix: vasista22 sits at approximately 1.0 SFR across Telugu, Hindi, and Tamil, unlike the language-conditional behavior of Whisper-v3 and its LoRA variant.Figure 2 compares vanilla Whisper-v3, Praxy-STT-r2, and vasista22 on CV25.
  • Language-conditional Script Collapse fix: WER regresses by 20–160% relative (+19 to +69 pp absolute) and SFR falls as low as 0.43 on Hindi and Tamil with the same recipe.The paper therefore contraindicates this recipe outside Telugu and recommends first computing vanilla SFR on a small development sample.
  • Open-source versus commercial read-prose: On non-overlapping read-prose holdouts, vasista22 wins or ties Deepgram on three of six relevant cells, while Deepgram wins the other three.CV25-Hi shows the largest stated open-versus-commercial gap: vasista22 0.278 versus Deepgram 0.363.

G. EDSA-isolation ablation

The EDSA-isolation ablation shows that entity-dense recognition comes from the synthetic EDSA training signal, not LoRA adaptation alone. A read-prose-only LoRA remains at the vasista22 baseline floor, while full EDSA-LoRA reaches 0.473 EHR.

  • EDSA-isolation ablation: EHR is 0.020 for the FLEURS-only LoRA control versus 0.473 for full EDSA-LoRA on the same held-out entity-dense Telugu set.The control uses the same LoRA recipe but replaces EDSA with approximately 2,281 read-prose FLEURS-Te clips.
  • EDSA-isolation ablation: The paper attributes approximately 100% of β-Te’s entity-dense gain to the EDSA corpus rather than to the LoRA process.The comparison holds the LoRA recipe fixed while changing the training corpus.
  • EDSA-isolation ablation: The FLEURS-only control is slightly below the 0.027 vasista22 baseline, indicating that LoRA adaptation without entity-dense data does not add entity-recognition capability.Its WER is identical to the vasista22 base at 0.582, so the control does not actively damage read-prose performance.
  • Motivation: Targeted niche-data adaptation addresses the entity-dense gap that remains despite strong read-prose performance from existing systems.The entity-dense holdout reports vasista22 EHR 0.027 and Deepgram EHR 0.16.

B. Why a TTS flywheel beats human-curated entity-dense data

The TTS flywheel offers a substantially cheaper route to entity-dense ASR adaptation than paid human transcription, while exposing language-conditional risks and several evaluation limits. Its practical use depends on diagnosing base-model suitability and expanding validation beyond small or imbalanced holdouts.

  • Cost and data strategy: $16 in generation plus $15 in Modal time produces the EDSA pipeline, versus $660 for 22 hours of paid human transcription.Vendor-ratecard ElevenLabs+Cartesia synthesis would cost approximately $400, but this work used promotional credits.
  • Language-conditional applicability: A 30-utterance vanilla-SFR diagnostic is proposed: apply the LoRA recipe only when SFR is <0.85 on at least two holdouts.The rule reflects Telugu’s under-trained base representations and avoids degrading Hindi and Tamil.
  • Validation boundary: β-Te transfers from synthetic speech EHR 0.473 to native-human Telugu EHR 0.516, but the sanity check uses only 20 utterances from one speaker.The paper identifies cross-speaker and cross-environment validation as the next step.
  • Evaluation limitations: No bootstrap confidence intervals are reported, so per-cell point estimates retain residual variance that is not formally quantified.Directional findings are described as stable across multiple holdouts.
  • Benchmark scope: The study benchmarks only Deepgram commercially, excludes ElevenLabs Scribe and Sarvam STT, and notes residual WER variance from differing post-processing.The exclusions reflect rate limits and uncertain Sarvam API availability at evaluation time.
  • Evaluation limitations: The Cartesia entity-dense holdout contains only 0–2 rows for some entity classes, which are reported as N/A rather than imputed.Future work will explicitly class-balance the held-out set.
  • Ablation scope: Synth-fraction and source-mix LoRA ablations were deferred because 24 additional retrains were unfundable at the available compute budget.The language-conditional applicability ablation was run and its negative Hindi/Tamil result was reported.

VIII. Reproducibility

The project releases code, datasets, predictions, model adapters, and evaluation artifacts needed to reproduce or re-score the reported experiments. Licensing and audited spending are also documented.

  • Released artifacts: Code, holdout JSONLs, predictions JSONLs, the EDSA corpus, and entity dictionaries are released in the stt-flywheel repository.The repository includes evaluation harnesses, ground truths, and per-utterance predictions for every reported system.
  • Released artifacts: Holdout files expose ids, text, audio paths, entity tokens, and entity classes for Telugu, Tamil, and Hindi evaluations.The files are released under CC-BY-4.0.
  • Released artifacts: Per-utterance hypothesis JSONLs enable third-party re-scoring against alternative metrics.These predictions cover every system reported in the paper.
  • Model releases: All six LoRA adapters are released on Hugging Face under Apache-2.0.The release includes both vasista22-base entity-dense adapters and Whisper-v3 language-conditional adapters.
  • Cost transparency: Total audited real spend is approximately $241, including $13.95 for Anthropic generation, approximately $130 for Modal compute, and approximately $5 for Deepgram.ElevenLabs and Cartesia synthesis used free credits.
Loading 2605.03073v1…