Source-linked AI summary
Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition
Kush Juvekar, Kavya Manohar, Aditya Srinivas Menon, Arghya Bhattacharya, Kumarmanas Nethil
TL;DR
Low-resource Indic ASR fine-tuning can improve read speech while degrading spontaneous conversational performance. The paper introduces a complexity-tiered benchmark and controlled optimization study, finding that early aggressive updates and, for some settings, hard-to-easy curricula improve robustness while enabling smaller models to match or exceed larger baselines.
Problem
Fine-tuned Indic ASR models perform well on studio-recorded read speech but degrade sharply on spontaneous conversational speech, a mismatch termed studio-bias.
Method
The study introduces Vividh-ASR for Hindi and Malayalam across acoustic-complexity tiers, decouples learning-rate timing and curriculum ordering in a controlled 2 × 2 factorial study, and analyzes representations with CKA and SVD.
Results
Early aggressive learning rates yield ∼12 absolute WER points, while reverse hard-to-easy curricula provide smaller, language-dependent gains, especially for spontaneous Malayalam speech.
Takeaways & Limitations
R-MFT enables a 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts while adapting the decoder and preserving the encoder’s acoustic geometry.
Takeaways & Limitations
The analysis assumes successful adaptation requires learning new linguistic priors in the decoder while preserving the pre-trained encoder’s acoustic invariance.
Abstract
from arXiv · showhide
Fine-tuning multilingual ASR models like Whisper for low-resource languages often improves read speech but degrades spontaneous audio performance. To diagnose this mismatch, we introduce Vividh-ASR, a complexity-stratified benchmark for Hindi and Malayalam across four tiers: studio, broadcast, spontaneous, and synthetic noise. Through a controlled study of learning-rate timing and curriculum ordering, we find that early large parameter updates improve global WER by 12 absolute points, while a hard-to-easy curriculum adds gains for spontaneous speech. These findings motivate reverse multi-stage fine-tuning (R-MFT), a training recipe that enables a parameter-efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts. Representational analysis via CKA and SVD reveals effective schedules concentrate adaptation in the decoder, preserving the pre-trained encoder's acoustic geometry. We release the benchmark and models.
1. Introduction
Vividh-ASR addresses studio-bias in low-resource Indic ASR by testing learning-rate timing and curriculum ordering under controlled conditions. The study finds that reversing standard fine-tuning heuristics improves robustness, especially when high-magnitude updates begin with difficult data.
- Studio-trained fine-tuned models perform well on clean read speech but degrade sharply on spontaneous conversational audio, termed studio-bias.
- Current practice commonly uses conservative 1e−5 learning rates and easy-to-hard curricula to limit catastrophic forgetting.
- A controlled 2×2 factorial study isolates learning-rate timing and curriculum direction while holding training data, architecture, and optimizer configuration constant.
- High-magnitude updates applied initially on the hardest data yield drastic WER improvements, whereas conservative initialization traps the model in a sub-optimal basin.
- R-MFT combines spontaneous-first ordering with high initial learning rates, and CKA/SVD analyses examine how these schedules reshape model representations.
2. Related Work
Prior Indic ASR resources span read, broadcast, and spontaneous speech, while existing benchmarks commonly organize evaluation by domains. Vividh-ASR instead emphasizes acoustic complexity, and this work studies curriculum direction during multilingual-model fine-tuning.
- Kathbath, Shrutilipi, and Indic Voices provide read speech, broadcast news, and crowdsourced spontaneous speech resources, respectively.
- Existing benchmarks such as Vistaar evaluate models across multiple domains, whereas Vividh-ASR isolates performance along an acoustic-complexity axis.
- Curriculum learning in ASR typically exposes models to progressively noisier data, while anti-curriculum strategies remain underexplored for large pre-trained speech models.
- Prior representation studies often examine frozen speech models, leaving fine-tuning schedules’ internal effects less directly characterized through downstream WER alone.
3. The Vividh-ASR Benchmark
Vividh-ASR is a diagnostic benchmark for Hindi and Malayalam organized by acoustic and prosodic complexity rather than domain. Its four tiers progress from controlled studio speech to synthetic noise, with spontaneous speech as the primary real-world bottleneck.
- Vividh-ASR targets Hindi and Malayalam and aggregates data from multiple publicly available speech corpora.
- The benchmark uses four complexity tiers: studio, broadcast, spontaneous, and synthetic noise.
- Tier C contains unscripted recordings with disfluencies, varying prosody, background noise, and non-professional hardware, making it the primary real-world bottleneck.
- The corpus is intentionally weighted toward Tier C, while Tier D is evaluation-only.
4. Methodology
The methodology systematically varies learning-rate timing and curriculum direction, then uses the resulting findings to define R-MFT. Training evaluates Whisper models while reserving synthetic-noise data for evaluation.
- Standard Whisper fine-tuning commonly uses a conservative 1e−5 learning rate because large updates are assumed to risk damaging pre-trained priors.
- The study compares decreasing 2e−4 →1e−4 →1e−5 and increasing 1e−5 →1e−4 →2e−4 learning-rate schedules.
- It also compares easy-to-hard Tier A →Tier B →Tier C ordering with hard-to-easy Tier C →Tier B →Tier A ordering.
- R-MFT pairs a high initial learning rate with spontaneous-first training, using Tier C at 2e−4 followed by broadcast and easier stages.
- Experiments evaluate Whisper-small and Whisper-medium with AdamW, warmup followed by cosine annealing, batch size 128, and gradient checkpointing.
- Tier D synthetic-noise data is held out from training and validation, making it evaluation-only.
5. Results
Results show that early high learning rates matter more than curriculum direction, while hard-to-easy ordering provides an additional Malayalam robustness benefit. The R-MFT recipe also achieves strong performance with substantially fewer parameters.
- 5.1. Learning Rate Effect: An order-of-magnitude higher loss results from the conservative 1e−5 schedule plateauing within 7K steps, versus 2e−4.The trend is reported for Malayalam Whisper-medium and is identical for Hindi and Whisper-small.
- 5.2. Overview: Both high-LR strategies outperform the conservative single-stage low-LR baseline, whose global WER is 77.79% for Malayalam and 25.25% for Hindi.R-MFT is best for Malayalam at 39.36%, while single-stage high LR is slightly better for Hindi at 16.67% versus 18.82% for R-MFT.
- 5.3. Effect of Learning Rate Timing: A low-LR start produces a consistent ∼13-point Malayalam penalty across curricula because later high-LR stages cannot reverse the resulting representational trajectory.The locking effect is most severe under R-MFT, where low-LR initialization causes an initial failure from which the model does not fully recover.
- 5.3. Effect of Learning Rate Timing: High-to-low schedules converge to ∼18.8% global WER for Hindi, improving from 28.49% under standard and 22.71% under reverse low-to-high schedules.The result is reported regardless of curriculum direction, supporting the importance of applying high-magnitude updates at the outset.
- 5.4. Effect of Curriculum Ordering: Hard-to-easy ordering reaches 39.35% Malayalam WER versus 42.25% for easy-to-hard ordering when the LR schedule is held constant.The ∼3-point gain is smaller than the ∼13-point timing effect and is attributed to adapting first to spontaneous disfluencies and non-professional acoustics.
- 5.5. Parameter Scale: The 244M R-MFT model achieves 44.41% Malayalam and 21.41% Hindi global WER, outperforming a 769M low-LR baseline by 33.38 and 3.84 absolute points.The smaller model also exceeds the 769M IndicWhisper baseline in both languages.
6. Analysis
The analysis finds that successful adaptation shifts substantially in the decoder while preserving the encoder’s pre-trained acoustic geometry. IndicWhisper instead disrupts encoder representations, a pattern associated with poorer spontaneous-speech performance.
- Decoder vs. encoder adaptation: Successful high-LR adaptation concentrates substantial parameter displacement in the decoder, with mean ∆θ = 0.122 for R-MFT.Decoder activation distributions also shift, with EMD = 0.069, indicating remapping of the linguistic prior.
- Optimization dynamics: The conservative low-LR baseline produces only ∆θ ≈0.01, insufficient for meaningful adaptation.This contrasts with the larger decoder displacement observed for successful high-LR schedules.
- Decoder vs. encoder adaptation: Encoder CKA remains 1.000 with near-zero EMD for Baseline-High and R-MFT, preserving the pre-trained acoustic geometry.The analysis reports that the aggressive initial learning rate is 2e−4.
- Decoder vs. encoder adaptation: IndicWhisper has lower overall displacement than R-MFT (∆θ = 0.025) but reduces encoder CKA to 0.775.Its representational signature differs from the high-LR methods by disrupting rather than preserving encoder geometry.
- Spectral signatures of studio-bias: SVD shows that the base model and R-MFT maintain Encoder Effective Rank ζ ≈14, whereas IndicWhisper expands it to ζ = 25.The authors note that differing training distributions preclude a strictly causal comparison, while linking the expansion to studio-specific adaptation and 66.09% WER on spontaneous speech.
7. Conclusion
The paper introduces Vividh-ASR to diagnose studio-bias and studies learning-rate timing and curriculum direction in Hindi and Malayalam. It proposes R-MFT, which enables a smaller Whisper model to match or exceed conventionally fine-tuned larger counterparts while preserving encoder geometry.
- Contributions: Vividh-ASR is a complexity-tiered benchmark designed to diagnose studio-bias in Indic speech recognition.The benchmark supports the paper’s analysis across Hindi and Malayalam.
- Optimization findings: Aggressive early learning rates are the dominant factor, yielding ∼12 absolute WER points, while later high-LR stages cannot recover losses from conservative initialization.Curriculum direction contributes a smaller, language-dependent gain, especially for spontaneous Malayalam speech.
- Optimization findings: R-MFT uses a hard-to-easy curriculum with high initial learning rates and outperforms standard MFT most notably on spontaneous Malayalam speech.The conclusion characterizes the curriculum benefit as smaller and language-dependent than the learning-rate effect.
- Model efficiency and representations: R-MFT enables a 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts.CKA and SVD analyses associate this strategy with decoder adaptation and preservation of encoder acoustic geometry.
- Future work: The authors plan to extend Vividh-ASR to additional languages and test whether the optimization dynamics generalize beyond Whisper.They also identify selective encoder freezing as a future regularization strategy for mitigating studio-bias.
8. Generative AI Use Disclosure
The paper discloses the use of Gemini 2.5 Pro for linguistic refinement and technical polishing, with final content reviewed and approved by the authors.
- Disclosure: Gemini 2.5 Pro assisted with linguistic refinement and technical polishing, while the authors reviewed, verified, and approved the final manuscript.The authors retain responsibility for the research and its presentation.