PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Songs to Improve Kazakh Automatic Speech Recognition

Rustem Yeshpanov

arXiv:2603.00961v3eess.AS

TL;DR

Scarcity of transcribed corpora limits ASR for low-resource languages, including Kazakh. The study constructs a small song dataset and fine-tunes Whisper with songs and speech corpora, finding that mixtures improve adaptation over zero-shot models while remaining below the large KSC2-corpus upper bound.

  • Problem

    Scarcity of large, diverse, carefully transcribed corpora limits ASR development for low-resource languages such as Kazakh.

  • Method

    The study curates 3,013 song audio-text pairs and evaluates Whisper fine-tuned under seven mixtures of Songs, CVC, and FLEURS across CVC, FLEURS, and KSC2 benchmarks.

  • Results

    27.6% normalised WER on CVC, 11.8% on FLEURS, and 39.3 versus 81.2 on KSC2 show that the triple mixture improves over the zero-shot Turbo baseline.

  • Takeaways & Limitations

    Songs are not a standalone solution but can amplify the value of small existing speech corpora for low-resource ASR.

  • Takeaways & Limitations

    The song dataset is small, non-exhaustive, sometimes contains residual background music, and relies on manual alignment and lyric correction.

Abstract

from arXiv · show

Developing automatic speech recognition (ASR) systems for low-resource languages is hindered by the scarcity of transcribed corpora. This proof-of-concept study explores songs as an unconventional yet promising data source for Kazakh ASR. We curate a dataset of 3,013 audio-text pairs (about 4.5 hours) from 195 songs by 36 artists, segmented at the lyric-line level. Using Whisper as the base recogniser, we fine-tune models under seven training scenarios involving Songs, Common Voice Corpus (CVC), and FLEURS, and evaluate them on three benchmarks: CVC, FLEURS, and Kazakh Speech Corpus 2 (KSC2). Results show that song-based fine-tuning improves performance over zero-shot baselines. For instance, Whisper Large-V3 Turbo trained on a mixture of Songs, CVC, and FLEURS achieves 27.6% normalised WER on CVC and 11.8% on FLEURS, while halving the error on KSC2 (39.3% vs. 81.2%) relative to the zero-shot model. Although these gains remain below those of models trained on the 1,100-hour KSC2 corpus, they demonstrate that even modest song-speech mixtures can yield meaningful adaptation improvements in low-resource ASR. The dataset is released on Hugging Face for research purposes under a gated, non-commercial licence.

1. Introduction

Kazakh remains low-resource for ASR because large, diverse, carefully transcribed corpora are scarce. This study evaluates songs as an accessible supplementary resource and finds measurable benefits, especially when combined with small speech corpora.

  • Motivation: Kazakh ASR remains low-resource despite recent resource-building efforts.Kazakh is spoken by more than 15 million people worldwide, but resources remain limited by global standards.
  • Motivation: Songs offer widely available audio with lyrics that can serve as approximate transcriptions, but music and non-conversational prosody create ASR challenges.The study treats their usefulness as an empirical question rather than assuming songs transfer directly to speech recognition.
  • Approach: The experiments compare zero-shot Whisper with seven fine-tuning scenarios using Songs, CVC, FLEURS, and their combinations.The study also compares results with an upper-bound scenario based on more than 1,100 hours of KSC2 recordings.
  • Findings: Songs improve performance over zero-shot baselines on multiple benchmarks, while mixtures with small speech corpora strengthen generalisation.The gains are uneven across evaluation domains and remain below the KSC2 upper bound.

2. Related Work

Kazakh ASR research benefits from foundational speech and text-to-speech resources, but available multilingual datasets remain modest in scale and diversity. This motivates investigating accessible, linguistically broad alternatives such as songs, whose recordings and lyrics offer a promising yet underexplored resource despite acoustic and prosodic challenges.

  • KSC2 provides 1,128 hours of transcribed audio across more than 520,000 utterances, supporting recent progress in Kazakh ASR.
  • CVC and FLEURS include Kazakh and support training and benchmarking, but their scale and diversity remain modest relative to English or Mandarin corpora.
  • Prior low-resource ASR work uses data augmentation and transfer learning, including transliteration and cross-lingual speech-to-text translation.
  • Songs offer accessible, linguistically broad data through widely available recordings and approximate lyric transcriptions, although background music and nonstandard prosody create challenges.

3. Dataset

The study constructs a diverse Kazakh songs dataset by selecting vocal-focused tracks across artists and genres, manually correcting lyrics, separating vocals, and aligning audio with lyric lines. The resulting resource contains 3,013 audio-text pairs over approximately 4.5 hours and is positioned as a proof-of-concept corpus because of its limited size, coverage, and residual music.

  • Selection: 195 Kazakh songs by 36 artists were selected, including 14 female and 22 male performers.
  • Selection: Tracks were chosen for prominent solo vocals and genre diversity spanning pop, pop-estrada, folk-pop, folk-rock, R&B, and hip-hop.
  • Preprocessing: Vocals were separated from accompaniment with Spleeter, then manually audited so unintelligible segments could be discarded.
  • Preprocessing: Lyrics were manually reviewed and corrected to match sung content, including repetitions and colloquial pronunciations.
  • Dataset composition: 3,013 audio-text pairs totaling approximately 4.5 hours were created through manual line-level alignment and Audacity label export.
  • Limitations: The resource is limited by its 4.5-hour duration, incomplete genre and regional coverage, and residual background music, so it is a proof-of-concept rather than a comprehensive corpus.

4. Methodology

The methodology fine-tunes Whisper under seven combinations of Songs, CVC, and FLEURS, evaluates models on CVC, FLEURS, and KSC2, and standardizes transcript handling through normalized metrics. It also compares compact Whisper models with Whisper Large-V3 Turbo and a KSC2-trained upper-bound model.

  • Training design: Seven training scenarios covered Songs, CVC, FLEURS, their pairwise mixtures, and the three-dataset mixture.
  • Training design: Experiments used a 5×10^-6 learning rate, 50 warm-up steps, batch size 60, and early stopping with patience of two epochs.
  • Evaluation data: CVC, FLEURS, and KSC2 served as test sets, with preprocessing removing selected English or Russian-only sentences and replacing homoglyphs with Kazakh letters.
  • Transcript conventions: The datasets use different script and formatting conventions, including casing, punctuation, digits, and Latin or Cyrillic characters.
  • Model selection: Whisper-tiny and Whisper-small were evaluated alongside Whisper Large-V3 Turbo, a faster pruned model, plus a community KSC2-trained upper-bound model.
  • Metrics: WER measures word-level transcription accuracy, whereas CER evaluates at the character level and is useful for morphologically rich Kazakh.
  • Metrics: Normalized WER and CER remove casing, punctuation, and extra whitespace; KSC2 results use only normalized metrics because of its formatting.

5. Results

Fine-tuning Whisper Large-V3 Turbo on songs and speech mixtures substantially improves normalised and orthographic WER over zero-shot baselines, though gains vary by domain and remain below the KSC2-trained upper bound.

  • Baselines: 12.5 on CVC, 11.3 on FLEURS, and 9.3 on KSC2 normalised WER define the 1,128-hour KSC2-trained upper bound.Zero-shot Whisper models perform substantially worse overall, with Turbo reaching 81.2 on KSC2.
  • Single-source fine-tuning: 37.3 on CVC and 45.2 on KSC2 show that Songs-only fine-tuning improves over the Turbo baseline, but FLEURS worsens to 23.7.The corresponding Turbo baseline values are 47.7 on CVC, 81.2 on KSC2, and 21.0 on FLEURS.
  • Mixtures: 28.1 on CVC, 11.8 on FLEURS, and 39.3 on KSC2 show that CVC + FLEURS mixtures outperform single-source training across benchmarks.The triple mixture is described as the most balanced overall, reaching 27.6 on CVC, 11.8 on FLEURS, and 39.3 on KSC2.
  • Mixtures: 27.6 on CVC, 11.8 on FLEURS, and 39.3 on KSC2 make Songs + CVC + FLEURS the most balanced mixture overall.The triple mixture is best on CVC and ties the best results on FLEURS and KSC2; Songs + FLEURS also reaches 11.8 on FLEURS but is weaker elsewhere.
  • Orthographic view: 32.0 versus 70.6 on CVC and 19.5 versus 38.0 on FLEURS show large orthographic-WER reductions for the triple mixture.These correspond to approximately 55% and 49% relative reductions, respectively.
  • Community-model fine-tuning: Starting from the KSC2-trained upper bound leaves limited headroom and introduces domain-drift risks.With the KSC2 baseline, Songs-only training degrades CVC, FLEURS, and KSC2 normalised WER; mixtures remain more balanced but can still degrade KSC2.
  • Cross-domain generalisation: The largest relative gains occur for the study’s Turbo fine-tuning, with song-based training improving spontaneous and conversational KSC2 domains.The comparison is made across six KSC2 speech domains, including crowdsourced, parliamentary, podcasts, radio, talkshows, and television news.

CVC FLEURS KSC2

Across CVC, FLEURS, and KSC2, song-based adaptation improves recognition quality and stability, especially when songs are combined with modest speech corpora. These gains remain below the large-corpus KSC2 upper bound and are constrained by the song dataset’s small size and copyright status.

  • Qualitative analysis: Song-adapted models remain in Kazakh on difficult KSC2 examples, reducing cross-lingual drift and recovering intended meaning with minor variation.Zero-shot outputs shift into another language or produce nonsensical tokens, while song-adapted systems show more stable decoding.
  • Qualitative analysis: Song adaptation improves lexical, morphological, punctuation, and segmentation accuracy across CVC and FLEURS examples.Models recover core vocabulary and suffixes, preserve clause boundaries, and more consistently restore punctuation.
  • Quantitative results: 27.6 normalised WER on CVC and 11.8 on FLEURS: the triple mixture achieves the best CVC result and ties the best FLEURS result for Turbo.It also reaches 39.3 on KSC2 versus 81.2 for the Turbo baseline.
  • Limitations and scope: The 1,100-hour KSC2 upper bound remains stronger, while further song or small-corpus fine-tuning after large-scale training yields marginal gains and can induce forgetting.This comparison frames songs as complementary data rather than a replacement for large transcribed speech corpora.
  • Limitations and scope: The song corpus contains only 4.5 hours and roughly 3,000 lyric lines, limiting adaptation of a billion-parameter model; its recordings are also copyrighted works used without explicit artist permission.The study therefore treats songs as an exploratory, non-deployment-ready resource and calls for larger, more diverse, and legally integrated alternatives.

6. Conclusion

Songs provide a meaningful signal for Kazakh ASR when combined with modest speech corpora, but they do not fully transfer to conversational or broadcast speech and remain below the large-corpus upper bound. The approach is practical to explore at low cost, yet copyright constraints limit its deployment.

  • Songs alone do not consistently improve recognition, whereas mixtures with CVC and FLEURS consistently outperform single-corpus baselines.The best results come from combining Songs, CVC, and FLEURS.
  • 27.6 normalised WER on CVC and 11.8 on FLEURS mark substantial improvements over zero-shot Whisper models.The combined Songs + CVC + FLEURS setting also narrows the gap to the 1,100-hour KSC2 model.
  • Song-based training does not fully transfer to conversational or broadcast speech, and gains remain modest relative to the large-scale upper bound.Orthographic errors involving casing and punctuation also remain challenging, although songs reduce them in some scenarios.
  • $25 in compute covered the fine-tuning experiments, indicating that low-resource ASR exploration can use modest computational resources.
  • Copyrighted recordings without explicit permission make the Songs dataset a proof of concept rather than a deployable approach.Short vocal excerpts are available on Hugging Face under a gated, non-commercial research licence, while full recordings are not distributed.
  • Synthetic songs could provide a scalable, copyright-free complement designed to cover diverse phonetic contexts, prosody, and stylistic registers.Testing whether synthetic music can match or surpass real-song benefits is identified as a next step.
Loading 2603.00961v3…