Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptation

Introduction

• Singing is harder than conventional speech recognition because pitch excursions, sustained vowels, melisma, irregular rhythm, accompaniment, and expressive articulation challenge speech-trained models.The task of converting sung audio into text is known as Automatic Lyric Transcription (ALT).Source: §1 · p. 1
• Greek singing voice lacks an established ASR benchmark, and multilingual models’ behavior under singing-domain adaptation remains unclear.Low-resource languages often lack curated singing corpora, aligned annotations, and reproducible evaluation protocols.Source: §1 · p. 1
• Large-scale transcription-only and two-stage adaptation are reported as the most effective Greek ALT strategy, while multitask learning improves WER for smaller-capacity models.The study also presents a task-pure batching scheme with language-aware preprocessing and a Greek-lyrics error taxonomy.Source: §1 · p. 1
• The authors present the first systematic Greek ALT benchmark and a processed, aligned Greek Audio Dataset (GAD) with translation pairs and reproducible train-validation-test splits.The dataset includes source-separated and aligned segmentation.Source: §1 · p. 1
• The controlled study investigates Whisper adaptation across model checkpoints, transcription-only training, multitask learning, and two-stage learning for Greek singing voice.The work examines transfer from speech to singing in a low-resource setting.Source: §1 · p. 1

2.1. ALT

• ALT research moved from music-tailored HMM/GMM or DNN ASR and benchmark datasets to CTC/AED and multimodal cues, yet lyric transcription remains challenging.Lip movements and note-level transcriptions can stabilize decoding in low-SNR conditions.Source: §2.1 · pp. 1–2

2.2. Foundation Models and adaptation

• wav2vec 2.0 and XLS-R transfer to singing via fine-tuning, while Whisper is robust zero-shot but degrades on polyphonic audio; low-resource singing remains understudied.Proposed directions include Whisper–LLM error correction, LoRA specialization, and speech-text joint pretraining; most existing research focuses on high-resource languages.Source: §2.2 · p. 2
• For low-resource ASR, translation multitask learning can regularize small-data adaptation, while staged fine-tuning [16], voice-to-singing augmentation, and consistency loss address training stability and domain mismatch.Whisper’s transcription and translation tokens provide a route for multitask adaptation.Source: §2.2 · p. 2

2.3. Greek Speech and Singing Recognition

• Greek ASR has spoken corpora and pseudo-label adaptation, but no standardized Greek ALT benchmark or segment-aligned lyrics in existing resources, motivating controlled Whisper evaluation.Lyra, GAD, and GMD provide acoustic resources for general music-information-retrieval tasks, but lack segment-level aligned text annotations for end-to-end audio-to-lyrics transcription.Source: §2.3 · p. 2

3. The GAD-ALT Dataset

• The corpus extends the 1,000-song GAD, originally designed for genre classification, into an ASR-ready dataset spanning multiple Greek music genres.GAD entries include genre annotations, lyrics, manually annotated mood labels, metadata, audio features, and YouTube links.Source: §3 · p. 2

3.1. Source Separation

• For reproducible ALT data, the curation resolved metadata gaps and mismatches, standardized lyrics, and used htdemucs ft to separate vocals from accompaniment.Vocals were downmixed to mono, resampled to 16 kHz for Whisper, and converted into Kaldi format.Source: §3.1 · p. 2

3.2. Forced Alignment and Bilingual Augmentation

• GAD-ALT contains 17,458 aligned lyric segments (19.65 hours), split by song into 13,750 training, 1,892 validation, and 1,816 test segments.Each entry includes an aligned audio segment, normalized Greek transcription, and English translation; mean segment duration is 4 seconds.Source: §3.2 · p. 2

4.1. Overview

• The methodology first curates GAD-ALT, then adapts multilingual pretrained Whisper models under controlled variations in scale, task composition, and staged speech-to-singing training.The pipeline is designed for Greek automatic lyric transcription.Source: §4.1 · p. 2

4.2. Whisper Adaptation Strategies

• Zero-shot inference provides the out-of-domain baseline; Small, Medium, and Large-v3 are evaluated with transcription-only training or multitask transcription and English translation at 2:1 and 4:1 ratios.A deterministic sampler interleaves task-specific batches while keeping each batch task-homogeneous, enabling evaluation of translation as a regularizer.Source: §4.2 · pp. 2–3
• The staged strategy uses about 37.5 hours of Greek Common Voice read speech for decoder-only language adaptation, then fully unfreezes Whisper for fine-tuning on singing.Stage 1 freezes the entire encoder; Stage 2 fine-tunes on the singing corpus.Source: §4.2 · p. 3

4.3. Training Details

• Training uses AdamW for five epochs, with learning rates of 5 × 10−5 for Small and Medium and 3 × 10−5 for Large-v3, plus FP16 on multi-GPU A100 nodes.Per-GPU batch sizes range from 4 to 8 segments by model scale; the authors plan to release the pipeline and model configurations.Source: §4.3 · p. 3

5.1. Domain Gap Analysis

• Zero-shot WER falls from 92.3% for Whisper Small to 65.1% for Medium and 53.6% for Large, but supervised fine-tuning is needed to address the speech-to-singing gap.The remaining gap reflects singing-specific melodic prolongation, vowel stretching, rhythmic compression, and altered phoneme realizations.Source: §5.1 · p. 3, Table 1 · p. 3

5.2. Capacity and Regularization

• Small achieves its best WER with 2:1 transcribe:translate training (33.6%), while Medium performs best with transcription-only training (30.3%).For Small, 4:1 training gives 34.9% versus 36.7% for transcription-only, suggesting moderate translation exposure regularizes smaller models; for Medium, 2:1 and 4:1 yield 32.3% and 31.6%. Larger models benefit more from focused transcription adaptation.Source: §5.2 · p. 3, Figure 2 · p. 4

5.3. Two-Stage Adaptation

• 27.2% WER is the best result, achieved by two-stage Whisper Large-v3 adaptation, versus 28.4% with transcription-only training and 53.6% zero-shot.Two-stage adaptation yields 36.6% WER for Small and 30.1% for Medium; Small performs better with direct singing-data fine-tuning. The authors attribute Small’s weaker two-stage result partly to the read-speech characteristics of Greek Common Voice, which remain acoustically distant from singing.Source: §5.3 · pp. 3–4, Table 1 · p. 3, §5.2 · p. 3

5.4. Ablations of Source Separation and Augmentation

• Vocal-only training and evaluation outperform raw polyphonic data, which scores 33.4% WER, while remixing, reverberation, and mixed raw-plus-vocal training degrade performance.The authors conclude that vocals-only training without artificial remixing is the most reliable tested configuration.Source: §5.4 · p. 4

5.5. Qualitative Error Analysis

• Among 200 inspected errors, semantic substitutions (25.5%) and boundary drift (24.0%) are most common, followed by corrupted content (19.5%), orthographic ambiguity (17.0%), and function-word errors (12.0%).Morphological drift accounts for 2.0%; larger models mainly reduce error severity rather than changing the error-type distribution, so melodic variability and articulation distortions remain central challenges.Source: §5.5 · p. 4

6. Conclusions & Future Work

• Targeted fine-tuning reduced Word Error Rate to 27.2%, addressing the severe speech-to-singing gap seen in zero-shot inference.The study presents this as the first systematic benchmark for Greek ALT using Whisper adaptation strategies.Source: §6 · p. 4
• GAD was expanded into GAD-ALT with aligned segments, translations, and Hugging Face splits.The work presents GAD-ALT as a curated resource for Greek automatic lyric transcription.Source: §6 · p. 4
• Multitask learning with translation regularized smaller models, while larger models benefited most from transcription-only and two-stage adaptation.These findings indicate that the most effective adaptation strategy depends on model capacity.Source: §6 · p. 4
• Future work will expand the singing corpus, test expressive and spontaneous speech for staged adaptation, use parameter-efficient fine-tuning, and integrate Greek-specific language models.The language models are intended to better handle the morphological and rhythmic complexities of singing.Source: §6 · p. 4

• Generative AI assistance was limited to language editing and clarity; the authors developed the experimental design, analysis, and scientific conclusions independently.Source: §8 · p. 5