Source-linked AI summary
Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptation
Maria Frangiadaki, Dimitrios Damianos, Kosmas Kritsis, Vassilis Katsouros
TL;DR
Automatic Lyric Transcription is especially difficult for singing and remains underbenchmarked in low-resource Greek. This paper adapts Whisper, curates an aligned Greek singing corpus, and studies scaling, multitask composition, and staged adaptation. Targeted adaptation reaches 27.2% WER with Whisper Large-v3, while multitask regularization chiefly benefits smaller models.
Problem
Greek lacks a prior benchmark for Automatic Lyric Transcription, a task made difficult by melodic, rhythmic, and accompaniment differences from speech.
Method
The study curates aligned Greek singing data and systematically adapts Whisper across model scales, task compositions, and two-stage speech-to-singing training.
Results
27.2% WER is achieved by Whisper Large-v3 after two-stage adaptation, while multitask learning primarily regularizes smaller-capacity models.
Takeaways & Limitations
The study establishes the first Greek ALT benchmark and shows that adaptation strategy matters alongside model scale.
Abstract
from arXiv · showhide
Automatic Lyric Transcription (ALT) remains substantially more challenging than speech recognition due to melodic variability, rhythmic irregularity, and accompaniment interference. This is heightened in low-resource languages like Greek, where no prior benchmark for ALT exists. We present the first controlled study of Whisper adaptation for Greek ALT, investigating model scaling effects, task composition via multitask training in transcribe-translate ratios, and two-stage speech-to-singing adaptation. We also curate a segment-level aligned singing dataset based on the Greek Audio Dataset (GAD) using source separation and CTC forced alignment. Results show that scaling consistently improves performance, while multitask learning acts as a beneficial regularizer primarily for smaller-capacity models. The 2-stage adaptation in Whisper Large-v3 achieves a Word Error Rate (WER) of 27.2%, a significant improvement over zero-shot baselines, establishing the first Greek ALT benchmark.
1 Institute for Language and Speech Processing, Athena R.C., Greece
The section lists five email contacts.
- The listed contacts are maria.frangiadaki, d.damianos, kosmas.kritsis, and vsk at athenarc.gr.
1. Introduction
Automatic Lyric Transcription is difficult because singing differs acoustically and linguistically from speech, with the challenge intensified in low-resource Greek. This work establishes a controlled Greek ALT benchmark through Whisper adaptation and curated aligned data.
- Singing makes lyric transcription harder than conventional speech recognition through pitch excursions, sustained vowels, melisma, rhythmic irregularity, accompaniment, and expressive articulation.
- Greek lacks a standardized singing-voice benchmark, curated aligned singing corpora, and controlled evidence on multilingual model adaptation.
- The study develops an end-to-end Greek ALT pipeline and evaluates Whisper across scale, transcription-only versus multitask training, and staged adaptation.
2. Related Work
Related work traces ALT from conventional music-adapted ASR to end-to-end and foundation-model approaches, while emphasizing persistent speech-to-singing mismatch and missing Greek benchmarks.
- ALT evolved from HMM, GMM, and DNN pipelines toward end-to-end CTC and attention-based models with unified acoustic modeling and alignment.
- Self-supervised models such as wav2vec 2.0 and XLS-R transfer to singing through fine-tuning, whereas Whisper’s zero-shot accuracy degrades substantially on polyphonic audio.
- Multitask learning and staged fine-tuning are established strategies for adapting ASR models to low-resource domains and reducing overfitting or training instability.
- Greek ASR has advanced through spoken corpora and unsupervised adaptation, but Greek singing recognition remains virtually absent and lacks a standardized ALT benchmark.
3. The GAD-ALT Dataset
GAD-ALT transforms the Greek Audio Dataset into a reproducible singing ASR resource by separating vocals, aligning lyrics temporally, and adding segment-level translations. The resulting corpus contains 17,458 aligned lyric segments spanning 19.65 hours.
- GAD-ALT extends 1,000 genre-diverse Greek songs with the temporal synchronization required for automatic lyric transcription.
- Source separation extracts vocals and accompaniment, then vocals are downmixed to mono, resampled to 16 kHz, and converted into Kaldi format.
- 17,458 aligned lyric segments cover 19.65 hours after CTC forced alignment, confidence filtering, and segment-level English translation.
4. Methodology and Experimental Setup
The study builds an end-to-end Greek ALT pipeline and evaluates Whisper adaptation across model scale, task composition, and staged speech-to-singing training.
- Experimental design: The methodology curates GAD-ALT and adapts multilingual Whisper models under controlled scaling, task-composition, and staged-adaptation settings.The pipeline begins with the curated GAD-ALT dataset and varies model scale, transcription versus multitask training, and speech-to-singing adaptation.
- Task composition: Three Whisper scales are compared using transcription-only training and multitask transcription-plus-translation batches mixed at 2:1 or 4:1 ratios.A deterministic sampler creates task-homogeneous batches for testing whether translation acts as a regularizer.
- Staged adaptation: Staged adaptation first fine-tunes the decoder on approximately 37.5 hours of Greek read speech, then fully unfreezes the model for singing adaptation.The first stage freezes the encoder for language adaptation before the second stage fine-tunes on singing data.
- Implementation: Training uses AdamW for five epochs with scale-dependent learning rates, multi-GPU A100 nodes, mixed precision, and per-GPU batches of four to eight segments.Whisper Small and Medium use 5 × 10−5, while Large-v3 uses 3 × 10−5.
5. Results
Results show that singing-domain adaptation is essential, scaling improves robustness, and task composition helps smaller models more than larger ones.
- Domain gap: 92.3%, 65.1%, and 53.6% WER are obtained by zero-shot Whisper Small, Medium, and Large, respectively, showing that scale alone leaves substantial singing-domain errors.Supervised fine-tuning substantially reduces WER across the larger models.
- Capacity and regularization: 33.6% WER makes the Small model’s 2:1 multitask configuration better than transcription-only at 36.7% and 4:1 multitask training at 34.9%.The result supports a regularization benefit from moderate auxiliary translation exposure at low capacity.
- Capacity and regularization: 30.3% WER for Medium and 28.4% for Large-v3 show that transcription-only training becomes preferable as model capacity increases.Medium transcription-only outperforms its 2:1 and 4:1 mixtures, while Large-v3 also favors focused transcription adaptation.
- Overall results: 27.2% WER is achieved by Whisper Large-v3 with two-stage adaptation, the strongest reported result across the evaluated configurations.This result reflects the final staged speech-to-singing adaptation of the largest model.
- Source separation and augmentation: 33.4% WER on raw polyphonic data is worse than isolated-vocal training, while SNR remixing, reverberation, and mixed raw-plus-vocal augmentation consistently degrade performance.The study therefore identifies vocals-only training without artificial remixing as the most reliable ablation configuration.
- Qualitative error analysis: Among 200 inspected errors, semantic substitution accounts for 25.5% and boundary drift for 24.0%, while larger models reduce severity without changing the error distribution.The analysis also reports hallucinated or corrupted content, orthographic ambiguity, function-word errors, and morphological drift.
6. Conclusions & Future Work
The work establishes the first systematic Greek ALT benchmark and finds that targeted Whisper adaptation substantially improves transcription, with different strategies favored by model capacity.
- Contributions: The study presents the first systematic benchmark for Greek ALT by expanding GAD into GAD-ALT with segments, alignments, translations, and Hugging Face splits.The benchmark supports reproducible evaluation of Greek singing transcription.
- Conclusions: 27.2% WER demonstrates that targeted Whisper fine-tuning substantially reduces the severe speech-to-singing gap seen with zero-shot inference.The conclusion identifies this as the principal performance outcome.
- Implications and future work: Multitask translation regularizes smaller models, whereas larger models benefit most from focused transcription-only and two-stage adaptation.Future work includes expanding the corpus, using expressive speech for staged adaptation, applying parameter-efficient fine-tuning, and integrating Greek-specific language models.
8. Generative AI Use Disclosure
The authors disclose that generative AI assisted with language editing and clarity, while experimental design, analysis, and scientific conclusions were developed independently.
- Disclosure: Generative AI tools were used to refine language and clarity, not to develop the experimental design, analysis, or scientific conclusions.The disclosure separates editorial assistance from the study’s scientific work.