KunSet: A Corpus and Benchmark for Lyrics Alignment and Rhyme Retrieval in Kunqu Opera

1. INTRODUCTION

• Existing resources leave two gaps: syllable-level alignment of sung and spoken lyrics to recordings, and phonological attributes linked to each syllable.KunquDB provides sentence-level timestamps but lacks finer syllable boundaries; other resources do not directly link searchable phonological attributes to performance recordings.Source: §1 · p. 1
• Forced-alignment models pretrained on speech or modern singing perform poorly on Kunqu, where sustained melismatic syllables and stylized pronunciation fall outside their training distribution.The introduction identifies this as a challenge for obtaining syllable boundaries at scale.Source: §1 · p. 1
• KunSet is the first syllable-aligned Kunqu performance corpus with linked phonological attributes, comprising 482,164 syllable-level timestamps across 106.3 h of lyrics in 281 recordings.The annotations support comparison between actual interpretation and traditional phonological knowledge.Source: §1 · p. 1
• An iteratively fine-tuned SOFA model preannotates syllable boundaries, which over 200 trained annotators correct under two-round review; Yunxue Lizhu is also digitized to link syllables to candidate phonological attributes.Quality checks include two independent blind annotations and acoustic analyses.Source: §1 · p. 1
• Both Kunqu-specific MIR systems built and benchmarked on KunSet outperform their baselines.They address syllable-level lyrics alignment and same-rhyme retrieval.Source: §1 · p. 1

2.1. Dataset Format and Label Ontology

• Each recording excerpt is organized into lyric sentences and syllables, whose onset and offset times use the recording timeline and whose phonology links to Yunxue Lizhu candidates.A unique lineId associates each sentence with its constituent syllables; attributes are populated only when a single candidate or candidate consensus supplies a non-missing value.Source: §2.1 · p. 2
• KunSet contains 281 recordings; singing accounts for 47.3% of annotated material and stage speech 51.9%, while 56.8% of syllables have unambiguous values for all six attributes.Of 482,164 syllable annotations, 461,164 (95.6%) have dictionary candidates; sung sentences have a longer median duration than spoken sentences (8.81 s versus 2.77 s).Source: §2.1 · p. 2

2.2. Annotation and Review Pipeline

• Syllable timestamps were generated by an iteratively fine-tuned SOFA model, corrected by one annotator, and reviewed in two rounds by different reviewers.The iterative training set grew from one to seven excerpts before the model generated initial timestamps for the remaining 274 recordings.Source: §2.2 · p. 2

2.3. Searchable Phonological Database

• Yunxue Lizhu was converted into a searchable database linking candidate entries to fanqie, tone, yin–yang, rhyme, wuyin, and voicing attributes.It contains 10,235 entries for 8,300 direct-lookup characters, including 1,664 characters with multiple entries and 2,520 variant-alias relations.Source: §2.3 · p. 3

2.4. Annotation Quality Assessment

• M–A and M–B discrepancies were within 18.14 ms of A–B, while disagreement was greater for singing than speech and offsets than onsets.The authors interpret the shared patterns as annotation difficulty rather than a discrepancy specific to ML-assisted production.Source: §2.4 · p. 3

2.5. Acoustic Consistency Verification

• 0–200 ms onset-strength contrast at syllable starts was 0.47 standard deviations, remaining 0.43 after excluding sentence-initial syllables.Both contrasts compare annotated starts with random times in the same sentence; the reported recording-level bootstrap 95% CIs are 0.46–0.49 and 0.42–0.45, respectively.Source: §2.5 · p. 3
• Annotated syllable starts show sharp increases in onset strength and RMS energy from near-onset troughs, unlike random-time controls, which decline over the same interval.The contrast remains after sentence-initial syllables are excluded, supporting a corpus-level association with acoustic transitions within sentences.Source: §2.5 · p. 3

3.1. Syllable-level lyrics alignment

• The alignment evaluation uses 251 Train, 20 Dev, and 10 Test recordings, comparing pretrained and Kunqu-adapted SOFA models with original or acoustic-anchor decoding.Adapted models are fine-tuned on either 7 excerpts or 251 excerpts; the 251-recording model trains for 9,000 steps with three seeds.Source: §3.1 · pp. 3–4
• 13.04% lower MAE and Acc@200 rising from 79.52% to 81.69% with acoustic anchors; the 251-excerpt model also beats the seven-excerpt model under both decoders.Results are recording-macro averages; the 251-excerpt model uses three seeds.Source: Table 3 · p. 4, §3.1 · p. 4
• The acoustic anchor decoder jointly selects boundaries and strong phone-evidence frames within candidate phone intervals.It uses an exact O(TS) search, then groups phone intervals into syllables; inference changes without changing acoustic training.Source: §3.1 · p. 4
• On sung sentences the adapted model’s error remains above human disagreement, while spoken-sentence error is of the same magnitude.The reported errors are 462.48 ms versus 272–409 ms for singing, and 123.77 ms versus 108–165 ms for speech.Source: §3.1 · p. 4

3.2. Same-rhyme retrieval

• 17.16% P@10 under traditional supervision exceeds modern Mandarin finals by 0.81 percentage points and frozen HuBERT by 5.80 percentage points.The traditional–modern comparison should be interpreted in light of the retrieval labels matching the traditional taxonomy.Source: Table 4 · p. 4, §3.2 · p. 4
• Same-rhyme retrieval uses frozen Chinese HuBERT-base features and retains instances with unambiguous traditional-rhyme and modern-Mandarin-final labels.The split contains 256,333 Train, 20,674 Dev, and 10,287 Test instances, with supported Test queries for all 29 rhyme groups.Source: §3.2 · p. 4
• The retrieval projection head is trained with supervised contrastive loss, and Test queries search the Train library by cosine similarity.Checkpoint selection uses Dev P@10; evaluation excludes candidates sharing the query’s character identity or recording and averages P@10 equally across rhyme groups.Source: §3.2 · p. 4

• KunSet comprises 281 recordings with 482,164 syllable-level annotations linked to Yunxue Lizhu phonological attributes.Source: §4 · p. 4
• Experiments demonstrate improved lyrics alignment and the benefit of traditional phonological supervision for same-rhyme retrieval.Source: §4 · p. 4
• The dataset, code, and model weights are released for research-purpose only, providing a foundation for MIR research on Kunqu.Source: §4 · p. 4