PaperlayerView exact revision on arXiv

Source-linked AI summary

Unadapted Multilingual ASR on a Garrusi Kurdish Evaluation Set: A Common-Reference Staged Normalization Analysis

Hiwa Asadpour

arXiv:2608.16379v1cs.CLcs.SD

TL;DR

Cross-script evaluation of Garrusi Kurdish ASR risks scoring script differences as recognition errors, while normalizing both sides also changes reference tokenization. This paper evaluates unadapted MMS-1B-all with staged hypothesis-only normalization against a fixed reference; RAW-to-FOLDED reduces WER by 13.85 percentage points and CER by 49.72 percentage points.

  • Problem

    Garrusi ASR lacks a trained recognition system, and cross-script scoring confounds writing-system differences with recognition errors while joint normalization changes reference tokenization.

    Source: §1 · p. 2

  • Method

    The paper evaluates released, unadapted MMS-1B-all with the Central Kurdish adapter using staged hypothesis normalization against one fixed folded reference.

    Source: §1 · p. 2

  • Results

    13.85 percentage points in WER and 49.72 percentage points in CER separate RAW and FOLDED; MMS-1B-all yields 97.85% WER and 51.20% CER.

    Source: §9 · p. 11

  • Takeaways & Limitations

    Without conversion, cross-script comparison cannot be interpreted as a recognition measurement.

    Source: §5 · p. 8

    The transliteration is incomplete and uncorrected, so reported rates include replacement characters, unconverted Arabic characters, and folding-table gaps.

    Source: §6 · p. 10

Abstract

from arXiv · show

Evaluating speech recognition for a Kurdish variety written in a Latin field orthography, using a model that outputs Arabic script, creates a measurement problem before a modelling one: direct scoring treats writing-system differences as recognition errors. Jointly normalizing reference and hypothesis avoids this, but also changes reference tokenization, mixing agreement gains with a change in the scoring denominator. I evaluate MMS-1B-all with the Central Kurdish (ckb) adapter, used as released without adaptation, on 1,722 Garrusi questionnaire segments from five speakers (9,763 reference word tokens; 117.9 minutes). I use a common-reference design: the reference is folded once and fixed at 9,763 tokens, while only the hypothesis representation varies. The raw Arabic-script hypothesis scores 111.70% WER and 100.92% CER, with zero exact word matches. Latin transliteration gives 102.36% WER and 57.89% CER; folding it into the reference's reduced orthography gives 97.85% and 51.20%. Thus RAW-to-FOLDED reduces measured WER by 13.85 points and CER by 49.72 points; folding alone accounts for 4.51 and 6.69 points. Substantial error remains: 14.53% of reference tokens are exact matches, edits are substitution-dominated, and per-segment WER is higher for shorter segments. A Southern Kurdish fine-tuned system (aranemini/southern-kurdish-asr), scored under the same design, performs worse on every speaker (1,703 segments), with 109.56% WER and 55.85% CER. However, 12,330 output characters fall outside the folding table, so these rates must be recomputed against the corrected fixed reference. The MMS output also contains 613 unconverted or unmapped characters, showing that part of the residual error reflects scoring-pipeline limits rather than recognition alone. I will release the fixed reference and segment-level results, subject to source-corpus sharing terms, to support independent checking.

1 Introduction

This pilot measures zero-shot ASR on Garrusi questionnaire speech while separating hypothesis normalization from reference-tokenization changes. It addresses the lack of Garrusi-specific recognition evaluation and makes the scoring design explicit for checking.

Evaluation gap

  • No evaluation reports recognition rates specific to Garrusi, despite one Southern Kurdish benchmark including a Garrusi speaker.That benchmark reports character error rates by speaker rather than vernacular, and four of five vernaculars have only one speaker, preventing separation of speaker and vernacular effects; the authors attribute score spread to speaker characteristics.Source: §1 · p. 1

Measurement problem

  • A Latin-field-orthography reference and Central Kurdish Arabic-script output make direct scoring confound recognition errors with writing-system differences.Normalizing both sides can also alter reference tokenization, mixing increased agreement with a changed scoring denominator; prior Kurdish ASR work notes this can inflate WER relative to CER (Mohammadamini & Tahon, 2026; Mohammadamini et al., 2026).Source: §1 · p. 2

Contribution

  • With no Garrusi-trained recognizer available, the pilot transfers a related-variety model, motivated by Garrusi’s morphosyntactic overlap with Central Kurdish despite phonological and phonetic differences.The authors frame the result as a zero-shot measurement, not a benchmark with a released dataset and shared protocol.Source: §1 · p. 2
  • The pilot evaluates MMS-1B-all with its Central Kurdish adapter using a fixed folded reference and three hypothesis-side normalization conditions.The reference tokenization remains constant, so differences between conditions reflect hypothesis transformations; the fixed reference and segment-level results are intended for release subject to source-corpus sharing terms.Source: §1 · p. 2

2 Background: Kurdish Varieties and Orthographic Setting

Kurdish classifications and orthographies are varied, and Garrusi’s placement is unsettled. This context makes the Central Kurdish adapter’s scope and the Latin-reference/Arabic-output comparison important to interpret.

Background: Kurdish Varieties and Orthographic Setting

  • Five commonly distinguished Kurdish varieties include Northern, Central, Southern, Gorani and Zazaki, but classifications disagree and Garrusi’s placement is unsettled.Fattah’s classification, as reported by Belelli, places Bijāri (Garrusi) among Southern Kurdish subgroups; Belelli also describes the area as a Southern Kurdish enclave with lexical items shared with neighboring Central Kurdish varieties.Source: §2 · p. 2
  • Because Central Kurdish includes regional varieties, its adapter should be understood as adapted to some portion of that range, not all of it.The model’s training-variety coverage cannot be audited, so no claim is made about which Central Kurdish varieties are represented.Source: §2 · p. 2
  • The study uses “Garrusi” as its speech variety label without attempting to resolve its broader classification.The evaluated speech and the linguistic description relied on were recorded in Mehraban District, Hamadan Province.Source: §2 · p. 2
  • Latin field-orthography references and Central Kurdish Arabic-script outputs differ before recognition accuracy is assessed, so hypotheses are scored under staged normalization with the reference fixed.This reveals how orthographic processing changes measured agreement, but does not divide remaining error into orthographic and acoustic components.Source: §2 · p. 2

3 Data and Methods

This section defines a fixed-reference evaluation of unadapted Kurdish ASR on five speakers’ elicited Garrusi questionnaire recordings. It describes the recognition systems, text transformations, and scoring choices used to compare hypotheses against one unchanged reference.

3.1 Data

  • The evaluation set contains 1,722 segments, 9,763 reference words, and 117.9 minutes of scored audio from five Phase 1 speakers.Segments under 0.3 seconds were excluded, and 247 duplicate rows from a resumed run were removed; the speaker selection was not random or designed, and the set contains no conversational speech.Source: §3.1 · p. 3

3.2 Recognition

  • MMS-1B-all used its Central Kurdish adapter as released, with greedy CTC decoding and no Garrusi training, fine-tuning, or adaptation.The reported rates are not comparable to the model’s multilingual benchmark results, which used an n-gram language model; upstream pretraining contents could not be audited, and recording channel and speaking style also differ from the training sources.Source: §3.2 · p. 3
  • The Southern Kurdish aranemini/southern-kurdish-asr system was run without adaptation and scored in FOLDED form against the same folded reference.Inference failed on 19 of 1,722 segments; the Southern-specific character ۊ occurs in 170 of the 1,703 scored segments and is outside the Central Kurdish transliterator’s inventory.Source: §3.2 · p. 3

3.3 Text transformations

  • KLPT transliterates only the Arabic-script hypothesis to Latin, then folding reduces both sides to a shared alphabet while the folded reference remains fixed.Folding lowercases and maps or deletes characters, collapsing phonemic distinctions; 613 hypothesis characters outside the table become whitespace and split tokens.Source: §3.3 · p. 4

3.4 The common-reference scoring design

  • All conditions score 1,722 segments against the same fixed 9,763-token folded reference; only the hypothesis changes among RAW, TRANSLIT, and FOLDED.Corpus-level WER and CER use total edits divided by total reference tokens or characters, respectively, with inter-word spaces counted for CER.Source: §3.4 · p. 4

4 Results

In the common-reference FOLDED condition, MMS-1B-all with the Central Kurdish adapter scores 97.85% WER and 51.20% CER, with exact matches for 14.53% of reference tokens. Results also show substitution-dominated errors, higher per-segment WER in shorter segments, and higher measured rates for the Southern Kurdish system on every speaker, though that comparison covers different segment sets.

4.1 Main result

  • 97.85% WER and 51.20% CER are the final FOLDED-condition results; 1,419 of 9,763 reference tokens (14.53%) align as exact matches.The RAW condition has zero word-level hits because its Arabic-script hypotheses and Latin-script reference have disjoint token inventories; its 111.70% WER is a scoring baseline, not a recognition rate.Source: Table 2 · p. 5, §4.1 · p. 5

4.2 Error composition

  • Of 9,553 FOLDED-condition edits, 6,487 (67.9%) are substitutions, 1,857 (19.4%) deletions, and 1,209 (12.7%) insertions.The 9,115-token hypothesis is 93.4% of reference length; 75 segments (4.4%) have empty hypotheses. RAW WER exceeds 100% because insertions accompany a full complement of substitutions and deletions, not because insertions predominate.Source: §4.2 · p. 5

4.3 Speaker-level results

  • FOLDED-condition WER ranges from 90.62% to 102.34% and CER from 43.74% to 57.57% across speakers, so the pooled result does not rest on one outlier.The paper does not attribute speaker differences: there are five speakers, no metadata for this phase, and no analysis of elicitation items by speaker.Source: Table 3 · p. 6, §4.3 · p. 6

4.4 Segment length

  • Per-segment WER tends to be higher for segments with fewer reference tokens (Spearman rho = −0.390, p = 1.74 × 10⁻⁶³).Median WER is 1.00 for bins of one to seven reference tokens and falls for longer segments; because the reference token count is the denominator, this pattern is descriptive, not an estimate of a length effect on recognition. The set has a median of 5 reference tokens and 28.7% of segments contain three or fewer.Source: Figure 2 · p. 6, §4.4 · p. 6

4.5 A Southern Kurdish fine-tuned system

  • The Southern Kurdish system has higher measured WER and CER on every speaker than FOLDED MMS, but the difference is directional, not a measured gap.Inference failed on 19 segments, leaving 1,703 scored versus 1,722 for MMS; the systems therefore use different segment sets. Southern Kurdish has 2,664 exact reference-token matches (26.9%) but produces 12,815 tokens, 129.3% of reference length, with 33.3% of its 10,860 edits being insertions.Source: §4.5 · pp. 6–7, Table 4 · p. 7

5 Discussion

With the reference tokenization fixed, staged normalization shows that converting the Arabic-script output changes measured agreement substantially, but a high WER remains after folding. The discussion qualifies that result as one scoring-design-dependent transfer observation, not a causal measure of script error or a general measure of Garrusi ASR performance.

Transfer scope

  • The comparison is a transfer result for one Central Kurdish adapter used without Garrusi data, not an estimate of what Garrusi ASR could achieve with in-variety data.The Southern Kurdish system was trained on edited read sentences, whereas this evaluation uses elicited questionnaire speech recorded in the field; the comparison does not isolate variety from speaking style or recording channel. The result is one observation on one evaluation set under one scoring design.Source: §5 · pp. 7, 9

Related benchmark limits

  • Mohammadamini and Tahon (2026) report 24.26 WER against 4.09 CER, but unspecified normalization and tokenization prevent recomputing those rates from their released artefacts.Their benchmark supports large cross-dialect transfer-gap comparisons, but its two fine-tuned-system ranking comes from single runs without intervals, and four of five vernaculars have only one speaker, so per-speaker error cannot be attributed to vernacular.Source: §5 · pp. 7–8

What the staged normalization measures

  • 13.85 percentage points lower WER follows RAW-to-FOLDED conversion; with the reference fixed at 9,763 tokens, FOLDED scores 97.85% WER and 51.20% CER.TRANSLIT scores 102.36% WER and 57.89% CER; folding alone reduces WER by 4.51 points and CER by 6.69 points. RAW has no word-level agreement, while transliteration yields 949 exact word matches.Source: §5 · p. 8

The residual

  • 14.53% of reference tokens are exact matches after folding, while substitutions make up 67.9% of edits; the remaining errors are not identified as recognition errors alone.Hypothesis length is 93.4% of reference length. Orthographic and word-boundary differences may remain, and the design does not establish how much error is due to any one cause.Source: §5 · p. 8
  • In example MR.141, the reference token tenabo is rendered as tenaw bo in the hypothesis, and word-level scoring counts the boundary difference among four substitutions.The segment has nine tokens, five exact matches, no insertions or deletions, and 44.44% WER; it was selected to illustrate this error type and is not typical of the corpus.Source: §5 · p. 8

The second system

  • The Southern Kurdish fine-tuned system performs worse on all five speakers and both metrics, but the difference’s size and cause are not established.The runs use segment sets differing by 19 segments and do not share a reference-token count; the systems also differ in architecture, training data, decoding, and pretraining. The fine-tuned system emits more text and inserts, and its scoring path may contribute error for characters outside the Central Kurdish transliteration specification.Source: §5 · pp. 8–9

Segment length

  • Per-segment WER is descriptively higher for segments with fewer reference tokens; 28.7% of segments contain three or fewer tokens, where one error in a three-token segment costs 33 percentage points.The discussion offers rate granularity and reduced context as possible explanations, but does not test either; it recommends holding segmentation policy constant in comparative evaluation.Source: §5 · p. 9

6 Limitations

The reported rates apply to a five-speaker questionnaire-speech set under one specific configuration, and limitations in transliteration and folding mean some residual error may come from the scoring pipeline. The staged design measures agreement changes between representations, not error causes or a ceiling, and the rates are not directly comparable across evaluations.

Scope of the evaluation set

  • The five-speaker results cover elicited questionnaire speech, not spontaneous Garrusi; the speakers were neither random nor designed, and a later run suggests they are not aggregate outliers within Phase 1.The later run excluded 410 empty-hypothesis segments, unlike the reported set’s 75; its 102.15% WER and 57.94% CER therefore use different treatment and are not directly comparable with 97.85% WER here.Source: §6 · p. 9

Correction to the folding table

  • The corrected folding table maps ü to u; an earlier omission inflated the reference from 9,763 to 9,933 tokens by splitting 170 word-internal occurrences.Folding still collapses diacritic distinctions and removes pharyngeals, so corrected reference tokenization does not fully match the transcriber’s orthography.Source: §6 · p. 9

Transliteration was incomplete

  • KLPT left 502 U+FFFD characters, 74 Arabic characters, and 36 Latin diacritics unconverted (612 total); in FOLDED these became whitespace, split hypothesis tokens, and may overstate recognition error by an unmeasured amount.KLPT’s detection of unwritten vowel i was evaluated at 39% accuracy (Ahmadi, 2020, p. 77).Source: §6 · p. 10

Second-system comparison

  • The Southern Kurdish comparison is weaker: 19 of 1,722 inferences failed, so rates use 1,703 segments and a smaller reference; direction is consistent across all five speakers and both metrics, not difference size.Failures fell mainly on FI, and 12,330 characters of Southern Kurdish transliterated output fall outside the folding table and become whitespace. Re-running the system over the full set against the fixed reference is the required correction.Source: §6 · p. 10

No causal claim or ceiling

  • The staged normalization establishes agreement changes under hypothesis transformations, but does not identify causes of residual errors, partition orthographic from acoustic errors, or measure a scoring ceiling.A control using correct Arabic-script renderings of reference sentences through the same transliteration and folding path is required for stronger interpretation.Source: §6 · p. 10

Exploratory annotation comparison

  • The exploratory annotation comparison should not be cited as a result because it was nonsignificant at conventional levels and used sentence-level morphological labels over an incomplete join.The comparison was morphological rather than syntactic.Source: §6 · p. 10

Configuration specificity

  • The results are configuration-specific: they use MP3 recordings with undocumented encoding parameters, existing reference alignments, and a Central Kurdish adapter applied to a variety outside Central Kurdish.No train/development/test split was used because the model was not trained or fine-tuned; the reported figures are not a benchmark result under a shared released protocol.Source: §6 · pp. 9–10

Comparability across Kurdish evaluations

  • WER figures for Kurdish varieties are defensible within their evaluation settings but not portable across studies when segmentation, evaluation-set composition, or scoring representation differ or are undocumented.The proposed reporting convention is to state all three and release the fixed reference alongside the rate.Source: §6 · pp. 10–11

7 Ethics and consent

The study uses previously collected recordings obtained with informed consent and stored under GDPR-compliant safeguards. No new recordings or participant recontacts were made, and speakers are reported only through anonymized two-letter codes without metadata.

7 Ethics and consent

  • Recordings came from the author’s fieldwork with assistant Masoumeh Zarei, were collected with informed consent, and are stored on institutional servers in accordance with GDPR.No new recordings were made for the study, and no participant was recontacted.Source: §7 · p. 11
  • No speaker metadata are reported, and speakers are identified only by anonymized two-letter codes.Source: §7 · p. 11

8 Data and code availability

The paper will release the fixed folded reference and segment-level results, subject to source-corpus sharing restrictions, so the common-reference scoring can be independently checked. It also distinguishes access to resources for running inference from access to the preprocessing and implementation needed to reproduce published metrics.

8 Data and code availability

  • The fixed 9,763-token reference and segment-level results will be released subject to source-corpus sharing restrictions, enabling independent checks of the common-reference design.The reference is the newline-joined set of 1,722 folded segments and is identified by SHA256 340d66a0563363e0c48bb68ae158f7189f6eb730b57e016fa2efa33f772d0d3a.Source: §8 · p. 11
  • Mohammadamini and Tahon (2026) provide openly accessible corpus and checkpoint resources for independent inference, but not the preprocessing or metric implementation needed to reproduce their published 24.26 WER and 4.09 CER.The release includes eleven Parquet shards totaling 4.41 GB and a 0.6-billion-parameter checkpoint, but lacks the reference and hypothesis preprocessing and rate-computation implementation.Source: §8 · p. 11

9 Conclusion

The conclusion reports an unadapted MMS-1B-all baseline under common-reference scoring and shows that orthographic processing substantially changes measured error. It also cautions that residual and cross-model differences cannot be attributed to recognition alone without further controls and consistent data infrastructure.

9 Conclusion

  • On 1,722 segments from five speakers, unadapted MMS-1B-all with the Central Kurdish adapter scores 97.85% WER and 51.20% CER under common-reference scoring.RAW-to-FOLDED reduces measured WER by 13.85 percentage points and CER by 49.72 percentage points; folding beyond transliteration contributes 4.51 and 6.69 points, respectively, while substantial measured error remains.Source: §9 · p. 11
  • Cross-script evaluations should report substitution, deletion, insertion, and hit counts and release the fixed reference so readers can assess orthographic effects and the scoring denominator.Without a fixed released reference and vernacular labels in evaluation metadata, rates can vary across references and per-variety comparisons require reconstruction rather than direct computation.Source: §9 · pp. 11–12
  • The Southern Kurdish model of Mohammadamini and Tahon (2026) did not outperform the Central Kurdish adapter on any speaker, but the comparison establishes direction, not magnitude.It covered 1,703 segments and used a pre-correction 9,912-token reference rather than the fixed 9,763-token reference; a full-set comparison under the fixed reference remains outstanding. It over-generated where the adapter deletes.Source: §9 · pp. 11–12
  • A control is still needed: pass correct Arabic-script renderings of reference sentences through the same transliteration and folding path to measure agreement for already-correct hypotheses.Without this measurement, the composition of the residual error after both transformations is not established.Source: §9 · p. 11
  • 12,330 Southern Kurdish output characters fall outside the folding table, versus 613 for the MMS adapter, making transliteration more damaging for the Southern Kurdish system.This twentyfold difference means part of its apparent underperformance is a scoring-pipeline artefact, not recognition error; the two cannot be separated without a control.Source: §9 · p. 12
  • Per-vernacular evaluation requires reconstructing speaker labels from Mohammadamini and Tahon (2026), because vernacular labels are not carried as a grouping variable in the distributed evaluation data.Adding speaker-level vernacular labels to the metadata would make per-variety evaluation, including Garrusi comparisons, directly available.Source: §9 · p. 12
Loading 2608.16379v1…