Source-linked AI summary
Unadapted Multilingual ASR on a Garrusi Kurdish Evaluation Set: A Common-Reference Staged Normalization Analysis
Hiwa Asadpour
TL;DR
Cross-script Garrusi Kurdish ASR lacks a reliable measurement because direct scoring confounds recognition errors with writing-system differences and normalization can alter reference tokenization. This paper fixes the reference while progressively normalizing hypotheses, finding that the transformation lowers measured WER by 13.85 percentage points and CER by 49.72 points, while substantial error remains.
Problem
Directly scoring Arabic-script hypotheses against Latin Garrusi references conflates writing-system differences with recognition errors, while joint normalization can change reference tokenization.
Method
The study evaluates an unadapted Central Kurdish adapter on Garrusi speech using staged hypothesis normalization against a single fixed reference tokenization.
Results
13.85 percentage points of WER and 49.72 points of CER are removed by RAW-to-FOLDED normalization, while the Southern Kurdish system performs worse on all five speakers and both metrics.
Takeaways & Limitations
Fixed references and released segment-level results make cross-script Garrusi ASR measurements reproducible rather than dependent on reconstructed scoring choices.
Takeaways & Limitations
Incomplete transliteration leaves replacement, Arabic, and unmapped Latin characters in the hypotheses, so the reported rates include scoring-pipeline defects.
Abstract
from arXiv · showhide
Evaluating speech recognition for a Kurdish variety written in a Latin field orthography, using a model that outputs Arabic script, creates a measurement problem before a modelling one: direct scoring treats writing-system differences as recognition errors. Jointly normalizing reference and hypothesis avoids this, but also changes reference tokenization, mixing agreement gains with a change in the scoring denominator. I evaluate MMS-1B-all with the Central Kurdish (ckb) adapter, used as released without adaptation, on 1,722 Garrusi questionnaire segments from five speakers (9,763 reference word tokens; 117.9 minutes). I use a common-reference design: the reference is folded once and fixed at 9,763 tokens, while only the hypothesis representation varies. The raw Arabic-script hypothesis scores 111.70% WER and 100.92% CER, with zero exact word matches. Latin transliteration gives 102.36% WER and 57.89% CER; folding it into the reference's reduced orthography gives 97.85% and 51.20%. Thus RAW-to-FOLDED reduces measured WER by 13.85 points and CER by 49.72 points; folding alone accounts for 4.51 and 6.69 points. Substantial error remains: 14.53% of reference tokens are exact matches, edits are substitution-dominated, and per-segment WER is higher for shorter segments. A Southern Kurdish fine-tuned system (aranemini/southern-kurdish-asr), scored under the same design, performs worse on every speaker (1,703 segments), with 109.56% WER and 55.85% CER. However, 12,330 output characters fall outside the folding table, so these rates must be recomputed against the corrected fixed reference. The MMS output also contains 613 unconverted or unmapped characters, showing that part of the residual error reflects scoring-pipeline limits rather than recognition alone. I will release the fixed reference and segment-level results, subject to source-corpus sharing terms, to support independent checking.
1 Introduction
This section identifies a gap in Garrusi ASR evaluation and frames an initial measurement around transferred recognition and orthographic normalization. The pilot evaluates a related-variety model under a staged common-reference design.
- Research gap: No evaluation reporting recognition rates for Garrusi as such was found in searches through 12 August 2026.The search covered major scholarly databases and multiple spellings of the variety name.
- Research gap: No Garrusi-trained recognition system exists, requiring initial evaluation through a model adapted to a related variety.The selected transfer setting reflects substantial morphosyntactic overlap between Garrusi and Central Kurdish despite phonological and phonetic differences.
- Measurement problem: Direct scoring would count the model’s Central Kurdish Arabic-script output against Latin field-orthography references as recognition error.The study therefore treats orthographic mismatch as a measurement problem before a modelling problem.
- Pilot evaluation: The pilot evaluates MMS-1B-all with the Central Kurdish adapter on elicited Garrusi questionnaire speech.The adapter was chosen because prior linguistic analysis found substantial morphosyntactic overlap between Garrusi and Central Kurdish.
- Normalization design: The scoring design folds the reference once, fixes it, and reuses it unchanged across conditions differing only in hypothesis representation.This is described as a common-reference staged normalization design.
2 Background: Kurdish Varieties and Orthographic Setting
Kurdish comprises multiple, internally varied varieties whose boundaries and classifications remain unsettled. This study evaluates Garrusi speech where Latin field-orthography references are compared with a model emitting Central Kurdish Arabic script, requiring staged normalization.
- Kurdish varieties: Five commonly distinguished Kurdish varieties are Northern Kurdish, Central Kurdish, Southern Kurdish, Gorani, and Zazaki, although their boundaries are unsettled.Northern and Central Kurdish are the least disputed members and have received the greatest concentration of speech-technology work.
- Kurdish varieties: Central Kurdish includes regional varieties such as Mukri, Hewlêrî, Silêmanî, Germiyanî, and Sineyî, so its adapter label does not cover the entire range.The model’s training-variety coverage cannot be audited, so no claim is made about which Central Kurdish varieties are represented.
- Garrusi: Garrusi is a Kurdish variety of Iran, and the evaluated recordings were collected in Hamadan Province’s Mehraban District.The cited general classification does not list Garrusi separately and treats Southern Kurdish as covering varieties including Kelhuri, Feyli, and Kirmashani.
- Orthographic setting: The references use Latin field orthography with phonemic diacritics, whereas the evaluated model emits Central Kurdish Arabic script.Kurdish is written in multiple scripts, and orthographic conventions can differ even when representing the same variety.
- Orthographic setting: Progressive normalization holds the reference fixed, separating agreement changes caused by orthographic processing from the recognition error that remains.This design does not partition residual error into orthographic and acoustic components.
3 Data and Methods
The study evaluates unretrained Kurdish ASR on a fixed 1,722-segment, 9,763-token reference using three hypothesis representations. It compares released MMS-1B-all decoding with a second released Southern Kurdish system under pooled WER and CER scoring.
- Data: 1,722 segments and 9,763 reference tokens formed the evaluation set after excluding segments shorter than 0.3 s and deduplicating resumed-run rows.The source corpus contained 1,765 aligned segments; 43 were excluded for duration, and 247 duplicate rows were removed.
- ASR systems: MMS-1B-all used the released Central Kurdish (ckb) adapter with 16 kHz mono audio, reference-alignment clipping, and greedy CTC decoding.No beam search, external language model, training, fine-tuning, or Garrusi adaptation was used.
- ASR systems: The Southern Kurdish aranemini/southern-kurdish-asr MP32Vec2-BERT CTC model was also run as released and without Garrusi adaptation on the same audio.It was included because Garrusi is classified among Southern Kurdish subgroups by at least one cited classification and the checkpoint is publicly available.
- Normalization: The reference was folded once and fixed, while RAW, TRANSLIT, and FOLDED varied only the hypothesis representation.Transliteration converted Arabic-script output to Latin with KLPT 0.1.7; folding lowercased and normalized strings, mapped characters to a reduced inventory, deleted selected marks, and collapsed whitespace.
- Scoring: WER and CER were computed with jiwer 3.0.3 as pooled corpus-level edit distance divided by total reference word or character counts.Inter-word spaces counted as characters for CER, and the fixed reference ensured S + D + H = 9,763 in every condition.
4 Results
Against a fixed 9,763-token reference, staged hypothesis normalization reduced MMS-1B-all’s measured error, from 111.70% raw WER to 97.85% folded WER and from 100.92% raw CER to 51.20% folded CER. Residual errors were substantial, varied with segment length, and the Southern Kurdish system performed worse under a non-equivalent segment comparison.
- Staged normalization: 97.85% WER and 51.20% CER were the final FOLDED MMS-1B-all results against the fixed reference, with 14.53% exact reference-token matches.RAW scoring produced zero word-level hits, making 111.70% WER a script-mismatch baseline rather than a recognition rate.
- Segment-level patterns: Median per-segment WER was 1.00 for segments of one to seven reference tokens and fell to 0.840 for segments with at least ten tokens.The dataset had a median duration of 3.0 seconds and median length of five reference tokens; 28.7% of segments contained three or fewer tokens.
- Error composition: 60.2% of the Southern Kurdish system’s 10,860 edits were substitutions, compared with 67.9% for FOLDED MMS, while exact matches were 26.9% versus 14.53%.The Southern Kurdish hypothesis contained 12,815 word tokens, or 129.3% of its reference length.
5 Discussion
The discussion treats staged normalization as a measurement analysis rather than a causal decomposition of recognition error. It finds substantial residual error and limits cross-system and transfer claims because the systems, data, segment sets, and scoring conditions differ.
- Staged normalization: 97.85% WER measures agreement after progressive hypothesis conversion with fixed reference tokenization, not the causal share of error attributable to script mismatch.Folding is lossy, collapsing diacritic distinctions and removing pharyngeals, so representation changes can also create or remove measured disagreements.
- Residual error: 67.9% of edits are substitutions, hypothesis length is 93.4% of reference length, and 14.53% of reference tokens are exact matches.Remaining errors may include unresolved orthographic and word-boundary differences, which this design does not identify separately from recognition failure.
- System comparison: The Southern Kurdish fine-tuned system performs worse on all five speakers and both metrics, but the comparison establishes only direction because segment sets and reference token counts differ.The systems also differ in architecture, training data, decoding, pretraining, and hypothesis processing, so the result cannot be attributed to variety adaptation alone.
- Transfer limits: Field-recorded elicited questionnaire speech did not transfer well from a system trained on edited read sentences, but this is one observation on one evaluation set under one scoring design.The comparison also mixes speaking style, recording channel, and variety, so it should not be read as a measurement of dialect distance.
- Segment length: Per-segment WER is higher for shorter segments, descriptively, with one error costing 33 percentage points in a three-token segment and 28.7% of the set containing three or fewer reference tokens.The discussion offers arithmetic instability and possible segment-content differences as explanations, but neither is tested.
6 Limitations
The evaluation is limited by incomplete and uneven coverage, normalization and transliteration defects, configuration-specific conditions, and non-comparable scoring setups. These constraints prevent causal interpretation of residual errors and require explicit reporting of segmentation, evaluation-set composition, scoring representation, and the fixed reference.
- Evaluation coverage: 5 of 30 Phase 1 speakers were analyzed because processing stopped after the fifth without a recorded log; a later run covered 8,240 of 8,650 segments.The later run included 47,160 reference tokens after excluding segments with empty hypotheses.
- Normalization and transliteration: 179 omitted ü occurrences across 165 segments corrupted reference tokenization, including 170 word-internal instances that each split one token into two.The folding table converted characters outside the reduced Latin inventory to whitespace.
- Normalization and transliteration: 502 replacement characters, 74 unconverted Arabic characters, and 36 unmapped Latin diacritics made transliteration incomplete and caused token splitting after folding.The affected outputs were therefore not wholly Latin in the TRANSLIT condition.
- System comparison: 19 of 1,722 segments failed for the Southern Kurdish system, so its rates use 1,703 segments and a smaller reference than the 9,763-token comparison set.Failures were concentrated mainly on FI, and their cause was not established.
- Interpretation: The staged normalization design measures agreement changes under hypothesis transformations but establishes neither causation nor a ceiling for the remaining errors.It does not partition rates into orthographic and acoustic components or measure agreement for an already-correct hypothesis.
- Interpretation: The exploratory annotation comparison was not significant at conventional levels, used sentence-level morphological rather than syntactic labels, and relied on an incomplete join.It should not be cited as a result.
- Configuration and generalizability: MP3 source encoding is undocumented, segmentation follows existing alignments, and the Central Kurdish adapter was applied to a variety outside Central Kurdish; these conditions are setup-specific.The figures should not be generalized to Garrusi speech recognition in general.
- Comparability and reporting: Kurdish ASR rates are not straightforwardly comparable when segmentation, evaluation-set composition, and scoring representation differ; reporting should state these details and release the fixed reference.Per-segment rates use the reference token count as denominator.
7 Ethics and consent
The recordings were collected during the author’s fieldwork with informed consent and are stored under GDPR-compliant institutional procedures. This study made no new recordings, recontacted no participants, and reports no identifying speaker metadata.
- The recordings were collected during the author’s own Garrusi Kurdish fieldwork with assistant Masoumeh Zarei.
- All material was collected with informed consent and stored in accordance with the General Data Protection Regulation on institutional servers.
- No new recordings were made or participants recontacted, and speakers are reported only through anonymized two-letter codes without metadata.
8 Data and code availability
The paper will release the fixed folded reference and segment-level results, subject to source-corpus sharing restrictions, alongside scoring scripts and implementation details. These materials are intended to support independent checking and distinguish running a system from reproducing a published measurement.
- Release materials: 9,763 tokens: the fixed folded reference and segment-level results will be released subject to source-corpus data-sharing restrictions.Results include speaker, segment identifier, timing, reference and hypothesis strings in each representation, and per-segment WER and CER.
- Reproducibility: Scoring scripts are released to distinguish running a system from reproducing a published measurement.The passage cites openly available Southern Kurdish corpus and fine-tuned checkpoint resources as illustrating both sides of that distinction.
- Implementation details: Software versions and model checkpoint revisions used in the study are specified for implementation reproducibility.The listed tools include Python 3.11.9, transformers 4.35.0, torch 2.7.0+cu118, and jiwer 3.0.3; checkpoints include facebook/mms-1b-all and aranemini/southern-kurdish-asr.
9 Conclusion
The pilot measures unadapted MMS-1B-all on Garrusi Kurdish under a common-reference design, yielding 97.85% WER and 51.20% CER. It also shows that fixed references and vernacular metadata are necessary for reproducible cross-system and per-variety comparisons.
- Main result: 97.85% WER and 51.20% CER were obtained by MMS-1B-all with the Central Kurdish adapter on 1,722 Garrusi segments from five speakers.The system was used as released and without adaptation, under a common-reference design with fixed reference tokenization.
- Comparison limits: 1,703 of 1,722 segments were scored against 9,912 pre-correction reference tokens rather than the fixed 9,763-token reference, so the comparison establishes direction, not magnitude.Repeating the comparison over the full set with the fixed reference remains outstanding.
- Comparison limits: 12,330 Southern Kurdish output characters fell outside the folding table, versus 613 for the MMS adapter, making part of the apparent underperformance a scoring-pipeline artefact.The Southern Kurdish system shares the transliteration step, and the passage reports a twentyfold difference.
- Data infrastructure: Without a fixed released reference and vernacular labels in evaluation metadata, reported rates can differ across references and per-variety comparison requires reconstruction.The distributed evaluation data does not carry the grouping variable needed to contrast Garrusi with other Southern Kurdish vernaculars.