PaperlayerView exact revision on arXiv

Source-linked AI summary

What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts

Mohamad Al Mdfaa, Nursultan Askarbekuly, Ahmed Helaly, Ubai Sandouk, Manuel Mazzara

arXiv:2609.12085v1cs.CL

TL;DR

Quran-recitation transcript differences do not by themselves show whether a discrepancy is an unresolved mistake, repetition, repair, or accepted spelling variant. The paper annotates and evaluates these typed events; seven of eight pilot runs score 0.79–0.89 within 20 minutes, versus both reported baselines.

  • Problem

    ASR transcript differences locate Quran-recitation discrepancies but cannot distinguish unresolved mistakes from repetitions, repairs, and accepted spelling variants, which may mislead learner feedback.

    Source: §1 · p. 1

  • Method

    The authors annotate production transcript-reference differences as typed, localized events, define label-and-position scoring, and run a coding-agent pilot alongside baseline comparisons.

    Source: §1 · p. 1, §4 · p. 3, §5 · p. 3

  • Results

    Seven of eight pilot runs scored 0.79–0.89 within 20 minutes, compared with 0.525 for a plain diff and 0.518 for adapted production components.

    Source: §7 · p. 6

  • Takeaways & Limitations

    Agent consensus can serve as a rubric audit by locating adjudication-defined boundaries for review, but it does not establish which reading is correct.

    Source: §7 · p. 6

    The pilot measures only the algorithm half of the task: no run annotated before building, leaving agents’ ability to produce labels that survive review untested.

    Source: §8 · p. 7

Abstract

from arXiv · show

Checking Quran recitation from an ASR transcript requires distinguishing unresolved mistakes from repetitions, repairs, opening formulas and accepted spelling differences. We report a completed human annotation of 100 production recording cases: 348 scored units and 162 localized events across ten combined labels. An executable evaluator scores labels and word positions together. A plain diff reaches label-aware F1 0.525 and localization F1 0.826; adapted production cleaner/alignment components reach 0.518 and 0.786, with exact-span F1 0.505 for both. Correcting the adapter's word coordinates recovers all five annotated repetition events, showing why annotation interfaces must be checked before interpreting baseline failures. In a preliminary pilot, eight single 20-minute runs across three coding agents and eight models span label-aware F1 0.143 to 0.892: seven land far above every baseline, and one collapses below the naive diff from a missing normalization step. Across the six, 970 of 972 gold-event instances draw an overlapping prediction, so what remains is not detection but convention: span extent, and the labels whose boundary is stipulated by adjudication rather than visible in the text. Seven of 162 events defeat all six same-day runs, five of them one orthographic rule, and the strongest run still misses the same ones. No run annotated before building, so the pilot measures the algorithm half of the task only.

1 The distinction the task turns on

Edit distance finds transcript differences but cannot determine whether they are unresolved mistakes; the paper instead annotates differences as typed, localized events judged from text rather than speaker attribution.

1 The distinction the task turns on

  • 74% of reviewed ayah units carry no event, and repetitions, immediate repairs, and accepted orthographic variants can be differences a human listener would not mention.Flagging these as unresolved mistakes may mislead learners; the study does not measure the relative user cost.Source: §1 · p. 1
  • Labels describe what differs between transcript and reference, not whether the reciter or ASR caused it, and unwritten vowel and tajweed errors are outside the transcript-only rubric.The annotation treats the text rather than the speaker as its basis.Source: §1 · p. 1
  • The work contributes a production-recitation corpus, a scheme giving repetition, repair, and accepted spelling variant distinct categories, and a first measurement of a coding agent asked to solve annotation and algorithm together.These categories are distinguished from degrees of error.Source: §1 · p. 1

2 Related work

Prior Quran-recitation systems and datasets mainly assess audio, while this work annotates transcript differences. Its rubric treats repetitions and repairs as disfluencies, and evaluates coding agents beyond their usual engineering benchmarks.

2 Related work

  • Prior Quran-recitation tools and datasets focus on audio or mark recordings and phones correct or incorrect, whereas this work labels, types, and locates transcript differences.Prior work includes verse delimitation, tajweed classification, Quranic ASR, and corpora of labeled recitations or aligned audio and text.Source: §2 · p. 2
  • The rubric adapts disfluency annotation, which treats repetitions and repairs as structures rather than errors, to an ASR transcript compared with a fixed reference.Pronunciation assessment instead scores a learner attempt against a reference.Source: §2 · p. 2
  • Coding agents are typically evaluated on software and ML engineering benchmarks, whereas this work scores them on Quran-transcript annotation.Source: §2 · p. 2

3 Taxonomy and annotation

The section defines a single-label rubric with explicit adjudication rules and reports a human-approved, purposively selected set of 100 recording cases. Annotation approval is withheld for recordings with unresolved questions, rather than represented by an uncertain label.

3 Taxonomy and annotation

  • The rubric replaces separate event-type and verdict fields with one combined event label, while distinguishing reviewed-clean units from units not yet reviewed.Event labels are substitution_mistake, omission_mistake, insertion_mistake, substitution_corrected, omission_corrected, repetition_benign, letters_benign, spelling_benign, basmala_benign and isti3adha_benign; clean marks a reviewed unit with no event.Source: §3 · p. 2
  • Adjudication includes omissions at recording endings, repetitions spanning both occurrences, and contextually accepted residual spellings after agreed normalization.A wrong-then-correct attempt is corrected; a correct-then-incorrect restatement is a substitution_mistake spanning both attempts. Localization uses verbatim whitespace tokens and half-open spans, with empty spans anchoring omissions or insertions.Source: §3 · p. 2
  • One reviewer approved each recording against its exact ayah reference, and any unresolved question kept the entire recording out of the set.Approval covered all 314 ayah units and 34 opening units; each of the 115 within-ayah events has a written note explaining the target difference and label.Source: §3 · p. 2
  • The purposively selected set contains 100 approved recordings, 314 ayah units and 34 opening units, and its counts do not estimate error prevalence or establish generalization.It includes 233 clean ayah units across 274 ayahs in 38 surahs and ten learner identities; gold answers and selected membership remain in a private bundle.Source: §3 · p. 2
  • Across 972 gold-event instances, six same-day runs were 84.0% correct, 6.1% localized but mislabelled, and 10.0% unmatched.Correct requires both the label and a span pair above tolerance; reviewed clean units are not event rows.Source: Table 1 · p. 3

4 An executable scorecard

The executable scorecard evaluates event labels and transcript/reference spans after ayah detection and segmentation are supplied. It scores opening formulas as separate units and uses explicit matching rules and multiple complementary metrics.

4 An executable scorecard

  • Systems receive reviewed ayah splits and references, then return labeled events with half-open transcript and reference spans; detection and segmentation are not scored.Those tasks are supplied and handled by a companion submission.Source: §4 · p. 3
  • Opening isti’adhah and basmala formulas are scored in separate units against empty references, testing whether a preceding basmala is benign or an insertion.A compound opening yields two events over one token array.Source: §4 · p. 3
  • Predictions and gold events are paired one-to-one by maximum similarity only when both spans overlap with minimum similarity of at least 0.50.Pairing ignores labels for separate localization scoring; invalid predictions count as false positives, and empty spans match only empty anchors exactly or within one token.Source: §4 · p. 3
  • The headline measure is micro label-aware event F1, supplemented by exact-span and macro F1, localization precision/recall/F1, and raw event-cost counts.The cost reports missed mistakes plus false flags per 100 units at r = 1 and r = 2; neither rate ranks systems, and predicting nothing has F1 0.Source: §4 · p. 3
  • Evaluator v2.1 passes synthetic regression tests and private gold-oracle checks, while a 0.50 overlap threshold is chosen for explainability rather than tuning.Sweeping tolerance from 0.05 to 1.00 moves baseline micro F1 by about four points but leaves exact-span F1 unchanged; one-token anchor slack changes no baseline score across 0 to 10.Source: §4 · p. 3

5 What existing systems do

The plain diff locates many events but recognizes only two labels; the adapted production cleaner/alignment baseline recovers repetitions but still has substantial label gaps.

5 What existing systems do

  • The plain diff locates 126 of 162 gold events (recall 0.778), with localization F1 0.826 and label-aware F1 0.525.Localization F1 is not the percentage of gold events found.Source: §5 · p. 3
  • Per-label F1 is 0.94 for omissions and 0.85 for substitutions, but zero for the other eight labels; 34 of 46 insertion predictions hit opening-text units, versus one gold insertion event.For compound openings with two gold formulas, the diff can return one insertion spanning both.Source: §5 · pp. 3–4
  • The second baseline adapts production cleaner and alignment components to the event schema, retaining token identities through deletion and mapping cleaner notes to reference phrases and both attempts.Its heuristic treats a stumble as a repair only when the following equally long surviving span matches the reference; unlocalizable removed fragments remain extra text, and production’s partial-start scoring policy is retained. These are baseline choices, not additional gold decisions; each baseline also retains its own comparison normalization.Source: §5 · p. 4
  • The adapter predicts five repetitions and recovers all five (F1 1.0); six corrected-substitution predictions recover one of four gold events (F1 0.20).It has nonzero F1 on four labels and zero on six, never predicting either opening-formula label, letters, contextual spelling, or omission corrections. An earlier adapter lost valid repetition credit through incorrect coordinates; the remaining gaps motivate explicit event modeling but do not establish that cleaning causes the errors or measure every deployed-application feature.Source: §5 · p. 4

6 A preliminary agent pilot

In the preliminary pilot, seven of eight runs scored far above every baseline, while the six same-day runs’ remaining errors largely concerned span conventions and stipulated rubric boundaries.

6 A preliminary agent pilot

  • 7 of 8 runs reached micro F1 0.79–0.89, far above every baseline, with zero invalid predictions across 348 units.Each model ran once, so the results do not rank agents or models generally.Source: §6 · p. 4

Where the six runs fail, and why

  • 970 of 972 gold-event instances drew an overlapping prediction, leaving span convention and label choice as the main residual errors.Of 97 unmatched instances, 95 had overlapping predictions below the matching tolerance; the rubric’s grouping of both attempts in a repeat or repair explains many span mismatches, while 59 mislabelled instances concentrated in spelling_benign and letters_benign.Source: Where the six runs fail, and why · pp. 4–5
  • 7 of 162 gold events defeated all six runs, including five shared wasl spelling cases labeled substitution_mistake.Opening-formula labels without teaching examples were handled at 100% and 88%, while spelling_benign with one example was handled at 24%; the contrast is structural versus adjudicated boundaries.Source: Where the six runs fail, and why · p. 5

How the six runs agree with each other

  • The six runs emitted 145 to 172 events against 162 in gold and raised only three spurious mistake flags across 233 reviewed clean units.False positives ranged from 16 to 31 per run and were dominated by substitution_mistake.Source: How the six runs agree with each other · p. 5, Table 3 · p. 5
  • Across six independent rubric-reading runs, all six matched the gold label on 114 of 162 events (70%); five events had unanimous disagreement, four label disagreements and one too-narrow span localization.These disagreements point to a rubric rule and span convention; consensus signals stipulative work but cannot decide whether the stipulation is right.Source: How the six runs agree with each other · p. 5
  • Fable 5.1 tied the best micro F1 at 0.892 and led the set in exact-span F1 (0.879), localization (0.947), and macro F1 (0.702).Its span conventions were strong, but eight of its nine mislabelled events were spelling_benign; as a single run a day later under a different permission setting, it cannot isolate the cause of the difference.Source: How the six runs agree with each other · p. 5
  • At micro F1 0.143, gpt-oss-120B performed worse than either baseline after omitting Arabic orthographic normalization before diffing.Ordinary hamza and alif-form differences drove 545 of 567 predicted events to substitution_mistake, and the run produced 22 invalid events.Source: How the six runs agree with each other · pp. 5–6

7 Discussion

The discussion argues that labeling recitation events requires distinguishing benign or corrected events from mistakes, while the pilot’s strong algorithmic results leave annotation untested. Remaining errors cluster around boundaries fixed by adjudication, making agent consensus useful for auditing the rubric rather than replacing human judgment.

What the label scheme buys

  • Of 162 located events, 73 (45%) were benign or corrected, so treating them as mistakes would mislead learners despite strong localization.The baselines scored 0.826 and 0.786 on localization but only 0.525 and 0.518 label-aware, a gap the passage attributes to naming rather than finding.Source: §7 · p. 6

What the loop did, and what it did not

  • Seven of eight runs reached 0.79 to 0.89 within 20 minutes, versus 0.525 for a plain diff and 0.518 for adapted production components.Across 1,398 clean-unit decisions, the runs raised only three spurious mistake flags.Source: §7 · p. 6
  • The pilot did not test annotation-first development: seven runs skipped the training pool, while Fable 5.1 used it only to check its finished detector.Studying that schema requires requiring annotation delivery and giving the loop time and incentive to produce annotations.Source: §7 · p. 6

Where the two halves meet

  • Errors cluster at adjudicated boundaries, so agent consensus can locate rubric clauses for human review but cannot establish which reading is correct.The runs handled input-visible boundaries well but struggled with context-dependent spelling acceptance and the extent of repeated-phrase spans; five events were missed by all six runs.Source: §7 · p. 6

8 Limitations and what comes next

The section limits claims from a small, single-reviewer sample whose inputs have prior public exposure, and frames the pilot as an informal algorithm-only step toward a controlled comparison.

Limitations

  • The purposive 100-case set is small and skewed (one learner supplies 46 recordings), making macro F1 a coverage check; one reviewer leaves inter-annotator agreement unmeasured.The authors would first add a second annotator on a sample, beginning with the five events where all six runs agree against gold.Source: §8 · p. 6
  • All 100 inputs are already public, so runtime isolation cannot establish no prior exposure; the prepared 127-case, 888-unit development release is not a frozen hidden test.Twelve cases also appeared in an earlier machine-labeled pilot; the development release excludes complete gold copies and event-bearing gold chunks.Source: §8 · pp. 6–7

What comes next

  • Future comparisons will fix unlabelled training data, rubric, examples, tools, and budget while allowing annotation, model training, or rule-writing as strategies.No pilot run annotated first, so the 20-minute pilot measured only the algorithm half; whether agents annotate and whether labels survive review require sufficient time and observable delivery.Source: §8 · p. 7
  • The pilot was an informal end-to-end harness check before preregistration, and isolation differed by runtime, with Antigravity network access and model selection not externally verified.All seven runs saw only unlabelled data, the guide, and teaching examples, and froze solutions before gold scoring; high scores support the combined process, not necessarily its training labels, which need independent review.Source: §8 · p. 7
Loading 2609.12085v1…