Evaluation of Phonetic Encoding Algorithms on Transcription Datasets

• Phonetic codes support comparisons of phonetically similar but orthographically distinct words, with applications including record linkage, search, and speech-transcription similarity.Phonetic representations have also been used in neural-network training for downstream language tasks.Source: §1 · p. 1
• American Soundex preserves the first letter and truncates codes to fixed length, while later variants refine consonant groups or transformations for pronunciation and borrowed words.Soundex maps consonants to digits by phonetic clusters, except for the preserved initial letter.Source: §1 · p. 2
• Prior encoder evaluations often measured spelling correction, duplicate detection, or recall; collision rate estimates potential recall, and low-granularity codes may suit some tasks.A Greek Soundex-like code was reported to retrieve similar-sounding words better than full phonetic transcription because of its higher recall.Source: §1 · p. 2
• The paper proposes a consistency measure for comparing phonetic codes with ground-truth transcriptions, addressing the reported absence of a systematic precision-focused evaluation method.Combined with collision rate, the measure supports comparisons between encoders with similar collision rates or similar consistency scores.Source: §1 · p. 3
• The evaluation adapts the H¨ullermeier-Rifqi Index with normalized edit distance and examines encoders in English, German, French, and Swedish, also treating writing systems as inherent phonetic representations.The paper presents this writing-system evaluation as a way to measure orthographic transparency.Source: §1 · p. 3

2.1. Rand Index

• The Rand Index measures clustering agreement by counting concordant sample pairs and dividing by the total number of pairwise combinations.A pair is concordant when its together-or-apart grouping agrees in the predicted and ground-truth partitions.Source: §2.1 · pp. 3–4

2.2. H¨ullermeier-Rifqi Index

• The H¨ullermeier-Rifqi Index generalizes the Rand Index to compare fuzzy partitions, using pairwise discordance and its complement, concordance.The index sums discordance across all sample pairs and subtracts the resulting ratio from one.Source: §2.2 · p. 4

2.3. Selection of the Distance Metric

• Because string comparisons depend on order, the method uses normalized edit distance to convert insertion, deletion, and substitution distance into pairwise similarity.Similarity is one minus edit distance divided by the maximum string length.Source: §2.3 · p. 5

2.4. Reformulation of the Index

• The reformulated index separates pairs using transcription-similarity thresholds and seeks positive and negative sets of comparable size, randomly subsampling negatives when needed.In this setting, θ+ = 0.5 and θ− = 0.Source: §2.4 · p. 5
• Positive concordance (PC) and negative concordance (NC) provide fuzzy analogues of sensitivity and specificity, respectively; overall concordance (OC) corresponds to accuracy and is adjusted against random codes.The random codes match each encoder’s code lengths and alphabet.Source: §2.4 · p. 6

3.1. Dataset; 3.2. Preprocessing

• The dataset covers four languages with phonetic encoders and six additional languages evaluated by their writing systems; 10,000 samples per language were selected after preprocessing.Preprocessing removed non-alphabetic characters, retained IPA vowels and consonants, and excluded multiple-transcription samples and heteronyms.Source: §3.1 · p. 7, §3.2 · p. 7, §3.3 · p. 7

3.3. Evaluation

• Metaphone (MP) achieved the highest English AOC at 63.39, with PC 89.41 and NC 93.57.The English results are summarized in Table 1.Source: §3.3 · p. 8, Table 1 · p. 9
• RI, SX, and MRA trade recall for precision as consonant handling grows granular; PHEX reaches the highest collision rate (11.08) while retaining moderate intrinsic recall.RI, SX, and MRA have progressively lower PC and higher NC; PHEX combines Metaphone features with a Soundex-like procedure.Source: §3.3 · pp. 8–10, Figure 1 · p. 10
• Phonet led German encoders at AOC 75.83, Early Henry exceeded the French writing system at 73.01 versus 59.08, and Wåhlin exceeded the Swedish writing system at 81.35 versus 79.31.Wåhlin's collision rate was 1.01; the German, French, and Swedish comparisons appear in Tables 2–4.Source: §3.3 · pp. 10–11, Table 2 · p. 11, Table 3 · p. 11
• Stricter maximum-length limits increase collision rates while decreasing AOC, especially for Early Henry compared with SX.The separate experiment varied maximum length for SX and Early Henry.Source: §3.3 · p. 11, Figure 2 · p. 12

3.4. Orthographic Transparency

• AOC ranks Finnish as having the most transparent orthography and English the most opaque among the ten evaluated languages.The rankings align with the cited literature's characterization of Finnish as fully transparent and English as notably opaque.Source: §3.4 · pp. 12–13, Figure 3 · p. 13
• AOC rankings correlate more strongly with OTEANN WRITE accuracy (τ = 0.82) than READ accuracy (τ = 0.46).The comparison trained OTEANN using 10,000 randomly selected samples per language, split 90% for training and 10% for testing.Source: Table 5 · p. 13, §3.4 · p. 13

4. Conclusion and Future Work

• The proposed scheme evaluates phonetic-code consistency with transcriptions using concordance measures and collision rates, without relying on external evaluation.The concordance measures serve as fuzzy analogues to sensitivity, specificity, and accuracy.Source: §4 · p. 14
• The scheme was used to assess encoders across multiple languages, with strengths and weaknesses aligning with earlier observations in the literature.Source: §4 · p. 14
• Applied to written forms, the introduced measure may indicate a language’s orthographic transparency, with results consistent with previous studies.Source: §4 · p. 14
• Future work could test additional distance metrics and score standardization, especially for syllabic writing systems, and datasets limited to proper names may yield different encoder rankings.Source: §4 · p. 14