Source-linked AI summary
PRiSM: Benchmarking Phone Realization in Speech Models
Shikhar Bharadwaj, Chin-Jou Li, Yoonjae Kim, Kwanghee Choi, Eunjung Yeo, Ryan Soh-Eun Shim, Hanyu Zhou, Brendon Boldt, Karen Rosero Jacome, Kalvin Chang, Darsh Agrawal, Keer Xu, Chao-Han Huck Yang, Jian Zhu, Shinji Watanabe, David R. Mortensen
TL;DR
Phone-recognition evaluations largely measure surface transcription accuracy, while metrics and datasets remain inconsistent and may not capture phonetic capability or practical utility. PRiSM addresses this gap with standardized intrinsic and extrinsic benchmarking, finding that multilingual exposure and encoder-CTC architectures support stronger, more stable performance while LALMs lag behind specialized models.
Problem
Phone-recognition evaluations lack standardized metrics and broader evidence beyond transcription accuracy, which may not reflect phonetic capabilities or practical utility.
Method
PRiSM benchmarks phone-recognition systems through standardized intrinsic transcription evaluation and extrinsic transcription and representation probes on downstream tasks.
Results
Multilingual training improves performance, encoder-CTC architectures are more stable across tasks, and Large Audio Language Models generally lag behind specialized phone-recognition models.
Takeaways & Limitations
Because models behave differently on transcription and downstream tasks, PR systems should be benchmarked with both intrinsic and extrinsic evaluations.
Takeaways & Limitations
PRiSM’s coverage of languages, dialects, accents, and speaking styles remains incomplete because curated datasets are limited and may contain corpus biases.
Abstract
from arXiv · showhide
Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis. Despite prolonged efforts in developing PR systems, current evaluations only measure surface-level transcription accuracy. We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception through intrinsic and extrinsic evaluation of PR systems. PRiSM standardizes transcription-based evaluation and assesses downstream utility in clinical, educational, and multilingual settings with transcription and representation probes. We find that diverse language exposure during training is key to PR performance, encoder-CTC models are the most stable, and specialized PR models still outperform Large Audio Language Models. PRiSM releases code, recipes, and datasets to move the field toward multilingual speech models with robust phonetic ability: https://github.com/changelinglab/prism.
1 Introduction
PRiSM addresses gaps in phone recognition evaluation by benchmarking both transcription accuracy and downstream utility, including explicit transcriptions and latent representations. It provides an open-source, reproducible framework for evaluating diverse PR systems and advancing robust, generalizable phonetic modeling.
- Motivation: Phone recognition preserves acoustic nuances beyond language-specific phonological constraints, supporting cross-lingual speech processing and clinical and educational applications.PR transcribes speech into phonetic units that capture physical sound realizations independently of language-specific phonological constraints.
- Motivation: Existing evaluations suffer from nonstandardized metrics, scarce phonetic transcription data, and transcription errors that may not reflect phonetic capability.Expanding test datasets to mitigate bias scales poorly because phonetically transcribed data are scarce.
- Motivation: Transcription-only metrics miss the utility of latent representations, whose rich phonetic cues can support downstream performance independently of explicit transcriptions.The relationship between transcription accuracy and downstream performance is often assumed rather than empirically established.
- PRiSM: PRiSM evaluates PR systems intrinsically through transcription error and extrinsically through clinical, educational, and multilingual tasks using transcriptions and hidden representations.The benchmark covers specialized PR models, general speech-to-text systems, and Large Audio Language Models.
- PRiSM: PRiSM is the first open-source benchmark for PR systems, releasing code, evaluation recipes, and datasets through a reproducible and expandable framework.The framework is intended to support analysis of model behavior and training strategies and inform model choices.
- PRiSM: The benchmark aims to establish a common evaluation basis for PR systems that capture robust and generalizable phonetic information across resource conditions.This goal motivates standardized evaluation beyond surface-level transcription accuracy.
2 Background
Phone recognition maps speech to phonetic symbols such as IPA, with language-agnostic systems targeting generalized phonetic representations across languages. The section motivates evaluating both inspectable transcriptions and richer internal representations through intrinsic and extrinsic tasks.
- Definition: Phone recognition maps speech to phonetic symbols such as IPA, and “PR system” broadly denotes language-agnostic speech-to-IPA systems.PR is presented as a variant of speech-to-text mapping from speech to phonetic symbols.
- Approaches: Modern PR systems are fine-tuned from ASR models or trained from scratch on ASR datasets with transcriptions automatically converted to IPA using G2P tools.Language-specific systems rely on phoneme inventories, whereas this work focuses on language-agnostic approaches.
- Transcriptions and representations: Explicit phonetic transcriptions are human-inspectable and clinically useful, but discrete symbols lose information about non-linguistic variation in continuous speech.Faithful transcriptions can help clinicians understand speech sound disorders and design interventions.
- Transcriptions and representations: Internal model representations complement transcriptions by retaining richer, temporally aligned acoustic-phonetic, articulatory, and semantic information.End-to-end ASR models such as Whisper and WavLM are identified as widely used sources of speech-model representations.
- Evaluation: Existing benchmarks evaluate phonological knowledge in text or speech modalities, while PRiSM evaluates PR through intrinsic and extrinsic tasks.The supplied related-work passages identify text tasks such as G2P and rhyme judgment and speech evaluation using representation output.
3 Evaluation Framework of PRiSM
PRiSM evaluates phone recognition through intrinsic transcription accuracy and extrinsic downstream utility, using feature-sensitive transcription comparison plus complementary transcript and representation probes across clinical, educational, and multilingual tasks.
- 3.1–3.2 Evaluation Framework: PRiSM combines intrinsic evaluation against gold transcriptions with extrinsic evaluation of transcriptions and internal representations on downstream tasks.Transcriptions provide interpretable explicit phonetic content, while representations may encode non-phonetic information.
- 3.1 Intrinsic Evaluation: PFER measures transcription distance by computing edit distance over articulatory features rather than treating phones as indivisible tokens, unlike PER.It captures feature-level errors such as roundness or voicing.
- 3.2 Extrinsic Evaluation: Downstream tasks vary seen-language speech, including regional and non-native speech, and unseen languages to test language-agnostic phonetic knowledge.Strict unseen-language verification is not possible for closed LALMs because their full training corpora are unavailable.
- 3.2 Extrinsic Evaluation: PRiSM uses a transcript probe with a text-based bi-GRU and a representation probe with temporally pooled final hidden states and an MLP.The two probes are complementary because phonetic transcripts and hidden representations differ substantially in form.
- 3.2 Extrinsic Evaluation: The downstream evaluation covers pathological speech assessment, L2 speech assessment, and multilingual speech identification, where phonetic information supports diagnosis, pronunciation feedback, accent classification, or identification.These tasks connect phonetic representations to clinical, educational, and multilingual applications.
4 Benchmarked Models
The benchmark compares fine-tuned Wav2Vec2 variants, multilingual encoder-CTC models, POWSM encoder-decoder and CTC variants, and state-of-the-art large audio language models.
- Wav2Vec2Phs: Wav2Vec2Phs variants differ in pre-training coverage and phone-recognition fine-tuning datasets.The family includes MultiIPA, W2V2P-LV60, and W2V2P-XLSR53.
- ZIPAs: ZIPA-CTC and ZIPA-CTC-NS are multilingual encoder-CTC models trained from scratch, with ZIPA-CTC-NS additionally trained on large-scale pseudo-labeled data.The pseudo-labeled data are generated from ZIPA-CTC.
- POWSMs: POWSM is an attention-based encoder-decoder trained on the ZIPA dataset, while POWSM-CTC is an encoder-CTC variant trained for comparison.POWSM is also augmented for other speech-to-text tasks.
- LALMs: The benchmark includes Gemini 2.5 Flash and Qwen3-Omni-Instruct as state-of-the-art large audio language models.Gemini 2.5 Flash is closed-source, whereas Qwen3-Omni-Instruct is open-weight.
5 Results and Discussion
Results show that intrinsic and extrinsic evaluations reveal complementary aspects of phone recognition, with performance depending strongly on language coverage, model architecture, dataset, and task. Encoder-CTC models are generally stable and competitive, while LALMs remain limited overall despite some task-specific strengths.
- ZIPA-CTC-NS performs well across settings, Whisper excels in RP, and LALMs generally remain less competitive.
- CTC-based models generally outperform LALMs, followed by AED models, while multilingual coverage benefits performance, especially when languages appear during pretraining but not fine-tuning.On PR-saa, POWSM performs poorly, whereas the text-based G2P model achieves a PFER of 10.2 despite modeling only canonical pronunciations.
- Specialized PR models exhibit a TP–RP trade-off: Wav2Vec2Phs are strong on TP for L2 speech, whereas ZIPAs excel on RP but underperform on TP.Pathological speech benefits more from RP, L2 speech falls in the middle, and multilingual tasks tend to favor TP.
- LALM performance is task-dependent, with Qwen3-Omni-Instruct competitive on pathological-speech TP but generally weak in zero-shot settings and on non-English tasks.Few-shot results improve pathological-speech tasks but have mixed effects elsewhere, with no consistent gains as the number increases.
- Combining intrinsic and extrinsic tasks is essential because multilingual training improves performance and encoder-CTC architectures provide more stable PR in new domains.Intrinsic evaluation alone may not fully capture phonetic capabilities, while extrinsic performance varies between TP and RP by task.
6 Analysis
The analyses show that PR performance depends on acoustic fidelity, multilingual exposure, and architecture or training objectives, while phone-based probes outperform hidden representations for dialectal geolocation. Large Audio Language Models remain biased and weak on zero-shot phonetic tasks.
- 6.1 Architectural Choices: Phone masking tests whether PR systems rely on phonotactic patterns rather than information derived directly from the acoustic signal.The experiment masks p% of TIMIT phones and measures PFER against references containing only the remaining phones.
- 6.1 Architectural Choices: CR-CTC improves stability beyond encoder-only design by encouraging similar representations despite noise in the input speech.ZIPAs are encoder-only Zipformer models trained with consistency-regularized CTC, indicating that stability is not determined by architecture alone.
- 6.2 Multilingual Generalization: Broader language coverage during pre-training and supervised training produces more precise phone predictions for unseen languages.POWSM-CTC is strongest for phone inventory induction, while extended multilingual pseudo-labeled training improves ZIPA-CTC-NS precision.
- Zero-Shot LALM Analysis: LALMs show geographic mode collapse on GEO-v and accent-cluster bias on L1-eda, with thinking mode reducing F1-score from 32.7% to 24.9%.Gemini 2.5 Flash reaches 6.5% hit@1 on GEO-v, while Romance bias misclassifies 25.8% of Slavic/Balkan and 28.5% of South Asian accents.
7 Conclusion
PRiSM introduces a standardized, open-source benchmark for evaluating phone recognition through transcription and downstream tasks. The results show that PR and downstream performance can differ, motivating evaluation in both categories.
- Benchmark contribution: PRiSM is presented as the first standardized benchmark for phone recognition transcription and downstream task performance.The benchmark measures capabilities across both evaluation categories.
- Resources: The project open-sources datasets in an easy-to-use format together with its toolkit.
- Evaluation recommendation: Models behave differently on phone recognition and downstream applications, so the authors recommend benchmarking both categories for comparison.
Limitations
PRiSM is limited by incomplete and potentially biased dataset coverage, the non-objective nature of phonetic transcription, and constraints in both transcript and representation probes. Its reported comparisons are intended as comparative insights rather than best-achievable model performance, and PR systems may be misapplied without expert supervision.
- Dataset coverage: Curated-dataset availability leaves PRiSM’s coverage of languages, dialects, accents, and speaking styles incomplete and potentially biased.These limitations may reflect biases in the underlying speech corpora.
- Transcription ground truth: Phonetic transcription lacks a single objective ground truth because it depends on annotation guidelines, annotator judgments, and the chosen phone inventory.The IPA-based interface may also miss or normalize away language-specific or gradient phonetic phenomena.
- Evaluation probes: Transcript probes can overfit to spurious cues, while representation probes may not capture all distributed phonetic information.Dataset bias and noisy transcripts from low PR quality can contribute to misleading transcript-probe results.
- Interpretation of results: PRiSM reports fundamental phonetic capabilities and comparative insights, not the best possible performance achievable for each model.Both intrinsic and extrinsic evaluations are necessary, but each has limitations.
- Ethical and practical risks: PR systems may unfairly label speakers when misapplied without appropriate expert supervision, particularly in educational, clinical, demographic, or geographic contexts.The work states that its data were ethically sourced through permissive licensing or proper consent, while acknowledging sensitive personal information in speech datasets, especially pathological-speech data.
A Dataset details and Licenses … B.1 Task Metric: F1 of Phone Inventory (F1-PI)
The benchmark documents dataset licenses and splits, spans seen and unseen languages plus clinical, educational, multilingual, and geolocation tasks, and uses F1-PI to measure phone-inventory overlap. Its datasets include varied accents, manually annotated phonetic transcriptions, dysarthric speech, L2 speech, language identification, and regional speech data.
- A.2 Datasets in Intrinsic Evaluation: Intrinsic evaluation covers seen-language variation through TIMIT, Speech Accent Archive, and L2-ARCTIC speech with phonetic transcriptions.TIMIT contains six regional varieties of American English, while the Speech Accent Archive spans non-native English speakers across 391 L1 languages.
- A.2 Datasets in Intrinsic Evaluation: Unseen-language evaluation uses DoReCo, VoxAngeles, and Tusom2021, covering broad or narrow phonetic transcriptions across diverse and low-data languages.VoxAngeles contains 95 languages, and Tusom2021 provides word-level data for the low-data language Tusom; tones were removed.
- A.3 Datasets in Extrinsic Evaluation: Extrinsic clinical evaluation predicts dysarthria severity or intelligibility from phonetic representations using the English UASpeech and Italian EasyCall corpora.UASpeech provides speaker-level intelligibility scores, and increasing dysarthria severity is associated with reduced intelligibility and impaired phoneme production.
- A.3 Datasets in Extrinsic Evaluation: Educational and sociolinguistic tasks assess L2 English proficiency and classify speakers’ L1 backgrounds using Speechocean762 and EdAcc data.Speechocean762 provides utterances and sentence-level proficiency scores on a 0-10 scale for L1 Chinese, L2 English speech.
- A.3 Datasets in Extrinsic Evaluation: Multilingual tasks include language identification over 102 FLEURS languages and speech geolocation identification using Hindi-belt data from Vaani.These tasks evaluate whether phonetic representations distinguish seen and unseen languages and capture geographic phonetic variation.
- B.1 Task Metric: F1 of Phone Inventory (F1-PI): F1-PI measures overlap between a system’s transcribed phones and a language’s ground-truth phone inventory using exact or fuzzy set membership.The F1-score is the harmonic mean of |A − B|/|A| and |B − A|/|B|, and the metric requires only a reference inventory for the target language.
B.2 Summary Metric: PRiSM Extrinsic Score
PRiSM aggregates six extrinsic tasks with logarithmic sample-size weighting, balancing statistical significance against domination by large datasets. Rankings remain broadly stable across alternative weighting schemes, with especially strong consistency for TP and RP.
- Score construction: The aggregate score uses logarithmic weights wi = ln(Ni), allowing larger datasets to contribute more without overwhelming smaller, high-variance datasets.The method addresses test-set sizes ranging from N = 7762 to N = 287.
- Score construction: The score aggregates K = 6 behavior-differentiating tasks: DYS-ez, DYS-ua, CSD-us, L1-eda, L1-arc, and L2-so.The task set includes the largest test set, N = 7762, and the smallest, N = 287.
- Weighting sensitivity: Ranking consistency is assessed against ln(Ni)-weighted rankings using Kendall’s τ across uniform, √Ni, Ni, and 1/Ni weighting schemes.Ni denotes the number of samples in task i.
- Weighting sensitivity: For TP, the top-3 cluster MultiIPA, ZIPA-CTC-NS, and W2V2P-XLSR53 persists under all schemes except 1/Ni, with τ ≥0.83.The 1/Ni scheme assigns 77% of total weight to CSD-us, whose test set has N = 287.
- Weighting sensitivity: For RP, rankings show τ ≥0.78 across schemes, reach τ = 1.0 under √Ni and Ni weighting, and retain Whisper as the top-ranked model.Whisper remains first under every tested weighting scheme.
C Experimental Setup … F Effect of Layer Selection on RP
The experimental setup specifies probe architectures, optimization and compute procedures, while the benchmark defines accent-cluster and Vaani-Hi data construction. Layer-selection experiments find that weighted fusion helps some tasks but offers no consistent advantage over last-layer representations.
- C Experimental Setup: Transcript probes use a two-layer bidirectional GRU with mean pooling over character-vocabulary transcripts, hidden dimension 256, input dimension 128, and dropout 0.1.The character vocabulary is built from all predicted transcripts.
- C Experimental Setup: Hidden-representation probes attention-pool the last-layer representation before a two-layer MLP whose output matches the downstream task.The MLP outputs a scalar for assessment, class logits for classification, or a unit [x y z] vector for geolocation.
- C Experimental Setup: Learning rates are 2e-4 for hidden-representation probes and 1e-3 for cascade probes, with validation-based early stopping after at least 10 epochs and patience 5.F1, Kendall Tau, or geolocation error selects the reported checkpoint according to task.
- C Experimental Setup: Probe runs require at most 15 minutes for TP and 3 hours for RP on a single 40GB GPU, with about 1k GPU hours spent on final evaluations.PRiSM also supports distributed inference across multiple GPUs and VLLM 8.
- D L1 to accent cluster mapping for EdAcc: The EdAcc corpus’s 41 L1 labels are consolidated into 13 accent clusters using phonological and typological similarities.Criteria include language family, vowel inventory, prosody, and shared phonetic transfer patterns to English.
- E Algorithm for Vaani-Hi: Vaani-Hi is a released Hindi-belt subset of the Vaani corpus constructed for the GEO-v task.The sampling focuses on 12 Hindi-belt states and randomly samples up to four districts per state.
- F Effect of Layer Selection on RP: Weighted-layer fusion improves some RP tasks, notably GEO-v and DYS-ua, but causes regressions elsewhere and does not consistently outperform last-layer RP.The best weighted-fusion results remain comparable to last-layer results, supporting the last layer as a reasonable default while allowing task-specific marginal gains.
G LALM Few-Shot Results … LID-fl: LID (FLEURS)
The section reports that LALM few-shot prompting improves pathological-speech tasks but produces inconsistent or negative effects elsewhere, while documenting evaluation prompts and label schemas for downstream tasks. It also provides confusion matrices for L1-eda bias analysis and defines 24-class language identification and geolocation outputs for LID-fl.
- G LALM Few-Shot Results: On LID-fl, Qwen3-Omni-Instruct degrades sharply as few-shot examples increase.The results indicate that the performance gap between LALMs and specialized PR models is not solely explained by prompting.
- I Prompts for LALMs: LALM prompts require expert phonetic analysis of segmental and prosodic cues, strict IPA transcription, and structured outputs without conversational filler.The IPA task requests exactly one IPA string sequence, without slashes or brackets.
- H Effect of Thinking Mode on L1-eda Classification: Figure 5 provides the full normalized confusion matrices for Gemini 2.5 Flash’s L1-eda classification across 13 accent clusters.Rows represent true labels and columns represent predictions.
- G LALM Few-Shot Results: Few-shot prompting improves pathological-speech tasks but produces mixed or negative changes elsewhere, with additional examples failing to yield monotonic improvement.For Gemini 2.5 Flash, only N-shot prompting was feasible because of resource constraints.