Source-linked AI summary
PEDBench: Can Speech Models Detect Phonetic Errors?
Zhonghao Shi, Zackary Rackauckas, Jasmine Yip, Savannah Halpern, Simon Campos, Lauren Brace, Emma Yang, Tiantian Feng, Xuan Shi, Shrikanth Narayanan, Teresa Satterfield, Ying Xu
TL;DR
Reliable speech-model detection of fine-grained phonetic errors remains unclear because high-quality evaluation data are lacking. PEDBench evaluates models across verified child, acted, and synthetic error-speech datasets and finds that models still fail to detect errors reliably, including in controlled settings. Synthetic-data fine-tuning improves Qwen2.5-Omni-7B’s word-level PED, but robust detection is not yet established.
Problem
Whether state-of-the-art speech models can reliably detect phonetic errors remains unclear, amid inconsistent annotations, limited model evaluation, and metrics that may not characterize detection adequately.
Source: §1 · pp. 1–2Method
PEDBench introduces a benchmark with verified real-world child, acted, and synthetic error-speech datasets and evaluates speech models across them.
Source: §1 · p. 2, §3.1.2 · p. 3Results
Takeaways & Limitations
PEDBench provides a basis for evaluating and training PED models, while synthetic speech has potential to scale training data, especially for rare errors difficult to collect in the real world.
Source: §7 · p. 9Synthetic-data fine-tuning does not yet establish robust error detection, and the pipeline requires human review to exclude TTS samples that do not match intended errors.
Source: §8 · p. 9
Abstract
from arXiv · showhide
Phonetic error detection (PED) aims to identify phoneme-level substitutions, deletions, and insertions and is essential for educational and clinical speech assessment and intervention. Despite recent advances in speech models for phoneme recognition, their ability to reliably detect fine-grained phonetic errors remains unclear because of a lack of high-quality evaluation data. To address this gap, we introduce PEDBench, the first PED benchmark spanning a continuum from natural to controlled speech. It comprises real-world recordings of spoken errors made by children (N=334), scripted error productions by human actors (N=1,188), and synthetic error productions generated using phone-conditioned text-to-speech (N=1,083). All phonetic annotations were transcribed and verified by two phonetically trained annotators. Our evaluation of 13 state-of-the-art speech models shows that they fail to reliably detect phonetic errors, even under controlled conditions with acted and synthetic speech, where acoustic variability is reduced. The models also exhibit low precision, frequently misclassifying correctly pronounced phonemes as errors. Our speech-language models, fine-tuned on synthetic data, outperform commercial and open-source baselines on word-level PED, although a sim-to-real domain gap remains. We plan to publicly release our code and recipes, along with our acted and synthetic datasets, to support the development of more accurate PED systems.
1 Introduction
Reliable phonetic error detection could support early screening and educational and clinical interventions, but its real-world reliability remains unclear amid inconsistent annotations, limited model evaluation, and inadequate metrics. PEDBench addresses these gaps with verified datasets, broad model evaluation, and synthetic-data fine-tuning experiments.
1 Introduction
- Reliable speech-model detection could expand access to early screening and supplement educational and clinical interventions.The motivation notes limited access to accurate early assessment and affordable, effective intervention.Source: §1 · p. 1
- PED research lacks consistent ground-truth annotations, systematic evaluation of general-purpose speech models, and metrics that adequately characterize error detection.Many specialized PED models also remain closed-source.Source: §1 · p. 2
- PEDBench introduces natural child, acted, and synthetic error-speech datasets with phoneme transcriptions independently produced or verified by two phonetically trained annotators.The datasets span naturalistic to controlled speech, and the evaluation covers 8 specialized phoneme-recognition models, one specialized PED model, and four speech-language models.Source: §1 · p. 2
- Speech models still fail to detect simple phonetic errors from word-level speech under controlled, noise-free conditions.The benchmark also evaluates natural child speech and reports fine-tuning Qwen2.5-Omni-7B on synthetic speech improves word-level PED over evaluated baselines.Source: §1 · p. 2
2 Related Work
Prior PED work is fragmented and lacks a standardized benchmark, while recent specialized phoneme recognizers and speech-language models have not been systematically assessed for PED. Several specialized systems also remain closed-source.
2.1 Phonetic Error Detection
- PED research lacks a standardized benchmark, and recent specialized PED models remain closed-source.Prior work commonly frames PED as phoneme recognition followed by alignment with a canonical sequence; newer approaches include phoneme similarity and K-WFST.Source: §2.1 · p. 2
2.2 State-of-the-art Phoneme Speech Models
- State-of-the-art specialized phoneme-recognition models have not been systematically evaluated for PED, and speech-language models’ PED capabilities remain underexplored.Prior reports found commercial and open-source SLMs competitive on public phoneme-recognition benchmarks, though behind specialized PR models.Source: §2.2 · pp. 2–3
3 PEDBench
PEDBench evaluates speech models on public and newly collected error-speech datasets spanning naturalistic and controlled conditions. It standardizes phoneme sequences, aligns them, and measures both error detection and diagnosis accuracy.
3 PEDBench
- PEDBench is designed to test whether state-of-the-art speech models detect phonetic errors in controlled and real-world settings.It includes the public L2-ARCTIC-Perceived (N=3599) and CMU-Kids (N=5180) benchmarks.Source: §3 · p. 3, §3.1.1 · p. 3
3.1.2 Our Novel PED Datasets
- The work introduces three phonetic-error datasets: naturally occurring child speech, scripted acted speech, and synthetic speech; child recordings were retained only when two annotators agreed on error labels and realized phoneme sequences.The synthetic evaluation set comprised 1,083 utterances in 361 matched word–sentence–passage sets; the acted-speech evaluation set comprised 1,188 utterances.Source: §3.1.2 · pp. 3–4
3.2 Evaluation Pipeline
- The pipeline derives canonical phonemes from transcripts, uses human annotations for realized phonemes, and obtains predicted phonemes from each speech model.It standardizes sequences to the CMU 39-phoneme inventory and aligns them on a common grid.Source: §3.2 · p. 4
3.3 Evaluation Metrics
- PEDBench reports phoneme error rate and phoneme-feature error rates alongside PED-specific detection and diagnosis metrics.Diagnosis-weighted precision, recall, and F1 credit a true rejection only when the predicted phone matches the realized phone, unlike standard MDD precision and recall.Source: §3.3 · pp. 4–5
4 Benchmarked Models
PEDBench evaluates specialized phoneme recognizers, a commercial PED system, and proprietary and open-weight speech-language models. The SLMs are also adapted for phoneme recognition through supervised fine-tuning on TIMIT.
4.1 Specialized Phoneme Recognition Models
- The benchmark includes eight specialized phoneme-recognition models from six model families, including wav2vec 2.0, Zipformer-based, and recently released systems.The evaluated models include MultiIPA, W2V2P-LV60, W2V2P-XLSR53, ZIPA-CTC, ZIPA-CTC-NS, HuPER, PhoneticXeus, and BabAR.Source: §4.1 · p. 5
4.2 Specialized PED Model
- Azure Pronunciation Assessment is evaluated as a specialized model configurable for word- and phoneme-level pronunciation assessment and phone recognition.Source: §4.2 · p. 5
4.3 Speech-Language Models
- The evaluation includes two proprietary SLMs, Gemini-3.8-Flash and GPT-Audio-1.5, and two open-weight SLMs, Qwen2.5-Omni-7B and Audio Flamingo 3.Qwen2.5-Omni-7B and Audio Flamingo 3 were fine-tuned on TIMIT’s training split and prompted to produce phoneme sequences using the CMU 39-phoneme inventory.Source: §4.3 · p. 5
5 Fine-Tuning Experiments with Synthetic Data
The experiments ask whether synthetic speech can address the difficulty of collecting large-scale real-world PED training data, including rare errors. They fine-tune three speech models on synthetic datasets designed to cover errors in acted and child-speech evaluation.
5 Fine-Tuning Experiments with Synthetic Data
- Synthetic training data is designed to cover common and rare phonetic errors that are difficult to collect at scale in real speech.The paper frames synthetic speech as a way to scale training data while covering both common and rare error types.Source: §5 · p. 5
- Three models—W2V2P-XLSR53, Qwen2.5-Omni-7B, and Audio Flamingo 3—were fine-tuned and evaluated on new child-speech and acted-speech datasets.Each model was trained separately on synthetic word data covering acted-speech errors and synthetic combined data that additionally covered child-speech errors.Source: §5 · p. 5
- TIMIT was added to the training mixtures for Qwen2.5-Omni-7B and Audio Flamingo 3.Source: §5 · p. 5
6 Results
Across real-world, acted, and synthetic evaluation settings, speech models struggle to detect phonetic errors reliably. Fine-tuning with synthetic data improves results on the paper’s child and acted datasets, but performance remains limited and error rates worsen with longer audio.
6.1 Can SOTA speech models detect phonetic errors from L2-Arctic and CMU-Kids?
- 34.32 F1 on L2-ARCTIC and 54.55 F1 on CMU-Kids were the best reported real-world results, yet models still failed to detect errors reliably.The best scores were achieved by TIMIT-fine-tuned Qwen2.5-Omni-7B on L2-ARCTIC and ZIPA-CTC-NS on CMU-Kids; human reannotations also had low agreement with original labels, at 42.25 F1 and 66.22 F1, respectively.Source: §6.1 · pp. 5–6, Table 1 · p. 6
6.2 Can SOTA speech models detect phonetic errors in our novel PED datasets?
- 59.51 F1 for BabAR, 49.96 F1 for Azure, and 58.26 F1 for Gemini were the best model results on the child-speech dataset, while acted-speech leaders reached 74.79, 75.98, and 76.17 F1, respectively.Even the more controlled acted setting did not yield reliable PED, and low precision suggested frequent false rejections.Source: §6.2 · p. 6, Table 2 · p. 7
6.3 Can synthetic data help fine-tune speech models for better PED?
- 63.82 F1 for fine-tuned AF3 and 63.47 F1 for fine-tuned Qwen on child speech, and 90.06 F1 for fine-tuned Qwen and 88.83 F1 for fine-tuned AF3 on acted speech, exceeded the cited baseline models.Child-speech comparisons were against Gemini (58.26), Azure (49.96 F1), and BabAR (59.51 F1); acted-speech comparisons were against GPT-Audio (76.17 F1), Azure (75.98 F1), and ZIPA-CTC (74.79 F1). Child-speech performance remained unsatisfactory, with the authors noting a possible adult-to-child distribution shift.Source: Table 2 · p. 7, §6.3 · pp. 7–8
6.2 Can SOTA speech models detect phonetic errors in our novel PED datasets?
- 74.32 F1 at word level, 39.27 F1 at sentence level, and 14.22 F1 at passage level show synthetic-speech PED degrades as audio length increases.Recall remained stable across levels, suggesting that lower precision and more false rejections primarily drove the degradation.Source: §6.2 · p. 7, Table 3 · p. 8
6.4 Error Analysis
- Models confuse acoustically similar sounds, including /T/–/f/, /S/–/tS/, /O/–/A/, and /eI/–/I/.Among four representative models, Qwen fine-tuned on TIMIT and SW had the lowest substitution, deletion, and insertion rates on acted speech; BabAR performed substantially worse on acted adult speech but remained competitive on child speech.Source: §6.4 · p. 8, Figure 3 · p. 8
7 Conclusion
PEDBench is introduced as a standardized benchmark for evaluating state-of-the-art speech models on phonetic error detection. The evaluation finds persistent reliability failures across real-world and controlled speech, while synthetic-data fine-tuning shows potential to scale training data.
7 Conclusion
- PEDBench is presented as the first standardized benchmark for evaluating state-of-the-art speech models on phonetic error detection.Source: §7 · p. 8
- State-of-the-art models still fail to detect phonetic errors reliably across word- and sentence-level speech in real-world and highly controlled settings.Source: §7 · p. 9
- Synthetic data may help scale PED training, especially for rare errors that are difficult to collect in real-world speech.The conclusion frames PEDBench as groundwork for more accurate evaluation and training toward reliable real-world applications.Source: §7 · p. 9
8 Limitations
PEDBench’s evaluation is limited in coverage and scope: its new datasets do not comprehensively represent real-world PED, and its synthetic-data fine-tuning results do not establish robust error detection. Annotation ambiguity, audio-only inputs, English-only data, and a non-exhaustive model set further bound the evaluation.
Evaluation Goals
- The three new datasets do not yet comprehensively cover real-world phonetic errors or speakers with diverse linguistic backgrounds and speech and language abilities.The authors say broader real-world data are needed for more comprehensive validation before deployment.Source: §8 · p. 9
Fine-Tuning Claims
- Synthetic-data fine-tuning shows potential to scale training data, but does not establish robust phonetic error detection.The TTS pipeline depends on reliably producing intended errors and requires human review to exclude inaccurate samples.Source: §8 · p. 9
Phoneme Transcription
- Despite verification by two phonetically trained annotators, some ambiguity and subjectivity may remain in phoneme transcriptions.Acted and synthetic speech reduce this ambiguity but may not eliminate it.Source: §8 · p. 9
Modality
- The evaluated models use audio only; visual articulatory information has not been explored and may help distinguish acoustically similar phonemes such as /T/ and /f/.The authors identify audio-visual input as a direction for future work.Source: §8 · p. 9
Language
- PEDBench’s three new datasets are English-only and use CMU-39, limiting applicability to multilingual settings.Source: §8 · p. 9
Benchmarked Models
- The benchmark covers prominent model categories but does not evaluate every available model.Source: §8 · p. 9
9 Ethical Considerations
The authors describe restrictions on data use and release alongside plans to release code and eligible model checkpoints.
Data
- Consent permits evaluating third-party commercial ASR models on the newly collected data, but does not permit public release of child speech recordings.The child-speech dataset is to remain a private held-out evaluation set; only acted and synthetic datasets can be released publicly.Source: §9 · p. 9
Models and code
- The authors plan to release fine-tuned model checkpoints subject to the underlying models’ licenses and to release code under a compatible license.The code builds on PRiSM, which follows the Lightning-Hydra-Template.Source: §9 · p. 9
10 Generative AI Disclaimer
The appendices document the paper’s AI-assistance disclosure, public datasets, phonetic ambiguity, synthetic error pairs, evaluation metrics and alignment, phone mapping, and model fine-tuning procedures.
10 Generative AI Disclaimer
- AI coding agents assisted development and AI writing assistants helped proofread and improve the manuscript; authors verified the resulting code and reviewed AI-assisted text.Source: §10 · pp. 9–10
A.1 Public Datasets
- L2-ARCTIC-Perceived contains 3,599 utterances from 24 non-native adult English speakers, while CMU Kids contains 5,180 prompted English utterances from 76 children aged 6–11.The paper uses processed L2-ARCTIC-Perceived data and the LDC-distributed CMU Kids version.Source: §A.1 · p. 12
A.2 Spectrum of Ambiguity in Phoneme Production
- Natural speech, especially child speech, can place realizations near phonetic boundaries such as /s/–/T/, making consistent ground-truth transcription difficult.Source: §A.2 · p. 12
B IPA Substitution Pairs
- Table 4 lists the canonical-to-substituted IPA phone pairs used to generate synthetic mispronunciations.Source: B IPA Substitution Pairs · p. 12, Table 4 · p. 12
C Evaluation Metrics
- PER and PFER summarize phone-sequence errors, while PED metrics separately assess detection and diagnosis because errors are sparse.PFER weights substitutions by articulatory-feature distance, penalizing confusions between similar phones less than dissimilar ones.Source: C Evaluation Metrics · p. 12
D MDD Alignment and Metrics
- The evaluation fixes mispronunciation events by aligning canonical and uttered sequences before aligning model predictions, then labels each row for detection and diagnosis outcomes.Diagnosis-weighted measures credit a detected error only when the model also identifies the phone the speaker produced.Source: D MDD Alignment and Metrics · p. 13
E Phone Normalization and CMU-39 Projection
- Model outputs are normalized and projected to a shared CMU-39 inventory, with some context-dependent symbols resolved using the aligned canonical phone.The paper notes that removing spaces can merge adjacent symbols into one phone.Source: E Phone Normalization and CMU-39… · pp. 13–14
F Fine-Tuning Details; F.1 Speech-Language Models; F.2 Scripted Training
- The fine-tuning experiments use LoRA for Qwen2.5-Omni-7B and Audio Flamingo 3, and full fine-tuning for W2V2P-XLSR53; the SLM audio encoders remain frozen.For scripted runs, the prompt includes the intended word and canonical phone sequence, while the perceived transcription is the training target.Source: F Fine-Tuning Details · p. 14, §F.1 · p. 14, §F.2 · p. 14
F.3 Training Data
The training setup combines TIMIT and synthetic speech in different model-specific mixes, with evaluation restrictions tied to training and checkpoint selection. Prompts and output parsing standardize model inputs and predictions for comparison.
F.3 Training Data
- TIMIT uses 3,696 training, 400 development, and 192 core-test utterances; SW has 396 recordings, while SC has 550.SC adds 154 synthetic recordings of authentic-child-speech error patterns to SW; all SW and SC recordings go to training.Source: §F.3 · p. 14
- Acted and Authentic results for TIMIT+SW and TIMIT+SC measure transfer on seen words, not generalization to new items.TIMIT+SC models are not evaluated on Acted because those utterances select checkpoints; synthetic recordings used in training are excluded from Synthetic evaluation.Source: §F.3 · pp. 14–15
F.4 W2V2P-XLSR53 CTC Fine-Tuning
- W2V2P-XLSR53 is fine-tuned with CTC while keeping its IPA vocabulary and output layer; its convolutional feature encoder is frozen.SW and SC models train on 396 and 550 recordings, respectively, with training steps selected by grouped five-fold cross-validation.Source: §F.4 · p. 15
F.5 Prompts
- Qwen2.5-Omni-7B, Audio Flamingo 3, Gemini, and GPT-Audio use the same system and user prompts, and fine-tuning uses the evaluation prompt.All main-paper results are unscripted unless marked otherwise.Source: §F.5 · p. 15, Figure 5 · p. 15
F.6 Output Parsing
- For three speech-language models, outputs are parsed as JSON or accepted as plain phone strings; unparseable responses become empty predictions.Unmappable phones are dropped after projection to the common phone inventory.Source: §F.6 · p. 16
G.1 Scripted vs Unscripted Results
Results vary with scripting, audio length, and model choice. Phone-level analyses show distinct error patterns across Authentic Child Speech and Acted Speech, including a substantial BabAR domain mismatch on acted recordings.
G.1 Scripted vs Unscripted Results
- -11.82 for Azure contrasted with gains of +9.38 for Gemini, +6.19 for GPT-Audio, and +5.14 for Qwen on authentic data with scripting.Scripted Qwen fine-tuned on TIMIT+SC scored 68.61, above scripted Gemini-3.8-Flash at 67.64.Source: §G.1 · p. 16
- -14.37 for Azure, -24.06 for GPT-Audio-1.5, and -10.42 for fine-tuned Qwen were the acted-data performance changes with scripting.Unscripted Qwen fine-tuned on TIMIT+SW scored 90.06, above scripted Gemini-3.8-Flash at 82.19.Source: §G.1 · p. 17
G.2 Analysis Focused on Target Words
- Specialized PR models tended to improve with longer audio, while Gemini-3.8-Flash and AF3 degraded, particularly Gemini.Specialized PED models, GPT-Audio-1.5, and Qwen models maintained relatively stable performance across audio lengths.Source: §G.2 · p. 17
H Phone-Level Error Analysis
- The phone-level confusion matrices compare perceived phones in rows with predicted phones in columns, with row-normalized counts and separate deletion and insertion columns.Four representative models are compared on Authentic Child Speech and Acted Speech.Source: H Phone-Level Error Analysis · p. 17
H.1 Authentic Child Speech
- 77.9% of 1,502 perceived phones were correct for Qwen+TIMIT+SC on Authentic Child Speech; Gemini reached 74.8%, BabAR 71.4%, and W2V2P-XLSR53 58.9%.All four models often deleted ô and confused 2 with nearby vowels; Gemini and W2V2P-XLSR53 had 58 and 94 insertions per 1,000 phones.Source: §H.1 · p. 17
H.2 Acted Speech
- 96.7% of 5,088 phones were correct for Qwen+TIMIT+SW on Acted, compared with 89.9% for Gemini and 87.0% for W2V2P-XLSR53.BabAR got 28.8% right, deleting 64.3% of phones; it returned no output for 501 of 1,188 adult utterances, consistent with domain mismatch rather than poor phone recognition.Source: §H.2 · p. 18