Source-linked AI summary
Deep Learning Techniques for Phoneme Recognition in Italian Children' s Speech
Nicola Barbaro, Cristina Gena, Francesco Petriglia, Andrea Meirone, Alessandro Mazzei, Arianna Viotti
TL;DR
Clinical language-sample analysis needs detailed IPA transcription, but transcription and coding are demanding and child-speech data are scarce. Broca uses adult-speech pretraining followed by fine-tuning on expert-annotated Italian child speech. The authors report that it reduces deletion and substitution errors while preserving word boundaries and valid Italian phoneme predictions.
Problem
Language-sample analysis requires time-consuming IPA transcription and specialized phonetic expertise, while large, high-quality child-speech datasets are scarce.
Source: §1 · pp. 2–3Method
Broca is pretrained on automatically transcribed adult Italian speech and then fine-tuned on child speech manually annotated by speech therapists.
Source: §4 · p. 12Results
Broca reduces deletion and substitution errors, preserves word boundaries, and restricts predictions to valid Italian phonemes.
Source: §5.5 · p. 21Takeaways & Limitations
The findings suggest that annotation reliability, phoneme-set adequacy, and domain-specific data matter more than training-data scale for accurate, practically usable clinical transcription.
Source: §6 · p. 22Broader testing on more diverse and disordered speech is needed, while dataset expansion is constrained by the manual effort required for expert transcription.
Source: §6 · p. 22
Abstract
from arXiv · showhide
Speech therapists often face difficulties diagnosing impairments due to the lack of efficient tools for transcribing speech into the International Phonetic Alphabet (IPA). This work addresses this challenge with Broca, a Conformer-based deep learning system pretrained on 8 days of adult speech and fine-tuned on a 165-minute dataset of Italian child speech collected through a range of standardized diagnostic tests for children aged 3.5-6.5. Broca was optimized to handle phonetic variability in children's speech, including tone, accent, and speech errors, and achieved a state-of-the-art weighted Phoneme Error Rate of 13.36% on Italian speech. Remarkably, this performance was obtained using less than three hours of child-specific data, underscoring the model's efficiency and robustness in low-resource clinical settings. This work demonstrates that accurate, vocabulary-independent speech-to-IPA transcription can be achieved with minimal data, paving the way for more accessible, data-efficient tools to support speech assessment and diagnosis.
1. Introduction
Clinical language-sample analysis can reveal children’s phonetic and broader language abilities, but IPA transcription and coding are time-consuming and require specialized expertise. The paper presents an Italian child-speech transcription platform and frames research questions about preserving speech errors, supporting clinical workflows, and the value of carefully curated data.
1. Introduction
- IPA transcription and language-sample analysis provide clinically relevant phonetic and linguistic information, but their time and expertise demands limit routine use.The passages identify transcription, coding, and analysis as labor-intensive barriers, with IPA adding further cognitive and operational demands.Source: §1 · p. 2
- Most ASR targets standard orthography rather than clinical IPA transcription that preserves speech errors, and existing phonetic models largely focus on other languages.The introduction identifies a lack of open-source ASR specifically tailored to Italian.Source: §1 · pp. 2–3
- Children’s acoustic, linguistic, and behavioral variability, together with scarce child-speech data and no native Italian child-speech dataset, complicates recognition.The passages describe higher pitch and formant frequencies, developing articulation, and spontaneous or less structured speech alongside dataset scarcity.Source: §1 · pp. 3–4
- Talkidz combines a semi-structured language-sample assessment with Broca, a Conformer-based pipeline that retains children’s speech errors and supplies structured phonological information for clinical use.The platform also offers dashboards with validated clinical metrics and statistics.Source: §1 · p. 4
1.1. Research objectives
- The study asks whether AI can transcribe Italian children’s speech while preserving errors and how transcription tools may support therapists’ workloads and clinical practice.Source: §1.1 · p. 4
- A further research question tests whether a small, curated, manually transcribed Italian phoneme-inventory dataset can outperform models trained on larger automatically transcribed corpora.Source: §1.1 · p. 5
2. Related Works
Prior work spans digital therapy platforms and phonetic ASR, but the reviewed approaches often emphasize autonomous practice, automatic transcription, or canonical pronunciation. This paper distinguishes its approach through clinician-supervised assessment and expert annotations designed to retain child speech errors.
2.1. Speech Therapy Technologies
- Therapy platforms range from autonomous feedback to collaborative tools, while Talkidz supports structured phonetic assessment under professional supervision.The stated design lets therapists observe, interact, and make immediate clinical judgments during data collection.Source: §2.1 · pp. 5–6
2.2. Automatic Phonetic Transcription
- Taguchi et al.’s multilingual phonetic ASR relies on G2P-converted training and test data and omits Italian, while Yusuyin et al. [68] use weakly supervised G2P transcriptions.The paper contrasts Broca’s child-speech fine-tuning with expert-annotated phonetic transcriptions.Source: §2.2 · p. 6
2.2.1. Child Speech Recognition
- Child-speech research includes English-focused evaluations [24] and French phoneme recognition, whereas this study uses expert phoneme-level annotation regardless of correctness for children aged 3.5–6.5.Medin et al. report PERs as low as 26.1% and annotate speech errors manually, while their participants perform reading tasks.Source: §2.2.1 · pp. 6–7
- IPA-CHILDES [31] and ZIPA [69] provide broad phonemic resources, but their automatic conversion can normalize atypical productions; this work uses expert annotations with word boundaries and an Italian phoneme inventory.Source: §2.2.1 · pp. 7–8
3. Dataset
The dataset combines Italian adult speech for phonetic pretraining with child speech collected through Talkidz and standardized assessments for fine-tuning. It prioritizes carefully curated, manually validated, word-level IPA labels and varied child-speech samples.
3.1. Pretraining Data (Adult Speech)
- Adult Italian speech from MLS was automatically converted to a simplified IPA subset with an in-house rule-based converter, then manually checked for systematic errors.The approach uses Phonitalia for known words and fallback rules for exceptions; it omits diacritics and suprasegmental features.Source: §3.1 · p. 8
3.2. Fine-Tuning Data (Child Speech)
- Child fine-tuning data came from the Talkidz Evaluation and two standardized neuropsychological tests, with additional tasks intended to reduce vocabulary overfitting and support generalization to spontaneous speech.Recordings were made in acoustically controlled environments using high-fidelity microphones.Source: §3.2 · p. 8
3.2.1. Part I: The Talkidz Evaluation (Drawing Description)
- The Talkidz Evaluation uses 40 illustrations to elicit child descriptions, with familiar concrete nouns selected for phonetic coverage and varied phonetic complexity.Children receive an open-ended prompt; the image-description task is presented as a compromise between examiner control and children’s verbal competence. Administration averaged 20 minutes per child, with 140 utterances on average.Source: §3.2.1 · pp. 8–9
3.2.2. Part II: Speech Evaluation Test for children aged 4 to 12 (Denomination and Articulation Task)
- The BVL 4-12 contributed naming and articulation, nonword repetition, and sentence-repetition tasks to assess phonological and morphosyntactic abilities.Nonword repetition limits the influence of prior lexical knowledge while testing phonological memory and phoneme sequencing.Source: §3.2.2 · pp. 9–10
3.2.3. Part III: Neuropsychological Lexical Test (Semantic Fluency)
- The Neuropsychological Lexical Test asks children to name words from a semantic category within a time limit, assessing lexical retrieval and semantic organization.The task also targets executive processes and lexical search, including focus, rule-following, and shifting between subcategories.Source: §3.2.3 · p. 10
3.2.4. On Dataset Scale and Annotation Quality
- 15 children contributed 165 minutes of word-level IPA annotations, manually checked by speech therapists and restricted to Italian phonemes.The sample was selected from 21 participants; the paper argues that high-quality phonetic annotation can matter more than raw dataset size.Source: §3.2.4 · p. 11
- The child dataset’s word-level labels support both phoneme generation and word-boundary learning, which the Talkidz Platform needs to compute evaluation statistics.The authors report that models trained on thousands of hours of automatically transcribed adult speech had higher PER than Broca, even after manual post-processing.Source: §3.2.4 · p. 11
4. Methods
Broca automates speech-to-IPA transcription while preserving children’s speech errors, using adult Italian speech pretraining followed by therapist-annotated child-speech fine-tuning. Hyperparameter experiments identify a low learning rate as effective when nearly all model layers are unfrozen.
4. Methods
- Broca is designed to transcribe uploaded recordings into phonetic notation while preserving speech errors, removing the need for further manual transcription.The system is intended for the Talkidz Platform’s analysis phase.Source: §4 · p. 11
4.1. Model Training Procedure
- The training pipeline uses a fully connected output layer trained with CTC loss, after adult-speech phonetic pretraining and therapist-annotated child-speech fine-tuning.The model is based on W2v-BERT 2.0, a 24-layer Conformer encoder with approximately 600 million parameters.Source: §4 · p. 12
4.1.1. Pretraining Phase on Adult Speech
- The adult Italian MLS pretraining phase kept all model layers unfrozen and used automatically transcribed data, with additional ASR checks to flag transcription anomalies.Approximately three hours of audio were manually reviewed and corrected; training used eight days of audio.Source: §4.1.1 · p. 12
4.1.2. Fine-Tuning on Child Speech
- Fine-tuning retained only child speech from assessment recordings, excluding therapist speech to ensure evaluation measured generalization to children rather than adult speech.The selected child data were manually annotated by speech therapists.Source: §4 · p. 12, §4.1.2 · p. 13
4.1.3. Hyperparameter Tuning
- The hyperparameter search varied learning rates and the number of frozen Conformer layers over 20 epochs; the smallest tested rate, 5e-7, did not improve on the pretrained model.The search included learning rates of 1e-5, 5e-6, and 5e-7, and configurations freezing 2 to 22 layers.Source: §4.1.3 · p. 13
- A 5e-6 learning rate produced the best results only when almost all layers were unfrozen, while that strategy risked degrading adult-speech accuracy.The authors note that a lower rate may allow gradual adaptation when fine-tuning the entire model.Source: §4.1.3 · p. 13
5. Results
Fine-tuning Broca on manually transcribed child speech yields the strongest phoneme-recognition results. Qualitative and statistical analyses further examine the clinical usability and comparative performance of the systems.
5.1. Metrics: The Phoneme Error Rate
- PER counts phoneme substitutions, deletions, and insertions relative to reference phoneme count; wPER weights substitutions by phonetic similarity.The weighted measure assigns greater cost to substitutions between less similar phonemes.Source: §5.1 · pp. 13–14
5.2. Model Evaluation and Comparisons
- 27.76% PER and 23.08% wPER: pretrained Broca improves on Whisper with G2P, but deletion errors remain high at 17.74%.XLSR-53, LV60, and MultIPA have PER values from 49.72% to 57.37% and deletion rates above 31%.Source: §5.2 · p. 15
- 16.10% PER and 13.36% wPER: fine-tuned Broca leads all metrics, cutting deletions to 8.60% and substitutions to 4.42%.Insertion errors rise slightly to 3.09%, from 2.58% for pretrained Broca.Source: §5.2 · p. 15, Table 2 · p. 16
5.3. Qualitative Analysis of Transcription Accuracy
- Broca closely matches speech-language pathologist references, preserves word boundaries, and produces clinically usable Italian-IPA transcriptions with minimal post-editing.Experts note minor gemination errors; adult-oriented systems show unsegmented or impractical outputs.Source: Table 3 · p. 16, §5.3 · p. 17
5.4. Statistical Evaluation
- Q=523.82 (p<5.75 × 10^-111) rejects equal performance; fine-tuned Broca significantly outperforms every other system with complete separation of scores (r=∞).The Friedman test across six models yielded Kendall’s W=0.126.Source: §5.4 · pp. 18–19
5.5. Discussion
- Broca’s child-speech fine-tuning diverges from inherited G2P errors, while unreliable word boundaries undermine the clinical applicability of other systems.The discussion identifies word-level patterns as important for diagnosing and tracking speech impairments.Source: §5.5 · pp. 20–21
6. Conclusions
Broca demonstrates feasible child-speech phoneme transcription with limited child-specific data, while the conclusions identify constraints on generalization, annotation, pretraining labels, and deployment. The authors argue that expert-curated data and clinical involvement matter more than dataset size alone.
6. Conclusions
- 165 minutes of child speech fine-tuning significantly improved Broca’s performance, and the authors report accurate automated transcription despite young speakers’ acoustic variability.They attribute performance to standardized vocabulary, consistent Italian IPA labeling, and speech therapists’ involvement in annotation and system design.Source: §6 · p. 21
- A small, manually transcribed Italian phoneme-inventory dataset can outperform models trained on larger automatically transcribed corpora, according to the authors.They argue that phoneme-set adequacy, reliable word-level annotations, and domain-specific training data shape accuracy and clinical usability.Source: §6 · p. 22
- Broca’s 600M parameters require substantial resources for real-time inference, limiting clinical use without dedicated infrastructure or cloud solutions.Source: §6 · p. 22
- The study’s child-speech dataset is limited, and broader testing is needed across voices, accents, speech patterns, and children with impairments.The authors describe the current dataset as one hour from a small number of participants and say the study primarily involved typically developing children.Source: §6 · pp. 22–23
- Future work could improve pretraining phoneme labels by refining or replacing the rule-based grapheme-to-phoneme converter, which misses some Italian exceptions.The authors also propose multilingual training with languages sharing phonetic structures with Italian as a possible way to improve generalization in low-resource child-speech settings.Source: §6 · pp. 22–23
1. OpenAI Whisper: https://openai.com/index/whisper
The passages record consent and ethics approval for child-participant data, and point readers to a reference about Mel-Spectrograms in ASR.
1. OpenAI Whisper: https://openai.com/index/whisper
- Child recordings and linguistic data were collected with written parental or guardian consent for research and dissemination, under University of Turin ethics approval 0332723/2024.Source: §1 · p. 23
- A cited reference provides more information about the Mel-Spectrogram and its role in ASR pipelines.Source: §1 · p. 23
Credit authorship contribution statement
The authors’ contributions span conceptualization, methodology, data curation, validation, supervision, and writing, with roles varying by author.
Credit authorship contribution statement
- The contribution statement assigns Nicola Barbaro roles across conceptualization, methodology, software, data curation, analysis, resources, validation, supervision, and writing.It also lists distinct contributions for Cristina Gena, Francesco Petriglia, Andrea Meirone, Alessandro Mazzei, and Arianna Viotti.Source: Credit authorship contribution statement · p. 23