Source-linked AI summary
Transfer Learning from Adult to Children for Speech Recognition: Evaluation, Analysis and Recommendations
Prashanth Gurunath Shivakumar, Panayiotis Georgiou
TL;DR
Children’s ASR is less accurate because children’s speech has substantial acoustic and linguistic variability, compounded by limited children’s training data. The paper evaluates adult-to-child DNN transfer learning across adaptation strategies, ages, and data amounts, finding that targeted transfer and standard adaptation substantially improve recognition while exposing remaining challenges.
Problem
Children’s ASR must handle high acoustic and pronunciation variability, yet prior analysis has focused mainly on GMM-HMM systems and children’s data are limited.
Method
The study evaluates adult-to-child DNN transfer learning for children’s LVCSR, comparing adaptation techniques, layer configurations, data amounts, and age-dependent transformations.
Results
SAT reduces WER to 21.31%, a 14.66% absolute improvement over raw features, while acoustic-variability transfer reaches 24.26% versus 26.97% for pronunciation-variability transfer and 39.32% for the adult baseline.
Takeaways & Limitations
Transfer learning offers distinct layer-targeted strategies for children’s acoustic and pronunciation variability, alongside recommendations across data availability and children’s ages.
Abstract
from arXiv · showhide
Children speech recognition is challenging mainly due to the inherent high variability in children's physical and articulatory characteristics and expressions. This variability manifests in both acoustic constructs and linguistic usage due to the rapidly changing developmental stage in children's life. Part of the challenge is due to the lack of large amounts of available children speech data for efficient modeling. This work attempts to address the key challenges using transfer learning from adult's models to children's models in a Deep Neural Network (DNN) framework for children's Automatic Speech Recognition (ASR) task evaluating on multiple children's speech corpora with a large vocabulary. The paper presents a systematic and an extensive analysis of the proposed transfer learning technique considering the key factors affecting children's speech recognition from prior literature. Evaluations are presented on (i) comparisons of earlier GMM-HMM and the newer DNN Models, (ii) effectiveness of standard adaptation techniques versus transfer learning, (iii) various adaptation configurations in tackling the variabilities present in children speech, in terms of (a) acoustic spectral variability, and (b) pronunciation variability and linguistic constraints. Our Analysis spans over (i) number of DNN model parameters (for adaptation), (ii) amount of adaptation data, (iii) ages of children, (iv) age dependent-independent adaptation. Finally, we provide Recommendations on (i) the favorable strategies over various aforementioned - analyzed parameters, and (ii) potential future research directions and relevant challenges/problems persisting in DNN based ASR for children's speech.
I. INTRODUCTION
Children’s ASR remains less accurate than adults’ because children’s speech varies acoustically and linguistically across development, while available DNN-focused analysis remains limited. This work evaluates transfer learning and adaptation strategies across these sources of variability and age-related conditions.
- Sources of variability: Children’s speech varies acoustically through spectral and temporal differences and linguistically through pronunciation variation and partial linguistic knowledge.
- Sources of variability: Acoustic variability reflects shifted formants, within-subject spectral variation, and inter-speaker differences linked to developmental vocal-tract changes.
- Motivation: Children’s ASR is 2 to 5 times worse in word error rate than adults’ ASR.
- Prior approaches: Prior work used VTLN, MLLR, MAP, and SAT to reduce acoustic variability, while pronunciation work used customized dictionaries and data-driven variation modeling.
- Study scope: The study compares GMM-HMM with DNN models, evaluates transfer learning against standard adaptation, and targets acoustic and pronunciation variability in children’s LVCSR.
- Study scope: Its analysis varies DNN parameters, adaptation-data quantity, children’s ages, and age-dependent versus age-independent transformations to derive practical recommendations.
II. PROPOSED TRANSFER LEARNING TECHNIQUE
The proposed transfer-learning framework uses adult ASR knowledge to initialize children’s models while adapting layers according to whether variability is acoustic or pronunciation-related.
- Transfer learning seeds a new task’s model with knowledge learned from a related task.
- The method maps acoustic variability to lower network layers and pronunciation variability to higher layers.
- Lower layers capture signal-specific characteristics, whereas higher layers capture task-related information such as phoneme classes.
A. Accounting for Acoustic Variability
The technique adapts different parts of an adult DNN to address acoustic and pronunciation variability in children’s speech, optionally updating both ends while retaining intermediate layers.
- A. Accounting for Acoustic Variability: Acoustic adaptation retains all adult hidden layers except the bottom-most layer and retrains that layer on children’s speech.
- A. Accounting for Acoustic Variability: This input-layer adaptation preserves higher-level adult mappings while estimating a children-specific feature subspace.
- A. Accounting for Acoustic Variability: MFCC features are augmented with i-vectors to provide speaker- and age-specific information for adapting the bottom layers.
- B. Accounting for Pronunciation Variability: Pronunciation adaptation updates higher layers while leaving lower-order layers unchanged.
- Joint adaptation: Joint adaptation updates the top-most and bottom-most layers while keeping middle hidden layers fixed, either simultaneously or alternately.
III. DATABASES
The study combines five children’s speech databases for training and uses CID for testing, alongside the 206-hour TED-LIUM adult corpus for transfer learning. Baseline systems include GMM-HMM and hybrid DNN-HMM models with adapted acoustic features.
- Corpora: Five children’s speech databases are used, with CID reserved for testing and the remaining corpora used for training.The multi-corpus setup is intended to make evaluation more challenging and relevant to real-world scenarios.
- Corpora: The adult training data come from TED-LIUM, comprising 206 hours of speech from 774 speakers.
- Acoustic-model baselines: The GMM-HMM baseline uses 13-dimensional MFCCs with first- and second-order derivatives and applies LDA, MLLR, and fMLLR adaptation.Features use 25 ms windows with a 10 ms shift.
- Acoustic-model baselines: The hybrid DNN-HMM uses a time-delay neural network with 7 hidden layers of 3500 units and a 3976-unit Softmax output.The model consumes contextual high-resolution MFCCs concatenated with i-vectors.
B. Baseline System
The baseline comparison evaluates children-only GMM-HMM and DNN-HMM systems, standard adaptation methods, and an adult-trained model used for transfer learning. DNN adaptation strongly favors SAT, while the children-only DNN provides only a modest gain over GMM-HMM.
- Baseline systems: The children-only evaluation includes GMM-HMM systems, a hybrid DNN-HMM baseline, and VTLN, SAT, and i-vector adaptation techniques.An adult-only ASR system is also evaluated on children’s speech for transfer-learning comparison.
- Baseline comparison: The hybrid DNN-HMM improves over its corresponding GMM-HMM system by 1% absolute.The authors attribute the modest improvement to insufficient children’s speech data for DNN generalization.
- Adaptation results: SAT reduces WER to 21.31%, a 14.66% absolute improvement over raw features.VTLN improves by 3.25% absolute, while combining VTLN and SAT provides no major additional improvement.
- Adaptation results: Offline whole-utterance i-vectors perform best among the tested i-vector configurations but remain 4.22% absolute worse than SAT.This pattern supports SAT as the strongest adaptation method across the evaluated GMM and DNN settings.
B. Transfer Learning Results
Transfer learning adapts adult DNN models to children’s speech by targeting acoustic and pronunciation variability. Jointly adapting the top and bottom layers produces the strongest reported results, while adaptation strategy depends on available data.
- 24.26% WER from acoustic-variability adaptation outperformed 26.97% WER from pronunciation-variability adaptation, versus 39.32% for the adult baseline.
- The results indicate that acoustic and pronunciation variability are complementary, and that transfer-learning configurations are task dependent.
- Updating both top and bottom layers reduced WER to 19.63%, a 23.1% relative gain over the children model and 50.1% over the adult model.
- A two-layer configuration achieved 17.8% WER, while adapting additional layers provided no significant improvement and adapting all layers produced the same 17.8% WER.
- Transfer learning outperformed children-trained SAT baselines by 16.5% relative and adult baselines by 54.7% relative, while exceeding combined adult-and-children training by 2.55% absolute.
- WER decreased with more adaptation data; two-layer adaptation was best below about 25 hours, after which configurations became similar.
A. Transfer Learning for low resource scenarios
Extreme low-data experiments evaluate disjoint training as a way to constrain adaptation when children’s data are scarce.
- Disjoint training further improved two-layer adaptation with only 35 and 45 minutes of adaptation data.
VII. AGE DEPENDENT ANALYSIS
Age affects both overall recognition performance and the usefulness of different transfer-learning layer configurations. Younger children show greater sensitivity to adaptation choices and benefit from adapting more layers.
- Recognition performance generally increased with age across adaptation configurations, while performance was worse for younger children.
- Adaptation-configuration effects were larger for younger children, with WER variance decreasing as age increased.
- Younger children benefited from adapting more layers, whereas older children could achieve stronger gains with fewer layers.
B. Amount of Adaptation Data vs. Age
More adaptation data improved performance across ages, but younger children required more data and remained harder to recognize even with extensive matched adaptation data.
- More adaptation data improved performance irrespective of children’s age.
- Younger children needed more data to reach the performance level of older children, with required data decreasing as age increased.
- Older children surpassed younger children using as little as 30 minutes of adaptation data, despite predominantly mismatched data.
- Two-layer adaptation was more beneficial with limited data, while adapting more layers became advantageous as adaptation data increased across ages.
VIII. ANALYSIS OF AGE DEPENDENT TRANSFORMATIONS
Age-dependent transformations are evaluated for portability across children’s ages. Matched transformations help younger children most, while age mismatch is more damaging for younger than older children and performance is relatively flat for ages 11–14.
- Age-dependent transformations were assessed across age groups using all-layer adaptation and confusion-matrix comparisons across four adaptation configurations.
- Matched age-dependent transformations significantly improve performance for younger children aged 6–10 years.
- For younger children, increasing age mismatch produces decreasing performance.
- Age mismatch causes a more drastic performance degradation for younger than older children.
- Performance is approximately plateaued for children aged 11–14 years, likely because limited data hinders meaningful transformation estimation.
IX. AGE DEPENDENT TRANSFORMATIONS VERSUS AGE INDEPENDENT TRANSFORMATIONS
The paper compares age-dependent and age-independent transformations while controlling for adaptation-data differences. More-data age-independent transformations initially outperform, but data-normalized comparisons favor age-dependent transformations for younger children and age-independent transformations for older children.
- Age-independent transformations trained on substantially more data consistently outperform age-dependent transformations across ages.
- The comparison is data-normalized using average age-independent transforms based on approximately 10 hours and matched age-independent transforms with age-matched adaptation amounts.
- Random sampling preserves the original corpus proportions with respect to age during data normalization.
- After data normalization, age-dependent transformations outperform age-independent transformations for younger children aged 6–10 years.
- Age-dependent gains become more prominent as age decreases, with the maximum gains observed for 6-year-olds.
- For older children aged 11–14 years, age-independent transformations outperform age-dependent transformations in both normalized comparisons.
A. Effect of adaptation layer configurations
Across adaptation-layer configurations, age-dependent transformations benefit younger children whereas age-independent transformations benefit older children. The interpretation of trajectory magnitudes remains constrained by age-specific data availability and age itself.
- Effect of adaptation layer configurations: Age-dependent transformations outperform age-independent transformations for younger children, while age-independent transformations benefit older children.
- Effect of adaptation layer configurations: The younger-versus-older trend observed with all-layer adaptation remains apparent across the other layer configurations.
- Effect of adaptation layer configurations: Absolute trajectory values depend on the amount of data available for each age and on age itself.
- Effect of adaptation layer configurations: The complex inter-relations among configuration trajectories limit straightforward interpretation of their absolute values.
- Effect of adaptation layer configurations: The study compares DNN and GMM-HMM systems and evaluates transfer-learning configurations across children’s ages and adaptation-data sizes.
- Effect of adaptation layer configurations: Future work will examine how adapted-layer weights vary to relate DNN transformations to linguistic and structural aspects of children’s speech.