Source-linked AI summary
Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems
Ting-Hui Cheng, Line Katrine Harder Clemmensen, Sneha Das
TL;DR
Performance disparities in English ASR may be associated with the structural linguistic distance between a speaker’s first language and English. The paper evaluates this relationship across ASR architectures and datasets, finding systematic LD-error associations and persistent L1/L2-related separation in deeper acoustic representations.
Problem
The paper addresses limited evidence on whether English ASR disparities are systematically associated with speakers’ first-language distance from English.
Method
The paper empirically evaluates architecturally distinct English-capable ASR models, relates LD to performance using regressions and Tweedie mixed-effects models, and analyzes layer-wise acoustic embeddings.
Results
LD generally has a positive association with ASR error metrics, remains significant after accounting for dataset variation, and L1/L2-related acoustic separation persists across deeper layers.
Takeaways & Limitations
The findings indicate that current architectures retain accent-related variation rather than consistently normalizing it, motivating accent-invariant representation learning.
Takeaways & Limitations
The analysis is limited by underrepresented geographic populations and by LD and L1 labels that may oversimplify multilingual histories and within-corpus variation.
Abstract
from arXiv · showhide
While automatic speech recognition (ASR) models have achieved remarkable improvements in recent years, performance disparities persist across different speaker populations. One such disparity is for speakers whose first languages (L1) are from families distant from English. This paper investigates the relationship between first language background and English ASR performance. Through empirical analysis, we observe that the correlation between speakers' L1 distance and ASR error rates yields a systematic effect on English Speech, with its strength varying across datasets and models. This association is statistically significant in a follow-up analysis accounting for dataset-level variation in Tweedie mixed-effects models ($p<0.001$ across evaluated models). In addition, analysis of the latent space reveals a L1-based spatial segregation across deeper acoustic layers in the majority of evaluated architectures
1 Introduction
ASR disparities persist across speaker populations, including differences associated with first-language background and linguistic distance from English. This work examines these associations across architectures and finds deeper-layer separation between L2 and English L1 acoustic representations.
- ASR systems continue to show performance disparities across diverse speaker populations.The paper frames robust performance across speaker populations as an essential requirement for widely deployed systems.
- L2 speakers exhibit higher Word Error Rates than L1 speakers across commercial English ASR systems and open-source models such as Whisper.
- Prior research links L1-L2 disparities to structural differences between languages and finds linguistic distance relevant to second-language processing and cross-lingual transfer.
- The paper tests whether L1-L2 English ASR gaps are systematically associated with structural Linguistic Distance across distinct ASR architectures.
- Across deeper layers, evaluated models maintain clear segregation between L2 acoustic representations and those of English L1 speakers.The authors interpret this separation as evidence that architectures isolate accented features rather than robustly normalizing them.
2 Related Work
Related work documents L1- and accent-associated variation in ASR performance, while prior studies have mainly quantified disparities rather than examining their structural linguistic association.
- Recognition accuracy varies with speakers’ language and accent, extending prior reports of L1-L2 performance disparities.
- Prior work has largely focused on quantifying L1-L2 performance gaps rather than investigating their underlying association with linguistic structure.
3 Methodology
The methodology quantifies linguistic distance between speakers’ first languages and English, models its association with ASR errors across datasets and architectures, and analyzes how linguistic structure appears in latent representations across network depth.
- Language linguistic distances: Linguistic distance (LD) is quantified as a composite score measuring dissimilarity between each speaker’s first language and English.The score ranges from 0 for identical languages to 100 for completely unrelated languages.
- Regression model: Regression analysis uses LD to estimate English ASR performance sensitivity, with the intercept representing the baseline error rate for English L1 speakers.The LD slope coefficient captures sensitivity to increasing linguistic distance, and performance metrics and LD are Min-Max normalized.
- Generalized Linear Mixed Model: Tweedie generalized linear mixed-effects models test the LD–WER association across datasets while modeling dataset-specific variation with random effects.The Tweedie likelihood accommodates continuous, non-negative error measures and a point mass at zero through a compound Poisson–Gamma representation.
- Embedding distance and correlation: Embedding distance is measured as cosine distance between each L2 utterance representation and the layer-specific centroid of English L1 utterance embeddings.Layer-wise Spearman correlations then relate embedding displacement to structural LD using ranked LD and embedding-distance vectors.
- Experiment Setup: The experiments compare transformer-based Whisper models with Conformer-based Parakeet and Canary models across datasets containing diverse L1 backgrounds.Evaluation includes WER, Word Information Loss, and Semantic Distance; SemDist uses RoBERTa-base embeddings and cosine distance to compare reference and hypothesis transcripts.
- Model layer depth influence on accent features: Layer-wise t-SNE visualizations examine whether L1-related clustering emerges and intensifies across the internal representations of Whisper Small.The visualizations use perplexity 30, 1,000 iterations, and random state 20.
4 Results
Across datasets and architectures, greater linguistic distance from English is generally associated with higher ASR error, though Fair-Speech is a counterexample. Latent representations also show increasing L1-based separation across deeper layers, with the association persisting after accounting for dataset variation.
- LD-ASR Association Across Datasets: LD showed a positive association with evaluated ASR error metrics across nearly all datasets, except Fair-Speech.Across regressions, LD explained 1%–56% of performance variance, with explanatory power varying by dataset and model setting.
- Fair-Speech Counterexample: Fair-Speech had negative regression coefficients across all evaluated models and metrics, with LD explaining 2%–31% of total variance.A preliminary human perception study found thin acoustic boundaries between the corpus’s L1 and L2 cohorts, with average classification accuracy of 58.8%.
- LD-ASR Association Across Datasets: β = 0.410, SE = 0.016, z = 25.20, p < 0.001 for Whisper-Small’s LD-WER association in the Tweedie mixed-effects model.With the log link, this corresponds to an approximately 52% increase in expected WER for a one-unit increase in LD, holding the dataset-specific random effect constant.
- LD-ASR Association Across Datasets: The LD-ASR error association persisted after accounting for dataset-level variation.The follow-up analysis used dataset-specific random intercepts in Tweedie mixed-effects models.
- Latent-Space Analysis: L1-related clustering emerged early and became increasingly pronounced in deeper layers across Whisper, Parakeet, and Canary.The separation remained evident in final layers, indicating that L1-related structure was preserved in the representations.
- Latent-Space Analysis: Spearman correlations between LD, embedding divergence, and WER intensified with network depth.Spearman’s rank correlation was used because of non-normality, and the stated statistical assumptions were satisfied.
5 Conclusion
The paper finds that linguistic distance from English is systematically associated with English ASR degradation across architectures, while Fair-Speech provides a counterexample. Its latent-space analysis shows persistent L1/L2 separation in deeper acoustic layers, with CanaryV2 emerging as an exception.
- As L1 linguistic distance from English increases, English ASR performance systematically degrades, and the relationship remains after accounting for dataset-level variation.
- L1/L2-related separation persists across deeper acoustic embedding layers, suggesting that models retain rather than normalize accent-related variation.
- CanaryV2 is a notable exception, indicating greater robustness to accent diversity within this investigation.
6 Limitations
The study’s limitations concern the representativeness of its speaker populations, the adequacy of linguistic distance and L1 labels, model coverage, and the English-only scope of the analysis.
- European and East Asian speakers comprise most L2 samples, leaving other regions underrepresented and limiting generalizability to all marginalized speaker groups.
- LD may not capture diverse linguistic backgrounds, while single L1 labels can omit multilingual influences, proficiency, acquisition age, and English exposure.
- No universally standardized LD measure exists, so observed associations may depend partly on the selected distance resource.
- The selected open-source models may encode L1 variation differently from massively multilingual or commercial systems with broader multilingual training.
- Because the empirical analysis focuses on English speech, further investigation is needed before generalizing language-distance effects to other target languages or broader settings.
7 Ethics Considerations
The paper identifies an ethical risk from L1-separated latent representations: internal bias may enable profiling or surveillance even when output metrics appear equitable.
- L1-separated latent representations could be extracted without consent to profile or surveil users based on their backgrounds.
- Output parity on WER, WIL, and SemDist can conceal internal separation, allowing structurally biased models to appear fair when deployed.
A Weak Explanatory Power of LD
LD explains ASR performance less strongly in some datasets because corpus conditions introduce additional variability beyond linguistic distance. The paper therefore interprets LD alongside dataset and speaker-population factors.
- LD has weaker explanatory power in EDACC and Afrispeech-200, where dataset-specific variability affects ASR performance beyond linguistic distance.
- EDACC contains spontaneous dyadic conversations with speaker overlap and background noise, while Afrispeech-200 contains clinical-domain speech.
- The observed variation should be interpreted alongside dataset design, sociolinguistic and demographic variation, multilingual histories, and other corpus-level factors.
C Tweedie Mixed Effect Models for Alternative Performance Metrics
Figure 5 indicates that the association between LD and ASR performance remains consistent when performance is evaluated with WIL and SemDist.
- The association between LD and ASR performance is consistent across the alternative metrics WIL and SemDist.
D Comparative Embedding Analysis Across ASR Architectures
Figure 7 uses layer-wise t-SNE projections to compare latent embeddings across ASR architectures, showing progressive segregation of L2 speech features. The observed pattern remains consistent across alternative t-SNE settings.
- D Comparative Embedding Analysis Across ASR Architectures: t-SNE projections trace the layer-wise evolution of latent embeddings across different ASR architectures.The projections use perplexity = 30, 1,000 iterations, and random state = 20.
- D Comparative Embedding Analysis Across ASR Architectures: Progressive segregation of L2 speech features appears across the evaluated ASR architectures.
- D Comparative Embedding Analysis Across ASR Architectures: The embedding patterns are consistent under alternative t-SNE settings.The alternatives use perplexity = 10, random state = 49, and perplexity = 30, random state = 42.
E Correlation Analysis for Alternative Performance Metrics
Figure 8 tracks Spearman correlations across network layers to compare internal representation shifts with alternative performance metrics. The plots examine alignment with WIL and SemDist.
- E Correlation Analysis for Alternative Performance Metrics: Figure 8 reports the evolution of Spearman correlation metrics across network layers.
- E Correlation Analysis for Alternative Performance Metrics: The plots compare internal representation shifts with alternative performance metrics.
- E Correlation Analysis for Alternative Performance Metrics: WIL and SemDist are the alternative performance metrics examined in the correlation analysis.