Source-linked AI summary
A Native-Reference Coordinate Geometry for L2 Pronunciation Deviation Using Self-Supervised Speech Models
Tina Raissi, Nhan Phan, Mikko Kurimo
TL;DR
Pronunciation information in self-supervised speech features has not yet yielded an interpretable metric for spontaneous L2 speech assessment. The paper compares phone-class representations to native-reference coordinates without matched recordings, and reports negative Spearman correlations up to around −0.5 with proficiency.
Problem
Self-supervised speech features encode phonetic information, but an interpretable metric for L2 pronunciation assessment in spontaneous speech remains unclear.
Source: §1 · p. 1Method
The method projects native and L2 phone-class averages into a low-rank subspace derived from native speech and measures distances for matching classes, without training on proficiency ratings.
Source: §1 · pp. 1–2Results
Up to around −0.5 Spearman correlation, native-reference distances are negatively associated with speaker proficiency across the evaluated modeling choices.
Source: §4 · p. 4Takeaways & Limitations
The results suggest native-reference distances capture meaningful proficiency-related information, with their correlation strength affected by phone-class, averaging, rank, encoder, and alignment choices.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Self-supervised speech models encode rich phonetic information, but it remains unclear how to transform this information into interpretable metrics for second-language (L2) pronunciation assessment in spontaneous speech. We propose a native-reference coordinate geometry in which phone-class averages from native speech define a low-dimensional reference subspace, and L2 speech is evaluated by its distance to matching native phone-class coordinates. Unlike prior distance-based approaches, our method does not require parallel recordings with matched linguistic content or dedicated pronunciation labels. Across different self-supervised encoders and modeling choices, the resulting native-reference distances show negative Spearman correlations up to -0.5 with speaking proficiency, indicating that higher-proficiency speakers tend to lie closer to the native-reference space.
1 Introduction and Motivation
Holistic L2 proficiency scores do not show how a speech model represents pronunciation, while phone-level GoP has focused on constrained speech with known transcriptions. The paper proposes comparing L2 and native phone-class representations in a native-derived subspace without matched linguistic content.
1 Introduction and Motivation
- Holistic CEFR-aligned speaking assessment does not necessarily reveal how pronunciation information is represented, and GoP [4] has focused on constrained speech with known transcriptions.CEFR levels run from A1 to C2, and phonological competence has a dedicated control scale.Source: §1 · p. 1
- Self-supervised features encode context-dependent phone information, including context beyond the corresponding phone, but this does not directly provide an interpretable L2 pronunciation metric.Source: §1 · p. 1
- Prior distance approaches use dynamic time warping and require parallel recordings of the same text, whereas the proposed comparison uses shared phone-classes without matched linguistic content.The prior approaches may also be sensitive to disfluencies common in L2 speech.Source: §1 · p. 1
- The method tests raw native-reference distances against proficiency rather than training a model to predict proficiency, with the hypothesis that higher-proficiency speakers have lower distances.The proposed method constructs native phone-class averages and a low-rank subspace from self-supervised features, then compares matching L2 phone-class coordinates.Source: §1 · p. 2
2 Method Description
The method builds a low-dimensional coordinate system from native phone-class averages, projects L2 phone-class averages into it, and compares matching classes. It evaluates several distance, averaging, and phone-class choices and tests the distance–proficiency association with Spearman correlation.
2 Method Description
- Encoder layer features are associated with framewise phone-class labels, where classes may be monophones, diphones, or triphones.Source: §2 · p. 2
2.1 Building Native-Reference and L2 Coordinates
- Native phone-class mean vectors are centered and decomposed with rank-K truncated SVD to define a K-dimensional native-reference subspace and coordinates.L2 phone-class means are projected into the same basis after subtracting the global native mean.Source: §2.1 · pp. 2–3
2.2 Native-reference distance definition
- Distances are computed only for phone-classes shared by native and L2 speech, then averaged across those classes.Euclidean distance measures absolute displacement, cosine distance angular difference, and diagonal Mahalanobis distance normalizes by native-space variance across phone-classes.Source: §2.2 · p. 3
- For monophones, center-phone and full-unit averaging coincide; for diphones and triphones, full-unit averaging additionally includes frames aligned to all phones in the unit.Source: §2.2 · p. 3
- Spearman’s rank correlation tests whether higher proficiency tends to accompany lower distances; a negative correlation supports that inverse association.The method evaluates monotonic association between distance and CEFR ordering.Source: §2.2 · p. 3
3 Experiments
Across the S&I task, native-reference distances show a negative association with speaking proficiency, reaching approximately −0.5. Experiments also find that the strength of this association depends on modeling choices.
3.1 Models and Setting
- Experiments use TIMIT as the native corpus and train and dev partitions of the Speak and Improve corpus for L2 speech.Modeling choices are evaluated on train data; other experiments additionally use dev data, with results reported across four proficiency-evaluation tasks.Source: §3.1 · pp. 3–4
3.2 Results
- HMM-based alignment yields stronger negative correlations than CTC-based alignment, although CTC alignment still gives reasonable correlations.The authors suggest CTC's blank label, used for both repetition and silence, may lead to less accurate segmentation.Source: Table 5 · p. 4, §3.2 · p. 4
- ρ ≈ −0.5 (95% bootstrap confidence interval [−0.5294, −0.4591]) indicates a negative association between native-reference distance and proficiency.Higher-proficiency speakers tend to be closer to the native-reference space; this is an association, not a supervised proficiency prediction.Source: §3.2 · p. 4
- Triphone classes perform best; the strongest averaging strategy depends on encoder, rank 64 is best in the tested rank analysis, and modeling choices affect correlation strength.The averaging comparison favors full-unit averaging for WavLM and center-phone averaging for wav2vec 2.0.Source: §3.2 · p. 4
4 Conclusions and Future Work
The paper proposes native-reference coordinate geometry to analyze L2 pronunciation deviation in self-supervised speech representations. Its resulting distances correlate negatively with proficiency, and their strength varies with modeling choices.
4 Conclusions and Future Work
- The proposed native-reference coordinate geometry analyzes L2 pronunciation deviation in self-supervised speech representations without requiring matched text between native and L2 speech.Distances reach negative Spearman correlations of around −0.5 with proficiency, while the choice of phone-class, averaging, rank, encoder, and alignment affects correlation strength.Source: §4 · p. 4
A Supplementary Material
Figure 1 compares center-phone and full-unit averaging for a triphone phone-class, showing aligned boundaries and the segments selected for averaging.
A Supplementary Material
- Figure 1 contrasts which aligned triphone segments are included in center-phone versus full-unit averaging.The blue plot shows averaged WavLM output over feature dimensions for an utterance from S&I train data.Source: Figure 1 · p. 7