PaperlayerView exact revision on arXiv

Source-linked AI summary

Phoneme-Aware Pronunciation Representations for L2-English L1-Background Accent Identification

Yangyang Qu, Massimiliano Todisco, Nicholas Evans

arXiv:2609.10466v1eess.AS

TL;DR

L2-English L1-background accent identification must distinguish unseen speakers using English pronunciation, where cues can depend on particular phonemes. PAID combines frozen acoustic features with transcript-derived phoneme alignments to form phoneme-conditioned tokens. On L2-ARCTIC, it achieves the highest mean performance among evaluated systems, with 81.41% accuracy and 81.21% macro-F1.

  • Problem

    Global utterance representations can obscure pronunciation cues tied to specific English phonemes, while the task is to identify an unseen speaker’s L1 background from English speech.

    Source: §I · p. 1

  • Method

    PAID uses frozen speech features and transcript-derived forced phoneme alignments to form segment-level acoustic tokens conditioned on aligned phonemes, without passing word- or sentence-level text to the classifier.

    Source: §I · p. 1, A. Frozen Speech Encoder and Phoneme… · p. 3, B. Phoneme-Token Construction and… · p. 4

  • Results

    81.41% accuracy and 81.21% macro-F1 were achieved on L2-ARCTIC, the highest mean performance among evaluated systems under four-fold speaker-disjoint evaluation.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The results support phoneme-aligned token construction in the WavLM diagnostic setting, with a smaller additional gain from phoneme-ID conditioning in the Whisper-based ablation.

    Source: §VI · p. 6

    The approach requires transcripts and forced phoneme alignments for training and evaluation, and the evaluation covers one read-speech corpus with 24 speakers.

    Source: §VI · p. 6

Abstract

from arXiv · show

We study speaker-disjoint accent identification for L2 English, where the goal is to predict a speaker's first-language (L1) background from English pronunciation. Most existing systems classify accents using a single utterance-level representation, but such global representations can obscure pronunciation cues that depend on specific English phonemes. We propose a transcript-assisted model that makes phoneme information explicit during accent identification. Instead of representing an utterance only as a global speech embedding, we represent it as a sequence of pronunciation units, each combining acoustic evidence from a spoken segment with the aligned English phoneme for that segment. A frozen speech encoder provides the acoustic features, while the transcript is used only to obtain phoneme-level forced alignments. No word-level or sentence-level text representation is passed to the accent classifier. Under a four-fold speaker-disjoint protocol on L2-ARCTIC, our model achieves 81.41% accuracy and 81.21% macro-F1, the highest mean performance among the evaluated systems. Diagnostic ablations support the importance of phoneme-aligned token construction, while a Whisper-based ablation shows an additional gain from phoneme information.

I. INTRODUCTION

This work addresses speaker-disjoint L1-background accent identification from English speech, where pronunciation cues may depend on specific phonemes. PAID pairs acoustic segments with aligned English phonemes and achieves the highest mean accuracy and macro-F1 among evaluated systems on L2-ARCTIC.

I. INTRODUCTION

  • The task predicts an unseen L2-English speaker’s L1 background from English pronunciation, rather than identifying the language being spoken.Speaker-disjoint evaluation addresses possible entanglement between accent labels and speaker identity.Source: §I · p. 1
  • Global or frame-aggregated representations can leave local pronunciation evidence unconnected to the English phoneme being realized.Accent cues may depend on vowels, consonants, rhotics, or segment timing; frame-level attention does not explicitly attach frames to intended phonemes.Source: §I · p. 1
  • PAID uses transcript-derived phoneme alignments to pair segment-level acoustic evidence with aligned phonemes, without passing word- or sentence-level text to the classifier.The transcript supplies phoneme spans and IDs; the classifier receives phoneme-aware pronunciation representations instead of only a global embedding.Source: §I · p. 1
  • PAID achieves 81.41% accuracy and 81.21% macro-F1, the highest mean performance among evaluated systems under four-fold speaker-disjoint L2-ARCTIC evaluation.The diagnostic WavLM setting shows the largest separation for phoneme-aligned token construction, while the Whisper-based ablation shows a smaller additional gain from phoneme-ID information.Source: §I · pp. 1–2

II. RELATED WORK

Prior accent-identification work includes broad accent recognition and pretrained acoustic representations, often pooled globally or across frames. Speaker-generalised representations address speaker leakage, while this work focuses on preserving local phoneme-level evidence.

A. Accent Identification with Pretrained Representations

  • Earlier accent-related systems addressed accent-aware ASR, broad accented-speech recognition, or general accent recognition rather than speaker-disjoint L2-English L1-background identification.Pretrained acoustic systems and resources also support broad accent identification, but representative systems commonly use global, frame-level, or attention-based summaries.Source: A. Accent Identification with… · p. 2
  • Pretrained speech representations contain accent-relevant information, but recent representative systems commonly aggregate them before classification.Examples include ECAPA-TDNN and Wav2Vec2/XLSR-style systems evaluated for broad accent identification using Common Voice.Source: A. Accent Identification with… · p. 2

B. Speaker-Generalised Accent Modeling

  • Speaker-generalised and non-timbral representations target speaker leakage, while PAID addresses the complementary issue of retaining local pronunciation evidence.A representation can be less speaker-dependent yet remain global; PAID organizes frozen speech features into phoneme-aligned tokens.Source: B. Speaker-Generalised Accent Modeling · p. 2

C. Phonetic and Segmental Evidence for Accent Modeling

Phonetic and segmental evidence motivates making aligned phoneme realizations the basic input units for accent classification. The method combines frozen acoustic features with transcript-derived phoneme spans and IDs, then groups features by aligned spans.

C. Phonetic and Segmental Evidence for Accent Modeling

  • Accent information is not uniformly distributed across an utterance, motivating representations that preserve phoneme-level pronunciation evidence before classification.Prior work studies phoneme-dependent discrimination, sub-phone modeling, and phonetic and prosodic information in SSL-based accent identifiers.Source: C. Phonetic and Segmental Evidence for… · p. 2

III. METHOD

  • PAID uses frozen speech features, phoneme-conditioned token construction, and token aggregation, with Whisper-small in the final system.The transcript supplies forced alignments and phoneme IDs, but word- or sentence-level text representations are not passed to the classifier.Source: §III · p. 2

A. Frozen Speech Encoder and Phoneme Alignment

  • The frozen encoder produces frame-level acoustic features, and the aligner supplies transcript-based phoneme spans and IDs for frame-to-span grouping.The Montreal Forced Aligner is fixed; silence, pauses, and other non-speech intervals are removed before valid phoneme intervals are retained.Source: A. Frozen Speech Encoder and Phoneme… · pp. 2–3
  • Freezing the encoder keeps the comparison focused on representation structure, since the 24-speaker corpus could make encoder fine-tuning prone to speaker overfitting.The baseline pools the same frame features globally, whereas PAID groups them by phoneme spans and conditions tokens on phoneme IDs.Source: A. Frozen Speech Encoder and Phoneme… · p. 3

B. Phoneme-Token Construction and Conditioning

The model groups encoder frames by valid aligned phoneme spans, pools each span into an acoustic token, and combines it with an embedding of the aligned phoneme. The resulting token represents both a segment’s acoustic realization and its intended English phoneme.

B. Phoneme-Token Construction and Conditioning

  • Each valid aligned phoneme span is converted to encoder-frame indices, and span mean pooling produces one acoustic token per phoneme realization.Silence, pauses, and other non-speech intervals are removed; padding is excluded from aggregation.Source: B. Phoneme-Token Construction and… · p. 3
  • The aligned phoneme ID is mapped to a learnable embedding and concatenated with the projected acoustic token.Together they form a phoneme-conditioned pronunciation token combining the segment’s acoustic realization with its intended English phoneme.Source: B. Phoneme-Token Construction and… · pp. 3–4

C. Token Mean Pooling and Classification

The model aggregates valid phoneme-conditioned tokens with masked mean pooling, then uses a linear classifier to compute the accent posterior. Training updates the projection, phoneme embeddings, and classifier using cross-entropy loss.

C. Token Mean Pooling and Classification

A. Dataset and Evaluation Protocol

The evaluation uses six-way L1-background accent identification on L2-ARCTIC with four speaker-disjoint folds. It compares speaker embeddings, utterance-level representations, frame-attention pooling, and phoneme-token variants under the same splits.

A. Dataset and Evaluation Protocol

B. Compared Systems

  • The comparison includes speaker embeddings, utterance-level pooling, frame-level attention, pretrained accent representations, and phoneme-token variants on common six-class, speaker-disjoint splits.Fixed-dimensional embeddings are evaluated with a lightweight classifier trained on the corresponding fold’s training labels.Source: B. Compared Systems · p. 4
  • System suffixes denote utterance-level pooling (Utt), speaker embeddings (Spk), and frame-level attention pooling (FrameAttn).Source: B. Compared Systems · p. 4
  • The baselines include ECAPA-TDNN speaker embeddings, Whisper-small mean pooling, GenAID, and Voxlect-Whisper.Voxlect-Whisper is treated as a large-data representation reference because its training corpus includes L2-ARCTIC.Source: B. Compared Systems · p. 4

C. Implementation Details

The implementation uses frozen speech encoders, offline forced phoneme alignment, and trainable downstream components. The supplied figure and table captions describe the reported comparison and diagnostic ablation formats.

C. Implementation Details

  • The full model uses frozen Whisper-small features; WavLM Base+ features are used for diagnostic systems, with downstream modules trained.Source: C. Implementation Details · pp. 4–5
  • Fig. 2 shows mean accuracy and macro-F1 with fold standard-deviation bars; Table I covers WavLM diagnostic ablations with mean ± standard deviation across folds.Source: Figure 2 · p. 5, Table I · p. 5
  • MFA supplies transcript-based phoneme spans and IDs at training and evaluation; the stress-stripped ARPAbet inventory contains 39 phones.Aligned intervals are converted to encoder-frame indices to form tokens.Source: C. Implementation Details · p. 5
  • For the final Whisper model, 768-dimensional features are projected to 192 dimensions and concatenated with 64-dimensional phoneme embeddings, yielding 256-dimensional tokens.Source: C. Implementation Details · p. 5
  • Training uses Adam and cross-entropy for 10 epochs with fixed hyperparameters and no patience-based early stopping; results are averaged over four folds.The learning rate is 2 × 10^-4 and batch size is 16.Source: C. Implementation Details · p. 5
  • The ECAPA and GenAID baselines use pretrained embeddings; Voxlect-Whisper extracts frozen utterance representations before fold-specific logistic regression.Source: C. Implementation Details · p. 5

V. RESULTS AND ANALYSIS

PAID achieves the strongest reported L2-ARCTIC results in the four-fold speaker-disjoint evaluation, outperforming utterance-level Whisper references. WavLM diagnostic results also favor phoneme-token organization over utterance-level representations.

A. Main Results

  • The WavLM-based phoneme-token variant improves substantially over WavLM-Utt, suggesting phoneme-level organization also benefits a non-Whisper frozen encoder.Source: A. Main Results · p. 5
  • Whisper-Utt scores 77.12% accuracy and 76.80% macro-F1; Voxlect-Whisper scores 77.39% accuracy and 76.76% macro-F1, despite its larger-data pretraining including L2-ARCTIC.Voxlect-Whisper is treated as a strong large-data reference, not a strictly data-disjoint baseline.Source: A. Main Results · p. 5
  • PAID reaches 81.41% accuracy and 81.21% macro-F1, exceeding Whisper-Utt by 4.29 percentage points in accuracy and 4.41 in macro-F1, and Voxlect-Whisper by 4.02 and 4.45, respectively.Source: A. Main Results · p. 5

B. Diagnostic Ablation Study

WavLM diagnostic comparisons favor phoneme-aligned token construction over utterance- or frame-level summaries, with phoneme-ID conditioning adding a smaller gain. Fold-wise results show gains on every speaker-disjoint fold, including the lowest-performing fold.

B. Diagnostic Ablation Study

  • 40.93% and 45.90% macro-F1 for WavLM without tokenisation and without span grouping, respectively, trail 72.04% and 72.65% for phoneme-token variants.Table I compares WavLM feature structuring; phoneme-aligned acoustic tokens are the dominant factor in this diagnostic setting.Source: B. Diagnostic Ablation Study · p. 6
  • Removing phoneme-ID conditioning reduces macro-F1 from 72.65% to 72.04%, a smaller gain than constructing phoneme-aligned acoustic tokens.Source: B. Diagnostic Ablation Study · p. 6

C. Fold-Wise Behavior

  • Phoneme-ID conditioning improves the Whisper-based model on all four folds, with the largest improvement on Fold 3, the lowest-performing fold for both variants.Source: C. Fold-Wise Behavior · p. 6
  • On Fold 3, accuracy rises from 65% to 71% and macro-F1 from 64% to 70%; this fold-level pattern should not be over-interpreted.Four-fold means and standard deviations provide a more stable view of cross-speaker generalization than a single split, which may be optimistic or pessimistic.Source: C. Fold-Wise Behavior · p. 6

D. Per-L1-Background Behavior

Per-L1-background F1 varies across groups, and PAID's gains over other systems are not uniform. The conclusions identify phoneme-level pronunciation evidence as beneficial while noting transcript, alignment, and corpus constraints.

D. Per-L1-Background Behavior

  • Across four speaker-disjoint folds, Hindi has high F1 for most systems, while Chinese and Korean show more system-dependent differences; PAID beats Whisper-Utt for all six groups but not Voxlect-Whisper for Hindi or Spanish.PAID has higher F1 than Voxlect-Whisper for Arabic, Chinese, Korean, and Vietnamese; the figure reports per-group F1 across evaluated systems.Source: Figure 3 · p. 6, D. Per-L1-Background Behavior · p. 6

VI. CONCLUSIONS

  • The conclusions state that preserving phoneme-level pronunciation evidence is beneficial for L1-background accent identification.PAID organizes frozen speech features into phoneme-conditioned tokens using forced alignments, without passing word- or sentence-level text representations to the classifier.Source: §VI · p. 6
  • The approach requires transcripts and forced phoneme alignments for training and evaluation, and the current evaluation covers one read-speech corpus with 24 speakers.The authors call for larger, more diverse L2-English corpora and broader speaking-condition evaluation.Source: §VI · p. 6
Loading 2609.10466v1…