Source-linked AI summary
OpenPA: Forced-alignment-free Multi-granular Pronunciation Assessment with Mixed-transcript Adaptation
Fu-An Chao, Bi-Cheng Yan, Berlin Chen
TL;DR
Open-response pronunciation assessment faces scarce fine-grained training data and pipelines that rely on forced alignment and separate feature extractors. OpenPA uses Whisper cross-attention and mixed-transcript adaptation to score words and utterances across closed and open responses. It leads in overall averaged PCC for in-domain closed and simulated open responses, with strong zero-shot MultiPA performance and exploratory oral-cancer transfer.
Problem
Open-response pronunciation assessment has scarce fine-grained annotated training data, while prior systems often rely on forced alignment and separate components that may impede optimization and deployment across response settings.
Source: §1 · p. 1Method
OpenPA uses Whisper cross-attention to jointly score words and utterances without forced alignment, adapting to ground-truth and ASR transcripts through mixed-transcript training.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2Results
OpenPA leads in overall averaged PCC for closed and simulated open responses, shows strong zero-shot MultiPA generalization, and has exploratory oral-cancer transfer potential.
Source: §3.2 · p. 4, §4 · p. 4Takeaways & Limitations
The paper establishes a consistent multi-granular open-response benchmark and releases its evaluation code.
Source: §1 · p. 1Clinical validity on oral-cancer speech remains unestablished; larger atypical-speech cohorts and finer phonetic granularity are future work.
Source: §3.2 · p. 4, §4 · p. 4
Abstract
from arXiv · showhide
Recent advances in automatic pronunciation assessment (APA) have driven a shift from scripted to free-form assessment. However, existing methods often rely on multi-stage pipelines with forced alignment and separate feature extractors, limiting their practicality across response settings. In light of this, we propose OpenPA, a novel APA framework that leverages Whisper's cross-attention mechanism to jointly predict word- and utterance-level pronunciation scores without forced alignment. Through mixed-transcript adaptation and adaptive substitution weighting, OpenPA accommodates heterogeneous transcripts across closed- and open-response settings within a single model. Trained solely on speechocean762, OpenPA achieves the strongest overall performance among evaluated systems under in-domain closed and simulated open-response conditions, while maintaining its performance zero-shot on MultiPA. An additional cross-task evaluation on oral-cancer speech suggests its potential applicability beyond non-native speech.
1. INTRODUCTION
Automatic pronunciation assessment is expanding from scripted read-aloud speech to open responses, where fine-grained training data are scarce and alignment-based pipelines remain cumbersome. OpenPA proposes a unified alignment-free approach for word- and utterance-level scoring across response settings.
1. INTRODUCTION
- Prior APA has shifted from scripted read-aloud speech toward open responses that better reflect communicative speaking but bring additional challenges.Open responses allow learners to answer freely without predefined text prompts.Source: §1 · p. 1
- Fine-grained annotated training data are scarce for open-response assessment; MultiPA's 50 utterances are better suited to pilot evaluation than model training.Existing approaches therefore typically learn fine-grained scoring from closed-response corpora such as speechocean762 and evaluate transfer on MultiPA.Source: §1 · p. 1
- Prior systems commonly use forced alignment followed by separate feature extraction and scoring components, which may impede end-to-end optimization and deployment across response settings.Source: §1 · p. 1
- OpenPA uses Whisper cross-attention for word- and utterance-level scoring without forced alignment, with mixed-transcript adaptation and adaptive substitution weighting for both response settings.The paper also establishes a consistent multi-granular open-response benchmark and releases its evaluation code.Source: §1 · p. 1
2. OPENPA
OpenPA uses Whisper representations to score pronunciation at both word and utterance levels without externally supplied word timestamps. Mixed-transcript adaptation trains the model with ground-truth and ASR transcripts, while joint optimization retains Whisper’s ASR objective alongside pronunciation scoring.
2.1. Forced-alignment-free pronunciation assessment
- A word-level <score> token representation attends to acoustic encoder states and transcript context, then an aspect-specific regression head predicts each word’s pronunciation score.The representation does not require externally supplied word timestamps.Source: §2.1 · p. 2
2.2. Mixed-transcript adaptation
- Training samples ground-truth, L2-adapted ASR, or general-purpose ASR transcripts, and maps ASR words to reference scores while masking insertions.Matched words receive unit weight; substitutions are weighted using reference word accuracy, with poorly pronounced reference words receiving greater weight when ωmin = 0.5.Source: §2.2 · pp. 2–3
- Table 1 reports in-domain PCC on speechocean762, distinguishing ASR and ground-truth transcript conditions and matched-plus-substituted, matched, and substituted word scores.Overall Avg. PCC averages 10 reported PCCs for ASR input and 6 for ground-truth input.Source: Table 1 · p. 3
- Table 2 reports zero-shot cross-dataset PCC on MultiPA using medium.en transcripts.Dashes indicate unreported results.Source: Table 2 · p. 3
2.3. Joint optimization
- The joint objective combines ASR generation with word- and utterance-level pronunciation losses, using LoRA adapters throughout Whisper and task-specific scoring modules.The word and utterance losses are mean squared errors, and the word loss incorporates adaptive weights.Source: §2.3 · p. 3
- Table 3 reports zero-shot cross-task speaker-level PCC on the PathBench YouTube oral-cancer benchmark.Source: Table 3 · p. 3
3. EXPERIMENTS
OpenPA is evaluated on in-domain closed and simulated open responses, zero-shot MultiPA transfer, and exploratory oral-cancer speech. The experiments report strong benchmark performance and show cross-attention behavior under ASR substitutions.
3.1. Experimental setup
- The setup evaluates speechocean762 in closed and simulated open conditions, zero-shot transfer on 50 MultiPA responses, and exploratory transfer on oral-cancer speech.OpenPA uses Whisper-medium fine-tuned on speechocean762; neither out-of-domain dataset is used in training.Source: §3.1 · pp. 3–4
3.2. Evaluation results
- OpenPA leads in averaged PCC for closed and simulated open responses, outperforms MultiPA* at both granularities, and beats both TextPA variants on shared utterance-level metrics.On MultiPA, prosodic PCC rises from 0.473 for MultiPA* to 0.644 for OpenPA.Source: §3.2 · p. 4
- OpenPA reaches the highest speaker-level PCC of 0.82 on oral-cancer speech using averaged fluency predictions, although clinical validity remains unestablished.This was an exploratory zero-shot evaluation; the subset contained 98 utterances from 21 speakers.Source: §3.1 · p. 3, §3.2 · p. 4
3.3. Qualitative analysis
- Under a person-to-pizza ASR substitution, earlier attention patterns and scores remain stable while the final score changes from 4.20 to 4.64.Low scores for always (4.29) and thought (3.95) coincide with diffuse or shifted attention, unlike localized peaks for well-pronounced words.Source: §3.3 · p. 4, Figure 3 · p. 4
4. CONCLUSION
OpenPA assesses word- and utterance-level pronunciation without forced alignment across closed- and open-response settings. It shows robustness to ASR-generated transcripts and strong in-domain and zero-shot MultiPA performance, while oral-cancer results suggest cross-task potential.
- OpenPA jointly assesses word- and utterance-level pronunciation without forced alignment across closed- and open-response settings.Mixed-transcript adaptation improves robustness to ASR-generated transcripts, including those from unseen recognizers.Source: §4 · p. 4
- In-domain and zero-shot MultiPA results show strong assessment performance and cross-dataset generalisation.Source: §4 · p. 4
- Exploratory oral-cancer evaluation suggests cross-task potential; future work will examine larger atypical-speech cohorts and finer phonetic granularity.Source: §4 · p. 4