Source-linked AI summary
POWSM: A Phonetic Open Whisper-Style Speech Foundation Model
Chin-Jou Li, Kalvin Chang, Shikhar Bharadwaj, Eunjung Yeo, Kwanghee Choi, Jian Zhu, David Mortensen, Shinji Watanabe
TL;DR
Phone-related tasks have largely been developed in isolation despite their shared links among audio, phones, and graphemes. POWSM jointly performs PR, ASR, audio-guided G2P, and P2G, outperforming prior PR models in-domain and out-of-domain while matching web-scale multilingual foundation models on low-resource ASR.
Problem
Phone-related tasks have traditionally used task-specific architectures and datasets, leaving a need for a unified framework that performs multiple tasks.
Source: §1 · pp. 1–2Method
POWSM reformats ASR datasets for four tasks and performs PR, ASR, audio-guided G2P, and audio-guided P2G within one attention-based encoder-decoder architecture.
Source: §1 · p. 2Results
POWSM outperforms prior phone-recognition models in-domain and out-of-domain and matches web-scale multilingual foundation models on low-resource ASR.
Source: §1 · p. 2Takeaways & Limitations
Audio-guided G2P and P2G support applications requiring fine-grained linguistic analysis, including atypical-speech assessment.
Source: §7 · p. 8Training uses cleaned, filtered phonemic transcriptions that are not fully faithful to audio phonetic or phonemic structure, and allophone-level data are insufficient.
Source: Limitations · pp. 8–9
Abstract
from arXiv · showhide
Recent advances in spoken language processing have led to substantial progress in phonetic tasks such as automatic speech recognition (ASR), phone recognition (PR), grapheme-to-phoneme conversion (G2P), and phoneme-to-grapheme conversion (P2G). Despite their conceptual similarity, these tasks have largely been studied in isolation, each relying on task-specific architectures and datasets. In this paper, we introduce POWSM (Phonetic Open Whisper-style Speech Model), the first unified framework capable of jointly performing multiple phone-related tasks. POWSM enables seamless conversion between audio, text (graphemes), and phones, opening up new possibilities for universal and low-resource speech processing. Our model outperforms or matches specialized PR models of similar size (Wav2Vec2Phoneme and ZIPA) while jointly supporting G2P, P2G, and ASR. Our training data, code and models are released to foster open science.
1 Introduction
POWSM unifies four phone-related tasks that have traditionally used separate architectures and datasets. The paper reports strong phone-recognition results and introduces a shared framework for mapping among speech, phones, and graphemes.
1 Introduction
- Phone-level representations offer a consistent cross-language basis for fine-grained analysis and cross-lingual generalization, supporting applications such as atypical-speech analysis and endangered-language documentation.Source: §1 · p. 1
- ASR, PR, G2P, and P2G address different relationships among audio, phones, phonemes, and graphemes, but have traditionally relied on isolated task-specific systems.Source: §1 · pp. 1–2
- POWSM jointly performs PR, ASR, audio-guided G2P, and audio-guided P2G in one architecture, using reformatted ASR datasets and an attention-based encoder-decoder.Source: Figure 1 · p. 1, §1 · p. 2
- POWSM outperforms prior phone-recognition models in-domain and out-of-domain, achieves low-resource ASR performance comparable to web-scale multilingual foundation models, and supports speech-grounded G2P and P2G across more than 70 languages.Source: §1 · p. 2
- The paper presents POWSM as a large-scale foundation model for multiple phone-related tasks, enabling conversion among speech, graphemes, and phones.Source: §1 · p. 2
- The authors analyze interactions among tasks, architecture components, and losses, and release data-preparation and evaluation scripts, code, and a model checkpoint.Source: §1 · p. 2
2 Related Work
Prior speech foundation models advanced multilingual ASR but did not explicitly target phone recognition or articulatory-level supervision. POWSM builds on related phone-recognition and G2P work by combining multilingual phone supervision with multiple tasks in an open foundation model.
Speech foundation models
- Whisper (Radford et al., 2023) and OWSM (Peng et al., 2023, 2024) advanced multilingual ASR and speech translation, but do not explicitly address phoneme recognition or articulatory-level supervision.Source: §2 · p. 2
- POWSM is the first open foundation model jointly trained on phone recognition and related tasks, integrating multilinguality, phonetic supervision, and multi-task scalability within one framework.Source: §2 · p. 2
Phone recognition
- Multilingual phone-recognition work includes language-specific models using phoneme or allophone inventories and language-agnostic approaches; POWSM follows the latter with data-driven phone representations.Source: §2 · p. 2
Speech foundation models
- WhisperPPT improved Whisper through data cleaning but remained limited in data coverage and task diversity, while POWSM is trained from scratch on open datasets.Source: §2 · p. 2
Phone recognition
- ZIPA (Zhu et al., 2025) repurposed ASR transcriptions for phone recognition using G2P; POWSM uses grapheme and G2P-generated phoneme transcriptions in ASR, PR, G2P, and P2G formats.Source: §2 · p. 2
G2P & P2G
- Text-based G2P imposes a one-to-one orthography-to-transcription mapping, whereas audio-guided G2P can represent acoustic variation across language varieties.Source: §2 · pp. 2–3
3 Methodology
POWSM is trained on multilingual speech with paired orthographic and phonemic transcriptions, formatted for four tasks. Its attention-based encoder-decoder uses hybrid CTC/attention training and phone targets for encoder alignment.
3.1 Data preparation
- Training uses IPAPack++, an open-source corpus of roughly 17,000 hours of multilingual speech paired with orthographic and phonemic transcriptions.G2P-generated transcriptions are manually inspected and cleaned; utterances longer than 300 phones are filtered, and IPA sequences are normalized to Unicode NFD.Source: §3.1 · p. 3
3.2 Multitask data format
- Each utterance is formatted for PR, ASR, audio-guided G2P, and P2G with text, language, and task prompts plus a target; G2P and P2G use graphemes and phones as prompts, respectively.PR and ASR use a blank text prompt.Source: §3.2 · p. 3
3.3 Training details
- POWSM follows OWSM v3.1 with an E-Branchformer encoder and Transformer decoder, trained from scratch using a hybrid CTC/attention loss with α_ctc set to 0.3.Source: §3.3 · p. 3
- The approximately 350M-parameter model has nine encoder and nine decoder layers; CTC aligns encoder outputs to simplified phone sequences with break and length diacritics stripped.Training used a 40 ms encoder stride, global batch size 256, and around 200 GPU hours on H100s.Source: §3.3 · p. 3
4 Experimental Setup
The setup evaluates phone recognition with a phonetic-feature edit metric and tests POWSM across in-domain, unseen-language, and English-variation data. It compares the model with specialized phone-recognition and multilingual ASR baselines.
4 Experimental Setup
- PFER measures articulatory-feature edit distance normalized by the number of phones in the reference transcription.Unlike PER's exact phone matches or PTER's separate treatment of diacritics and modifiers, PFER captures fine-grained phonetic similarity.Source: §4 · pp. 3–4
- Unseen-language evaluation uses a 45-language DoReCo subset, 95-language VoxAngeles, and Tusom2021, a low-data language absent from training.Tusom2021 contains narrow transcriptions of individual words; tones were removed.Source: §4 · p. 4
- English variation tests cover two dialect datasets and three L2-speaker datasets, using manually annotated L2-ARCTIC phoneme transcriptions.The L2-ARCTIC manual labels reflect speakers' actual productions rather than a single dictionary pronunciation.Source: §4 · p. 4
- Phone-recognition baselines include Allosaurus, Wav2Vec2Phoneme, MultIPA, Allophant, and ZIPA; ASR comparisons use OWSM and similarly sized OWLS models.ZIPA is evaluated both with and without additional pseudo-labeled data.Source: §4 · pp. 4–5
5 Results
POWSM matches or exceeds competitive baselines on phone recognition and ASR, particularly in low-resource settings. It also performs strongly on unseen-language phone recognition, while showing a comparative weakness on socio-phonetic variation.
5 Results
- POWSM achieves comparable or better PR and ASR performance than competitive baselines, particularly in low-resource settings.The authors suggest PR pretraining improves representation generality and reduces data needed to map acoustics to text tokens; they find no clear evidence that ASR benefits PR in this setup.Source: §5 · p. 5
5.1 Multi-task performance
- POWSM achieves the lowest average in-domain phone-recognition PFER, which the authors attribute to the decoder's strong language-modeling capability.They hypothesize English data cleaning may hurt Germanic-language PFER through a training-test mismatch, while reporting strong results on other languages against larger-data models.Source: §5.1 · p. 5
- POWSM's ASR is often comparable to similarly sized web-scale models, and PR-derived phone prompts significantly decrease WER.With gold phone labels for P2G, POWSM outperforms other ASR models by a large margin in most cases.Source: §5.1 · p. 5
5.2 POWSM generalizes well to unseen languages
- On unseen-language and language-variation datasets, POWSM beats ZIPA trained on the same data and ZIPA with extra pseudo-labels, but trails Wav2Vec2Phoneme on socio-phonetic variation.The authors report the best results on unseen languages while also performing three additional tasks.Source: §5.2 · p. 5
6 Analysis
The analysis examines how encoder targets and CTC weighting affect training and phone recognition, then probes how speech, text, and language prompts shape G2P, P2G, and PR. Its findings reveal trade-offs between phonetic detail and standardized output, and between in-domain performance and generalization.
6.1 Behavior of the speech encoder
- PanPhon targets without suprasegmentals show the earliest encoder-CTC validation PER drop, indicating earlier useful alignment and faster convergence.The comparison uses a 1k-hour subset and treats validation CER of encoder-CTC output as a proxy for training efficiency.Source: §6.1 · pp. 5–6, Figure 2 · p. 6
- Higher CTC decoding weights improve out-of-domain PR but degrade in-domain PR; increasing αctc only after convergence does not improve out-of-domain performance.Higher αctc from the start benefits unseen-language PFER while in-domain PFER is comparatively higher; randomly varying αctc also shows no improvement.Source: §6.1 · pp. 6–7, Table 5 · p. 7
6.2 Inspecting Task and Language Tokens
- Speech-only G2P performs comparably to PR, while adding text prompts degrades performance and produces more standardized pronunciations.Text-only prompting causes a sharp performance drop and highly standardized pronunciations.Source: §6.2 · p. 7, Table 6 · p. 7
- P2G outperforms ASR on the tested low-resource languages with gold phones, but PR-P2G improves performance inconsistently across languages.The gold-phone comparison is not entirely fair; PFER trends from PR do not explain the variable PR-P2G results, while ASR-aligned trends suggest phonotactic similarity may play a role.Source: §6.2 · p. 7, Table 7 · p. 8
- English-token P2G with post-processing often outperforms ASR and sometimes approaches P2G, while low- or high-PFER languages do not benefit.The method is tested for certain languages with Cyrillic or Gurmukhi transcriptions.Source: §6.2 · pp. 7–8
- On unseen-language PR, an unknown language token performs best, while detected-language tokens outperform always using English.The authors interpret the token effect as shifting the output distribution toward the assigned language.Source: §6.2 · p. 8
7 Conclusion
The conclusion presents POWSM as an open, scalable multilingual foundation model for phone recognition and related speech tasks. It reports strong PR performance and identifies applications requiring fine-grained linguistic analysis.
7 Conclusion
- POWSM is an open-source phonetic foundation model trained with a scalable multitask framework.The conclusion describes it as a strong phone-recognition foundation model and a resource for unseen languages and socio-phonetic variation.Source: §7 · p. 8
- POWSM achieves state-of-the-art PR performance while supporting ASR across more than 70 languages and audio-guided G2P and P2G.The model supports conversion among speech, orthography, and phones.Source: §7 · p. 8
- Audio-guided G2P and P2G support applications requiring fine-grained linguistic analysis, including atypical speech assessment..Source: §7 · p. 8
8 Future Work
Future work proposes adapting POWSM to socio-phonetic variation and studying its decoder and encoder behavior.
8 Future Work
- The authors propose unsupervised test-time adaptation or in-context learning to adapt to socio-phonetic variation.Source: §8 · p. 8
- Mechanistic interpretability is another proposed route for adapting to socio-phonetic variation.Source: §8 · p. 8
- The authors suggest early exiting may mitigate the decoder’s tendency to normalize socio-phonetic variation.This suggestion draws on a finding that earlier Whisper encoder layers preserve more phonetic detail.Source: §8 · p. 8
Limitations
POWSM’s limitations include transcription fidelity, language-resource bias, and engineering constraints in its multitask architecture. The authors also note ethical risks in modeling socio-phonetic variation and disclose LLM assistance with writing.
Limitations
- Training uses cleaned, filtered phonemic transcriptions that are not fully faithful to the audio, so POWSM is neither strictly phonemic nor phonetic.The authors also lack sufficient allophone-level data, which they describe as providing more language-independent information.Source: Limitations · pp. 8–9
- The model favors high-resource languages; its decoder language model and language tokens bias it toward the seen distribution.Source: Limitations · p. 9
- The AED architecture makes inference significantly slower than encoder-only models and does not easily support tone modeling, limiting use for tonal languages.Source: Limitations · p. 9
Ethics Statement
- The authors warn that normalizing socio-phonetic variation can threaten linguistic diversity, while accurate modeling may enable demographic inference.They state that POWSM’s uses should not leave low-resource varieties behind.Source: Ethics Statement · p. 9
The Use of LLMs
- LLMs assisted with grammar correction and clarity, while the authors say the conceptual, methodological, and experimental contributions were developed independently.Source: The Use of LLMs · p. 9
A.1 Refining English G2P
The appendix describes English G2P refinements, baseline and language-selection details, ASR comparisons, and multitask experiments. It also reports a Librispeech normalization correction and updated checkpoint results.
A.1 Refining English G2P
- English G2P rules refine plosive voicing and aspiration, lateral velarization, and vowel nasalization to address voice-onset-time confusion observed on unseen languages.The rules aspirate word-initial voiceless plosives, devoice word-initial voiced plosives, velarize syllable-final /l/, and nasalize vowels before nasal consonants.Source: §A.1 · p. 16
A.2 Baseline Implementation
- The appendix lists baseline training-data sources, language coverage, and checkpoint or repository links in Table 9.Source: §A.2 · p. 16, Table 9 · p. 17
A.3 FLEURS language selection for ASR
- The FLEURS ASR comparison retains nine low-resource languages after filtering by IPAPack++ hours, FLEURS coverage, and reported training-data amounts, then excluding ell.The authors note that comparison models use strictly more data than POWSM.Source: §A.3 · p. 16, Table 10 · p. 17
A.4 ASR Performance on In-Domain Data
- In-domain ASR performance is competitive with similar-size, similar-architecture models trained on much more data, with a smaller gap for languages contributing more training data.The gap is larger for midfrequency languages.Source: §A.4 · p. 16, Table 11 · p. 17
A.5 Multi-tasking at Different Scales
- Across task and data scales, there is no clear trend that multitasking benefits PR; performance degrades when model capacity is mismatched to available data.The authors say further evidence is needed to assess whether PR benefits less from scaling because data and model capacity are insufficient.Source: §A.5 · p. 16, Table 12 · p. 17
A.6 DoReCo (Paschen et al., 2020)
- The appendix enumerates DoReCo’s constituent datasets, including documentation datasets cited from 2022.Source: §A.6 · p. 16
A.7 Fixing ASR Text Normalization
- 20 GPU hours of continued training on H100s with corrected Librispeech data mostly resolved the normalization issue, yielding WER=14.2.A model trained from scratch with lowercased transcripts and punctuation removed except apostrophes and hyphens had similar or improved ASR and comparable out-of-domain PR with ctc=0.3, beam=1, enabling faster inference.Source: §A.7 · p. 17, Table 11 · p. 17, Table 13 · p. 17