Pronunciation Editing for Finnish Speech using Phonetic Posteriorgrams

1. Introduction

• Low-resource languages such as Finnish lack suitable L2 speech synthesis data, while accented-speech synthesis still requires recordings of accented speech.The paper motivates editing native speech to approximate specific L2 pronunciations.Source: §1 · p. 1
• Existing speech-editing models mask and refill word-level regions, use auxiliary edge losses for prosody consistency, and depend on external aligners.Earlier methods use mel-spectrogram reconstruction loss, while later diffusion-based methods improve synthesized sample quality.Source: §1 · p. 1
• PPGs are time-varying phoneme distributions that carry pronunciation information and can serve as interpretable content representations for speech editing.The paper also notes their use in pronunciation assessment, mispronunciation detection, and voice conversion.Source: §1 · p. 1
• PPG2Speech edits native speech to approximate L2 pronunciations using a multispeaker diffusion-based PPG-to-speech model enhanced with CFG and Sway Sampling.The approach uses Matcha-TTS’s flow-matching decoder.Source: §1 · p. 1
• The method uses PPGs as soft, explicit alignment information rather than requiring text input or hard alignment to determine the edited region.The paper evaluates naturalness, speaker similarity, and editing effectiveness, including with the proposed PAC metric.Source: §1 · p. 2

2.1. Flow matching for generative models

• Flow matching learns a vector field for a probability path from a simple prior toward the data distribution; OT-CFM uses linear interpolation between noise and data samples for an efficient loss.The OT-CFM field is linear, time-invariant, and depends only on the noise and data samples.Source: §2.1 · p. 2

2.1.1. Classifier-free guidance

• Classifier-free guidance combines conditional and unconditional score estimates at inference, with guidance strength controlling the trade-off between sample quality and diversity.The two estimates are learned by training one shared network conditionally and unconditionally.Source: §2.1.1 · p. 2

2.1.2. Sway Sampling

• Sway Sampling changes the timestep schedule from uniform sampling; negative and positive s values shift smaller steps toward early and late stages, respectively.At s = 0, sampling is uniform.Source: §2.1.2 · p. 2

3.1. PPG2Speech

• The PPG encoder upsamples Transformer/Conformer representations to mel resolution, while a CFG-enabled Matcha-TTS flow-matching decoder conditions on speaker and pitch and uses Sway Sampling at inference.A pretrained HiFi-GAN converts its mel output to waveform.Source: §3.1 · p. 2

3.2. Phonetic Aligned Consistency

• PAC compares edited and synthesized-speech PPG regions using framewise JSD and DTW for length differences, then normalizes cost by edited-region length so scores are comparable across edits.Source: §3.2 · pp. 2–3

4.1. Data

• The experiments combine Perso Synteesi and Finsyn Finnish speech, using 20 male and 30 female Perso Synteesi speakers and filtering spontaneous-style Finsyn utterances.Perso Synteesi prompts are shared across speakers; Finsyn has two female speakers.Source: §4.1 · p. 3

4.2. Model details

• The PPG encoder maps 32-dimensional PPGs to latent representations, then upsamples them; the decoder conditions on speaker embeddings, pitch, and a voiced/unvoiced flag.The decoder uses 256-dimensional speaker embeddings; the pitch embedding has dimension 16, and inference uses guidance strength 3 and sway coefficient −1.Source: §4.2 · p. 3

4.3.1. Objective Evaluation

• 0.86 unseen-set SECS: CFG raises PPG2Speech speaker similarity from 0.76, while PPG2Speech-CFG has MCD 3.69 and Pitch MAE 7.17 ¢.On seen data, Matcha-TTS-CFG reaches SECS 0.89, compared with PPG2Speech’s 0.83; PPG2Speech models perform better on unseen-speaker similarity than Matcha-TTS.Source: §4.3.1 · p. 4, Table 3 · p. 4
• Editing tests replace a selected source phoneme in the PPGs and text with a common L2 target, also editing adjacent identical phonemes in long vowels.The edited speech is evaluated by comparing edited PPGs with estimated PPGs in the aligned edited region.Source: §4.3.1 · p. 4
• In editing experiments, Matcha-TTS with CFG scored PAC 0.804, while PPG2Speech with CFG scored 0.709, a relative 11.3% reduction in PAC for PPG2Speech.PAC is the objective evaluation for editing effects.Source: Table 3 · p. 4, §4.3.1 · p. 5

4.3.2. Subjective Evaluation

• 3.51 ± 0.07 NMOS: PPG2Speech-CFG scores highest among synthesis models, while Matcha-TTS-CFG and PPG2Speech-CFG differ insignificantly in NMOS.The reported Wilcoxon signed-rank test gives p = 0.76 for Matcha-TTS versus Matcha-TTS-CFG NMOS.Source: §4.3.2 · p. 5, Table 5 · p. 5
• 0.87 ± 0.02 Positive Rate: Matcha-TTS-CFG scores highest for editing agreement, while PPG2Speech-CFG has the highest edited-speech NMOS at 3.66 ± 0.06.The Positive Rate measures whether edited speech agrees with the edited transcript.Source: §4.3.2 · p. 5, Table 5 · p. 5
• PPG2Speech-CFG achieves the highest NMOS, but Matcha-TTS-CFG has the highest Positive Rate, separating perceived edited-speech naturalness from editing agreement.The subjective tests used 16 native Finnish participants and 75 utterances per system per test.Source: §4.3.2 · p. 5, Table 5 · p. 5

• PPG2Speech synthesizes natural speech in synthesis and editing tasks and follows speaker identity, although timbre leakage slightly degrades SECS and SMOS.Source: §5 · p. 5
• Both PPG2Speech and text-based editing achieve more than 80% positive rates on editing test samples, but PPG2Speech scores better in PAC and worse in subjective evaluation.The authors note that PAC might not reflect actual editing effects in some cases.Source: §5 · p. 5
• Future work will add foreign phonemes and edit speech to mimic L2 errors in phoneme duration and open- versus closed-vowel placement.Source: §5 · p. 5