PaperlayerView exact revision on arXiv

Source-linked AI summary

Phoneme- and Word-Level Metrics Using Self-Supervised Speech Representations for Forced Alignment Evaluation

V. S. D. S. Mahesh Akavarapu, Michael Daniel, Gerhard Jäger

arXiv:2608.28508v1cs.CL

TL;DR

Forced alignment evaluation commonly depends on costly manual timestamps, limiting scalable multilingual assessment. This paper introduces reference-free PCMI and WACS from SSL-speech representations and evaluates them across perturbations, languages, and annotated references. The metrics consistently track alignment quality and support scalable screening, while remaining corpus-level surrogates rather than replacements for manual timestamps.

  • Problem

    Manual timestamp annotations remain costly, limiting scalable multilingual forced alignment evaluation.

  • Method

    The paper introduces PCMI and WACS, reference-free corpus-level metrics using SSL-speech representations to assess phoneme and repeated-word consistency.

  • Results

    Across synthetic perturbations and multilingual evaluations, the metrics degrade with alignment errors, correlate strongly with timestamp-based quality measures, and separate high- and low-quality alignments.

  • Takeaways & Limitations

    PCMI and WACS provide scalable alternatives for multilingual and low-resource alignment evaluation without manually annotated timestamps or MFA-derived references.

  • Takeaways & Limitations

    The metrics are corpus-level surrogate measures that do not directly diagnose individual utterances or local alignment errors.

Abstract

from arXiv · show

Forced alignment evaluation typically requires manually annotated timestamps, limiting large-scale and multilingual analysis. We introduce two corpus-level metrics based on self-supervised (SSL) speech representations for reference-free forced alignment evaluation: Phoneme-Cluster Mutual Information (PCMI) and Word Acoustic Consistency Score (WACS). PCMI measures agreement between aligned phoneme labels and clusters induced from SSL-speech representations, while WACS measures consistency of repeated word realizations using dynamic time warping similarity between word representation sequences. Using both random and systematic perturbations, we show that PCMI and WACS degrade consistently under alignment perturbations. We further analyze the metrics across multiple alignment systems on 85 languages from FLEURS, validate them against manually annotated alignments from 45 languages in DoReCo, and evaluate them on two phonologically complex low-resource languages. The metrics effectively separate high- and low-quality alignments and correlate strongly with timestamp-based alignment quality measures. Our results demonstrate that SSL-speech representations enable scalable, reference-free forced alignment evaluation. The metrics are available as an open-source Python package at https://github.com/mahesh-ak/forced-aligner-metrics.

1 Introduction

The paper introduces PCMI and WACS as reference-free metrics for forced alignment evaluation using SSL-speech representations, addressing dependence on costly manual timestamps. It evaluates their robustness and multilingual behavior across diverse languages and alignment settings.

  • Robustness analysis: Both metrics degrade consistently as alignments undergo increasing random or systematic perturbation.The analysis includes adversarial effects such as vowel and silence absorption.
  • Multilingual evaluation: Across 85 FLEURS languages, MFA alignments show a bimodal distribution separating failed and successful cases, whereas CTC-based aligners exhibit greater stability.The comparison includes MFA and evenly spaced phoneme baselines.
  • Contributions: PCMI and WACS provide reference-free, corpus-level forced alignment metrics based on SSL-speech representations.PCMI targets phoneme-label consistency with representation clusters, while WACS targets acoustic consistency across repeated words.
  • Validation: The metrics are validated using manually annotated alignments from 45 DoReCo languages and two phonologically complex low-resource languages.The low-resource evaluation covers Archi and Rutul.
  • Contributions: The paper releases multilingual phoneme recognition models used for alignment as a byproduct.

2 Related Work

Prior forced alignment evaluation relies on gold timestamp annotations and related temporal measures, creating scalability challenges for multilingual analysis. Information-theoretic and dynamic-time-warping methods provide relevant precedents for the proposed metrics.

  • Timestamp-based evaluation: Traditional evaluation measures boundary displacement or interval overlap against manually annotated timestamps.Reported measures include tolerance-based boundary accuracy, overlap rate, and average temporal shift.
  • Limitations of prior evaluation: Gold timestamps and multilingual annotation requirements make conventional evaluation difficult to scale.
  • Related metrics: Mutual information and cluster purity have been used in speech processing for feature selection, speaker clustering, and recognition evaluation.
  • Related methods: Dynamic time warping has also been widely used for spoken-term similarity and related speech-processing tasks.

3 Problem Formulation

The paper formulates alignment evaluation as estimating quality similarly to AAS without gold timestamps. Alignments assign transcript units to intervals bounded by predicted times, with silence tokens optionally included.

  • Alignment representation: An alignment assigns each transcript word or phoneme token an interval between successive boundary times spanning utterance duration T.Contiguous silence tokens may be inserted between units or at utterance edges.
  • Reference metric: Average Accumulated Shift measures temporal deviation between predicted and gold boundaries.
  • Evaluation objective: The proposed reference-free metrics aim to correlate with alignment quality similarly to AAS without requiring gold timestamp annotations.
  • Evaluation objective: Unless otherwise stated, AAS is reported as the maximum of word-level and phoneme-level AAS values.Across the considered datasets, the two levels differ by less than 5 ms; larger discrepancies mainly arise under artificial perturbations.

4 PCMI and WACS

PCMI measures mutual information between aligned phoneme labels and SSL representation clusters, while WACS compares repeated word realizations through DTW-based acoustic similarity. Their designs connect alignment quality to phonetic and word-level structure in speech representations.

  • Motivation: SSL-speech representations encode phonetic structure, enabling evaluation through consistency between aligned phoneme labels and induced representation clusters.
  • Phoneme-Cluster Mutual Information (PCMI): PCMI assigns frame-level representations to aligned phoneme labels, clusters them with K-Means, and computes normalized mutual information between labels and clusters.
  • Phoneme-Cluster Mutual Information (PCMI): Higher PCMI indicates that aligned phoneme assignments agree more strongly with emergent phonetic structure in representation space.Boundary errors corrupt labels near segment boundaries and increase conditional entropy, thereby lowering PCMI.
  • Word Acoustic Consistency Score (WACS): WACS tests whether repeated occurrences of the same aligned word are acoustically more consistent than unrelated word pairs.
  • Word Acoustic Consistency Score (WACS): WACS compares frame-level representation sequences for word occurrences using DTW with cosine similarity.Positive pairs come from distinct occurrences of the same word form, while negative samples use non-identical word forms.

5 Experiments

The experiments test PCMI and WACS under random, systematic, and layer-wise perturbations, showing consistent degradation but distinct vulnerabilities and robust operating conditions.

  • Random perturbations: Both PCMI and WACS decrease consistently as random perturbation severity increases, with approximately linear declines at lower shifts before plateauing.The resulting average accumulated shift after constraint enforcement ranges from approximately 45–210 ms.
  • Layer sensitivity: Middle transformer layers provide the clearest separation between clean and perturbed alignments, with layers approximately 5–16 showing greatest sensitivity.Layer 15 is selected for subsequent experiments by maximizing separation between clean and severely perturbed alignments.
  • Quantization sensitivity: PCMI’s clean-versus-severely-perturbed separation remains stable across 25 ≤n_cluster ≤100, varying from approximately 0.26 to 0.25 for MMS and 0.24 to 0.22 for XLSR.The experiments therefore use n_cluster = 50, approximately matching the average phoneme inventory.
  • Systematic perturbations: Vowel absorption primarily targets PCMI by changing phoneme assignments while preserving coarse acoustic continuity, whereas silence absorption targets WACS because DTW is relatively insensitive to leading or trailing silence.These systematic perturbations are designed to expose structured alignment errors that random perturbations may miss.
  • Systematic perturbations: Silence absorption produces substantially weaker degradation than random perturbations at comparable shift levels and emerges as a prominent adversarial failure mode for WACS.The perturbations achieve moderate AAS while weakening WACS less than random perturbations.
  • Robustness and limitations: PCMI remains informative under moderate realistic deviations, while WACS remains effective when silence regions are excluded during downstream segment extraction.PCMI also cannot be artificially inflated by degenerate phoneme-label merging, because PCMI approaches zero as phoneme-label entropy approaches zero.

6 Multilingual Evaluation

The multilingual evaluation examines PCMI and WACS across FLEURS and DoReCo, including comparisons across alignment systems, SSL representations, and phoneme-recognition quality. The metrics separate failed from successful alignments, correlate with timestamp-based quality, and reveal specific limits such as reduced sensitivity to silence absorption and fine-grained mergers.

  • Multilingual alignment models: MFA shows bimodal PCMI and WACS distributions separating failed and successful alignments, whereas CTC-based aligners produce more stable cross-language distributions.This pattern appears across both MMS and XLSR representations, with MMS providing slightly stronger separation overall.
  • Multilingual alignment models: Approximately 12k of 65k MFA utterances failed, compared with only 11 word-level failures for the IPA-based CTC aligners.The evenly spaced phoneme baseline remains competitive, and gold DoReCo alignments average approximately 0.33 PCMI and 0.13 WACS.
  • Validation against timestamp quality: PCMI correlates most strongly with AAS at approximately r ∼ −0.78, while WACS shows complementary correlations between r = −0.62 and −0.67.Because lower AAS indicates better alignment, these negative correlations indicate that higher PCMI and WACS scores correspond to lower alignment error.
  • Phoneme recognition and alignment quality: CER has moderate negative correlations with alignment metrics, while WER is weaker; this supports the metrics as alignment-quality indicators rather than mere phoneme-recognition proxies.CER correlations are approximately −0.5 with WACS and −0.3 to −0.5 with PCMI, whereas WER is approximately −0.3 with WACS in one configuration and weak in several others.

7 Discussion

PCMI and WACS offer more stable multilingual alignment evaluation than MFA- or timestamp-based comparisons, while exposing distinct sensitivity to silence-related errors.

  • Alignment systems: MFA achieves high PCMI when successful but fails on approximately 19% of alignments, whereas CTC-based models show near-complete robustness.This makes MFA-based reference comparisons unreliable in highly under-resourced settings.
  • Alignment systems: PCMI and WACS provide a more stable alternative to evaluating alignments primarily against MFA outputs in low-resource scenarios.The metrics do not depend on MFA training or successful MFA alignment generation.
  • Silence effects: AAS is highly sensitive to silence absorption, whereas PCMI and WACS remain comparatively robust to this perturbation.Manual DoReCo annotations often do not precisely trim short silences, so silence sensitivity can add variance unrelated to linguistic-unit alignment.
  • Cross-lingual evaluation: AAS may also be sensitive to speech-rate differences, while PCMI and WACS are comparatively rate-invariant by design.The authors frame the proposed metrics as scalable alternatives for multilingual evaluation without manually annotated timestamps.

8 Conclusions

The paper introduces PCMI and WACS as reference-free, SSL-based corpus-level metrics for forced alignment evaluation. Across perturbation and multilingual experiments, they correlate strongly with manual timestamp-based quality measures.

  • Contributions: The paper introduces PCMI and WACS, two reference-free corpus-level metrics based on SSL-speech representations.PCMI is Phoneme-Cluster Mutual Information; WACS is Word Acoustic Consistency Score.
  • Validation: Across synthetic perturbations and evaluations on 85 FLEURS languages, 45 DoReCo languages, and two phonologically complex languages, the metrics show strong negative correlation with manual timestamp-based measures.The experiments also examine silence absorption and phonological mergers, revealing complementary behavior among PCMI, WACS, and AAS.

Limitations

The metrics are scalable corpus-level surrogates rather than replacements for manual timestamps, and their coverage is limited by fine-grained phenomena and available phonemization resources.

  • Scope: PCMI and WACS screen corpus-level alignment quality but do not directly diagnose individual utterances or local alignment errors.They are intended as scalable surrogate measures, not direct replacements for manually annotated timestamps.
  • Measurement coverage: Systematic phonological mergers and coarticulatory effects may not be fully captured by the proposed metrics.The perturbation experiments include random and systematic errors, but not every linguistically fine-grained phenomenon.
  • Data and resources: Evaluation across 132 languages remains restricted to datasets with transcriptions and phoneme-conversion pipelines.Languages lacking reliable grapheme-to-phoneme resources remain comparatively underexplored, and alternative G2P systems were not ablated.

Ethics Statement

The supplied ethics passage reports no foreseeable ethical concerns for the publicly sourced datasets and temporal boundary annotations. The section also includes computational and methodological details of the evaluation pipeline.

  • Ethics: The datasets were publicly available, used under their licenses, and were not known to contain personally identifying or intentionally offensive content.The manually produced word-level annotations consist solely of temporal boundary labels over already public data.
  • Compute: The evaluations ran on one NVIDIA RTX 2080 GPU with 11GB VRAM and an Intel Xeon Gold 6140 system with 80GB RAM.The reported batch size was 2.
  • PCMI evaluation: PCMI samples up to 50 utterances per language and at most 10,000 acoustic frames before MiniBatch K-means clustering.Frame sampling is proportional to the square root of phoneme-label frequency to reduce dominance by frequent labels.
  • WACS evaluation: WACS samples up to 200 utterances and 200 word forms per language, requiring each occurrence to span at least four embedding frames.Positive and negative word-occurrence pairs are subsampled to limit frequency dominance.
  • Efficiency: The complete 85-language FLEURS benchmark requires approximately 40 minutes per embedding model, or roughly 30 seconds per language.The authors report that subsampling improves throughput while maintaining stable aggregate estimates with relatively small variances.
  • WACS computation: WACS represents each word occurrence as a sequence of frame-level SSL-speech embeddings and compares frames using cosine similarity.The embeddings are extracted from a selected SSL-speech model layer.
  • WACS computation: Dynamic time warping averages cosine similarity along an optimal monotonic path, producing duration-invariant similarity between word realizations.This accommodates differences in speaking rate and duration while preserving acoustic structure.
  • PCMI analysis: PCMI is normalized mutual information between phoneme labels and representation clusters, using Shannon entropy and mutual information.The paper argues that complete phoneme-label collapse cannot artificially inflate the metric.

D Phoneme Models Training

The study trains multilingual phoneme-recognition models and uses CTC segmentation for phoneme- and word-level forced alignment. It evaluates alignments across FLEURS and DoReCo while examining metric behavior under perturbations and representation choices.

  • Model training: MMS-300M-IPA, MMS-300M-DORECO, and Wav2Vec2-IPA are finetuned on multilingual IPA transcriptions, using language-specific adapter modules.The models are based on MMS-300M and Wav2Vec2-Phoneme, each approximately 300M parameters.
  • Model training: Training uses a 16-example effective batch size, AdamW optimization with learning rate 10−5, cosine scheduling, and 20 000 backbone steps followed by 500 adapter steps per language.The setup uses two NVIDIA RTX 2080 GPUs, gradient accumulation, fp16 precision, and a 0.01 warmup ratio.
  • Alignment pipeline: CTC segmentation produces phoneme- and word-level forced alignments, followed by sanity checks and postprocessing into valid TextGrid annotations.
  • Alignment processing: The silence-absorbing alignment variant trims low-energy leading and trailing regions within word intervals using a smoothed amplitude envelope and a threshold set to 5% of its 95th percentile.Intervals shorter than a 40ms minimum word duration receive additional handling.
  • Datasets and evaluation: DoReCo utterance chunks containing at least two words are split 0.7/0.1/0.2 into train, development, and test sets, while FLEURS retains its predefined splits.Alignment evaluations are performed on corresponding held-out data, and multilingual results are reported across the listed FLEURS and DoReCo languages.
  • Metric analysis: PCMI resolution remains stable across 25–100 K-means clusters for both MMS and XLSR representations as perturbation severity increases.The resolution is measured as the gap between clean and severely perturbed alignments.
Loading 2608.28508v1…