PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech

Bingshen Mu, Xian Shi, Xiong Wang, Hexin Liu, Jin Xu, Lei Xie

arXiv:2601.18220v2cs.SDeess.AS

TL;DR

Existing forced aligners are often language-specific and can accumulate temporal shifts in long-form speech, while SLLM next-token timestamp generation risks hallucinations and latency. LLM-ForcedAligner predicts discrete timestamps in transcript slots, and reports a 69%~78% relative reduction in accumulated averaging shift compared with prior methods across multilingual, crosslingual, and long-form speech.

  • Problem

    Existing forced aligners are tied to language-specific resources and can accumulate systematic temporal shifts in long-form speech; SLLM next-token timestamp generation can also hallucinate and incur latency.

    Source: §1 · p. 2

  • Method

    LLM-ForcedAligner inserts timestamp slots into transcripts and uses speech-conditioned SLLM predictions to fill discrete time indices, with dynamic slot insertion and non-autoregressive inference.

    Source: §1 · p. 2

  • Results

    69%~78% relative reduction in accumulated averaging shift (AAS) versus prior FA methods is reported across multilingual, crosslingual, and long-form speech scenarios up to 300 seconds.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results support LLM-ForcedAligner as an accurate and fast FA method for multilingual, crosslingual, and long-form speech scenarios.

    Source: §6 · p. 8

    The manually annotated test dataset covers only Chinese, limiting verification of performance on manually annotated data in other languages.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Forced alignment (FA) predicts start and end timestamps for words or characters in speech, but existing methods are language-specific and prone to cumulative temporal shifts. The multilingual speech understanding and long-sequence processing abilities of speech large language models (SLLMs) make them promising for FA in multilingual, crosslingual, and long-form speech settings. However, directly applying the next-token prediction paradigm of SLLMs to FA results in hallucinations and slow inference. To bridge the gap, we propose LLM-ForcedAligner, reformulating FA as a slot-filling paradigm: timestamps are treated as discrete indices, and special timestamp tokens are inserted as slots into the transcript. Conditioned on the speech embeddings and the transcript with slots, the SLLM directly predicts the time indices at slots. During training, causal attention masking with non-shifted input and label sequences allows each slot to predict its own timestamp index based on itself and preceding context, with loss computed only at slot positions. Dynamic slot insertion enables FA at arbitrary positions. Moreover, non-autoregressive inference is supported, avoiding hallucinations and improving speed. Experiments across multilingual, crosslingual, and long-form speech scenarios show that LLM-ForcedAligner achieves a 69%~78% relative reduction in accumulated averaging shift compared with prior methods. Checkpoint and inference code are available at https://github.com/QwenLM/Qwen3-ASR.

1 Introduction

Existing forced aligners face language-specific deployment and accumulated temporal shifts in long-form speech, while SLLM next-token prediction risks timestamp hallucinations and slow inference. LLM-ForcedAligner reformulates alignment as timestamp-slot filling and reports reduced accumulated averaging shift across multilingual, crosslingual, and long-form scenarios.

1 Introduction

  • Prior FA approaches include hybrid and end-to-end systems that align phoneme or token sequences through acoustic evidence and constrained path search.Examples include MFA, CTC, CIF, and WhisperX, which combines phoneme recognition with dynamic time warping.Source: §1 · pp. 1–2
  • Existing FA methods rely on language-specific phonemes, lexicons, or structural designs, and their local-similarity alignment errors can accumulate into temporal shifts in long-form speech.The authors say multilingual deployment typically requires independent systems, increasing engineering and maintenance complexity with the number of languages.Source: §1 · p. 2
  • SLLMs offer multilingual understanding and long-sequence processing, but treating FA as next-token timestamp generation risks temporal non-monotonic hallucinations and substantial inference latency.Existing SLLM applications have mainly addressed high-level semantic tasks, while FA is more sensitive to acoustic characteristics.Source: §1 · p. 2
  • LLM-ForcedAligner treats word or character start and end times as discrete indices predicted at special timestamp slots inserted into the transcript.Speech embeddings and the transcript augmented with slots condition the predictions.Source: §1 · p. 2
  • 69%~78% relative reduction in accumulated averaging shift (AAS) versus prior FA methods is reported across multilingual, crosslingual, and long-form speech scenarios up to 300 seconds.The comparison is reported with only a slight increase in real-time factor (RTF).Source: §1 · p. 2
  • Causal masking, non-shifted input and label sequences, slot-only loss, and dynamic slot insertion train each retained slot from preceding context and support arbitrary word or character positions.Non-autoregressive inference avoids hallucinations compared with autoregressive decoding and is faster.Source: §1 · p. 2

2 Related Work

Prior forced aligners include hybrid systems built around language-specific phoneme resources and end-to-end approaches based on acoustic-to-phoneme or token alignment. The paper contrasts these with SLLM timestamp generation, which can hallucinate and accumulate latency in long-form speech.

2 Related Work

  • Hybrid GMM–HMM systems use language-specific lexicons and phoneme models with Viterbi decoding; end-to-end models use CTC or attention weights to map acoustic features to phoneme sequences.Hybrid systems typically achieve high accuracy at phoneme-level boundaries, but generally require a separately trained or adapted model for each language.Source: §2 · pp. 2–3
  • CTC-based aligners can show backward shifts from sparse peak frames, while CIF can delay long-token end times and perform worse on English word boundaries than Chinese character boundaries.Scaled-CIF and fire-delay can partially alleviate CIF shifts, which remain difficult to eliminate in long-form speech.Source: §2 · p. 3
  • WhisperX uses phoneme recognition followed by dynamic time warping, and its local alignment errors can accumulate into global temporal shifts in long-form speech.It requires switching phoneme recognition models to accommodate different languages.Source: §2 · p. 3
  • Existing SLLMs treat timestamps as natural language during autoregressive inference, which can break guaranteed temporal monotonicity, produce excessive timestamps, and increase long-form inference time.Figure 2 contrasts this paradigm with transcript timestamp slots and non-autoregressive decoding.Source: §2 · p. 3, Figure 2 · p. 3

3 LLM-ForcedAligner

LLM-ForcedAligner inserts timestamp slots into transcripts and predicts discrete word- or character-level times directly from speech embeddings. Its training and inference strategies support flexible slot placement and simultaneous non-autoregressive prediction.

3.1 Overall Architecture

  • LLM-ForcedAligner predicts word- or character-level timestamp indices directly at transcript slots, rather than aggregating frame- or phoneme-level alignments.The SLLM conditions on speech embeddings and a transcript augmented with special timestamp tokens.Source: §3.1 · pp. 3–4
  • MFA-generated pseudo-timestamps provide training labels because manual annotation is prohibitively expensive, and the SLLM distills and smooths their noise and systematic shifts.The passage describes the resulting predictions as more stable and lower-shift than the pseudo-labels.Source: §3.1 · p. 4
  • The AuT encoder and multilingual LLM jointly support multilingual and crosslingual alignment without language-specific phoneme sets or lexicons.AuT supplies multilingual frame-level speech embeddings, while the LLM handles cross-language semantic information.Source: §3.1 · p. 4

3.2 Training Strategy

  • Causal training keeps input and label sequences non-shifted, uses preceding context, and computes cross-entropy loss only at timestamp slots.This lets the model perceive each slot while predicting its timestamp index.Source: §3.2 · p. 4
  • Dynamic slot insertion is applied with 50% probability per sample and gives each word or character a 50% chance of receiving start and end slots.Varying slot positions is intended to support timestamp prediction at arbitrary positions.Source: §3.2 · p. 5

3.3 Non-Autoregressive Inference

  • Users can place slots after any word or character, and the model predicts all requested timestamps non-autoregressively before converting indices to milliseconds.The conversion uses an 80 ms time interval.Source: §3.3 · p. 5

4 Experimental Setup

The experiments use multilingual speech data, an AuT–LLM model configuration, and accumulated averaging shift to evaluate timestamp prediction.

4.1 Dataset

  • The training and test data cover 56,000 hours and 10 languages, with read, conversational, podcast, and meeting speech among the scenarios represented.Datasets use MFA-generated pseudo-timestamps, and an internal Chinese test set has manual timestamps.Source: §4.1 · p. 5

4.2 Implementation Details

  • The model jointly optimizes an AuT encoder, Qwen3-0.6B LLM, and timestamp prediction layer, which has 3,750 output classes.The AuT encoder has 316.42M parameters; the timestamp prediction layer has 3.84M parameters.Source: §4.2 · p. 5

4.3 Evaluation Metrics

  • Accumulated averaging shift (AAS) is the mean absolute difference between predicted and ground-truth timestamps across test slots, with lower values indicating better timestamp prediction.The ground truth can come from MFA or manual annotations.Source: §4.3 · p. 5

5 Experimental Results

Across multilingual, crosslingual, and long-form evaluations, LLM-ForcedAligner reduces average AAS versus other aligners; ablations identify settings that balance timestamp fit and generalization.

5.1 Main Results

  • 66%~73% relative reduction in average AAS on multilingual raw speech, and 68%~78% relative reduction on human-labeled test datasets, versus other FA methods.The human-labeled datasets cover noisy, crosslingual, and long-form speech scenarios.Source: §5.1 · p. 6
  • On human-labeled long-form speech, average AAS is lower than on raw speech, unlike MFA-labeled long-form results, indicating correction rather than replication of MFA shifts.The authors attribute stronger long-form performance in part to using longer historical context when predicting current slots.Source: §5.1 · pp. 6–7
  • Non-autoregressive inference yields a substantial average AAS reduction with only a minimal increase in RTF, which rises slightly with model size.The paper frames method selection as an average AAS–RTF trade-off.Source: §5.1 · p. 7

5.2 Ablation Study

  • 80ms timestamp tokens are optimal: finer durations lower AAS on MFA-labeled tests but do not lower it on human-labeled tests, where fitting MFA timestamps reduces generalization.At 80ms, token duration matches the AuT encoder’s 80ms output frames.Source: §5.2 · p. 7, Table 5 · p. 7
  • Dynamic slot insertion reduces AAS on MFA-labeled and human-labeled tests, with larger gains on long-form speech, while supporting timestamps at arbitrary positions.Randomly inserting slots discourages reliance on previously predicted timestamps, which can lead to systematic temporal shifts.Source: §5.2 · pp. 7–8, Table 6 · p. 8

5.3 Visualization

  • 50% dynamic slot insertion gives the best human-labeled AAS: it decreases as the percentage rises to 50%, then increases above 50%.At 100% insertion, AAS reaches its highest value in the reported comparison.Source: §5.3 · p. 8, Figure 3 · p. 8
  • A 0.9B parameter size is optimal: smaller models lack capacity, while larger models show unchanged MFA-labeled AAS and increased human-labeled AAS.The authors associate the larger-model pattern with overfitting the MFA timestamp distribution.Source: §5.3 · p. 8, Figure 4 · p. 8

6 Conclusion

LLM-ForcedAligner casts forced alignment as slot filling, inserting timestamp slots into transcripts for direct prediction from speech and text. Its training and inference design supports timestamp prediction at arbitrary positions for multilingual, crosslingual, and long-form speech.

6 Conclusion

  • LLM-ForcedAligner treats timestamps as discrete indices and inserts special timestamp tokens into the transcript as prediction slots.The SLLM predicts indices at those slots conditioned on speech and the augmented transcript.Source: §6 · p. 8
  • Causal masking with non-shifted input and labels lets each slot predict its own timestamp from itself and prior context, with loss only at slot positions.Dynamic slot insertion supports predictions at arbitrary positions.Source: §6 · p. 8
  • Non-autoregressive inference avoids hallucinations and improves speed across multilingual, crosslingual, and long-form speech scenarios.The passage characterizes the method as accurate, fast, and customized for these settings.Source: §6 · p. 8

Limitations

Evaluation on manually annotated timestamps is limited to Chinese, while most training and test labels are MFA-generated pseudo-timestamps. The training data are unevenly distributed across languages.

Limitations

  • Manually annotated test data cover only Chinese, so they cannot verify actual performance in other languages.The authors use these data to assess whether predictions perceptibly deviate from human judgments in practical scenarios.Source: Limitations · p. 9
  • Most training and test data use MFA-generated pseudo-timestamps, so the model fits the MFA timestamp-prediction distribution.Manually annotated millisecond-level boundaries can themselves be ambiguous, complicating highly precise alignment.Source: Limitations · p. 9
  • The training dataset has an uneven language distribution, which may explain higher AAS for languages other than Chinese and English.The authors propose improving performance across other languages efficiently as future work.Source: Limitations · p. 9

A.1 Data Statistics

The data statistics describe a collection of speech resources and the construction of raw and mixed training and test sets. The manually annotated evaluation data are Chinese, with raw, noisy, long-form, and crosslingual test variants.

A.1 Data Statistics

  • A multi-domain Mandarin corpus comprises 10,000+ hours of high-quality labeled speech, 2,400+ hours of weakly labeled speech, and about 10,000 hours of unlabeled speech, totaling 22,400+ hours.Source: §A.1 · p. 11
  • Training data combine language- and source-specific speech resources, with raw data and mixed long-form speech built from randomly concatenated segments.For training, target durations are sampled uniformly from 30 to 500 seconds, and noise is randomly mixed into the concatenated speech.Source: §A.1 · p. 11
  • Evaluation includes MFA-labeled mixed test sets and an internal Chinese human-labeled source used for raw, noisy, mixed-duration, and crosslingual tests.The human-labeled mixed-duration sets have maximum durations of 60 and 300 seconds.Source: §A.1 · pp. 11–12

A.2 Details of Compared FA Methods

The compared aligners use different backbones across languages, and the comparison setup follows each method’s standard evaluation procedure. The listed methods include Monotonic-Aligner, NFA, and WhisperX.

A.2 Details of Compared FA Methods

  • Monotonic-Aligner supports only Chinese and uses Paraformer as its backbone; NFA generates token-, word-, and segment-level timestamps using NeMo CTC-based ASR models.The passage says the authors reproduced results on their test datasets following the methods’ standard evaluation procedures.Source: §A.2 · p. 12
  • The comparison switches backbone models across languages, with Table 8 listing each method’s backbone for each language.The table caption notes that a dash indicates no model is available for that language.Source: §A.2 · p. 12
  • WhisperX performs forced alignment using phoneme-recognition models specific to different languages.The passage directs readers to its standard evaluation procedures and available checkpoints.Source: §A.2 · p. 12
Loading 2601.18220v2…