Source-linked AI summary
A Training Criterion with Token-Level Tolerance to Transcription Ambiguity for Automatic Speech Recognition
Saurabh Kumar, Diptiman Mohanta, Prasanta Kumar Ghosh
TL;DR
CTC assumes the reference transcript is the only valid labeling, despite localized transcript differences that acoustics may not resolve. This paper moves OTC wildcard arcs to token granularity and combines token- and word-level bypasses. Token-level OTC improves over CTC on all 25 tasks, while the hybrid entropy-indexed system gives the lowest mean WER on every corpus and a 9.45% average relative reduction over CTC.
Problem
CTC treats the reference as the only valid labeling even when verbatim transcripts contain localized differences that the acoustics do not uniquely determine.
Source: §1 · p. 1Method
The paper adds token-level wildcard bypasses to OTC so unsupported tokens can be skipped while the rest of a word stays supervised, and combines them with word-level bypasses and entropy-indexed weight relaxation.
Source: §1 · p. 1Results
Token-level OTC improves over CTC on all 25 tasks; the hybrid entropy-indexed graph has the lowest mean WER on every corpus and a 9.45% average relative reduction over CTC.
Source: §1 · p. 1, §4.2 · p. 4Takeaways & Limitations
Validator transcriptions show that token-level models assign significantly more bypass probability to disputed characters than CTC while leaving agreed characters nearly unchanged.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Automatic speech recognition is typically trained assuming that the reference transcript is the only valid labeling of an utterance, yet even nominally verbatim transcripts contain localized differences in pronunciation, spelling, or lexical realization that the acoustics do not uniquely determine. Omni-temporal Classification (OTC) tolerates such noise by adding wildcard paths to the connectionist temporal classification (CTC) alignment graph, but its word-level arcs are too coarse, since bypassing one unsupported token discards supervision for the whole word. We move wildcard arcs to token granularity so unsupported tokens can be bypassed while the rest of the word stays supervised, and we combine token- and word-level arcs as complementary escape paths. Across 19 languages and three corpora, token-level OTC improves over CTC on all 25 tasks. We also replace epoch-indexed relaxation of the wildcard weights with a predictive-entropy-indexed schedule, which performs comparably while reducing dependence on training length. Combining this schedule with the hybrid graph gives the lowest mean word error rate (WER) on every corpus and a 9.45% average relative WER reduction over CTC. Independent validator transcriptions show that token-level models place significantly more wildcard-bypass probability than CTC on disputed characters, indicating that token-level tolerance targets localized transcript ambiguity.
1. INTRODUCTION
The hybrid token- and word-level graph with entropy-indexed relaxation gives the lowest mean WER on every corpus and a 9.45% average relative reduction over CTC. The paper targets localized transcript ambiguity that word-level bypasses handle too coarsely, and evaluates whether token-level tolerance aligns with disputed transcript characters.
1. INTRODUCTION
- CTC treats the reference as the only valid transcript, although verbatim transcripts can contain localized differences that the acoustics do not settle.Annotators may disagree on one or two characters while agreeing on the rest of a word.Source: §1 · p. 1
- Token-level wildcard arcs let the model bypass an unsupported token while retaining supervision for the rest of its word.The hybrid graph adds word-level bypasses as an escape path for wholly unsupported words.Source: §1 · p. 1
- Earlier OTC uses word-level arcs tuned on transcripts with 50% simulated errors and performs similarly to CTC on verbatim data.Retuning on clean speech helps only slightly because bypassing one unsupported token still skips supervision for the whole word.Source: §1 · p. 1
- Independent RESPIN transcriptions show that token-level models assign significantly more bypass probability than CTC to disputed characters while leaving agreed characters nearly unchanged.The authors interpret this pattern as linking token-level tolerance to localized transcript ambiguity rather than indiscriminate relaxation.Source: §1 · p. 1
- The hybrid token- and word-level graph with entropy-indexed relaxation gives the lowest mean WER on every corpus and a 9.45% average relative reduction over CTC.Token-level OTC also improves over CTC on all 25 tasks across 19 languages and three corpora.Source: §1 · p. 1
2. BACKGROUND
CTC sums over frame-level alignments that collapse to the reference transcript. OTC adds wildcard paths to the alignment graph, allowing unsupported transcript units to be bypassed.
2. BACKGROUND
- CTC sums over frame-level alignments that collapse to the reference transcript, so every valid path emits that transcript exactly.The graph is formed by composing the CTC topology with a linear acceptor of the transcript.Source: §2 · p. 2
2.1. Omni-temporal Classification
- OTC adds wildcard bypass arcs between transcript units and wildcard self-loops at each state, with wildcard emissions scored by the mean non-blank posterior.Bypass and self-loop arc counts contribute their respective log-weights to the path cost.Source: §2.1 · p. 2
- In published OTC, each transcript unit is a word, so bypass arcs span whole words and self-loops occur only at word boundaries.The segmentation of the transcript determines wildcard granularity.Source: §2.1 · p. 2
3. METHOD
The method moves OTC wildcard arcs to token granularity and combines token- and word-level bypasses as complementary paths. It indexes wildcard-weight relaxation by held-out predictive entropy rather than training epoch.
3.1. Arc granularity
- Token-level arcs let a single unsupported token be bypassed while the remaining tokens in its word stay supervised.Token-level bypass cost scales with the number of bypassed tokens, whereas word-level OTC replaces a bypassed word with one wildcard at a fixed penalty.Source: §3.1 · p. 2
- OTC-TW adds one word-level bypass per word to the token-level graph, complementing token bypasses for partially unsupported words with a single escape for a wholly unsupported word.The figure contrasts CTC, word-level OTC, token-level OTC, and the hybrid graph.Source: §3.1 · p. 2, Figure 1 · p. 2
3.2. Entropy-indexed decay
- Entropy-indexed decay measures training progress with held-out predictive entropy rather than epoch count, reducing dependence on training length.The procedure computes normalized non-blank posterior entropy, smooths it, converts it to progress, and geometrically interpolates wildcard weights between initial and final magnitudes.Source: §3.2 · p. 2
- Table 1 lists wildcard-arc settings, including initial weight, per-epoch decay, and the entropy-indexed schedule.The schedule types are denoted by E.Source: Table 1 · p. 3
4. EXPERIMENTS AND RESULTS
Across the evaluated tasks, token-level OTC improves over CTC, and the hybrid graph with entropy-indexed relaxation achieves the lowest mean WER on every corpus. Validator-transcription analysis finds greater bypass probability on disputed characters, supporting a link between token-level tolerance and localized transcript ambiguity.
4. EXPERIMENTS AND RESULTS
- Experiments cover 19 languages from LibriSpeech, FLEURS, and RESPIN, with WER evaluated using CTC prefix search without an external language model.The study also repeats LibriSpeech comparisons with a Conformer encoder to test whether results depend on the E-Branchformer.Source: §4 · p. 3
4.2. Overall results
- 9.45% relative WER reduction over CTC: OTC-TWH has the lowest mean WER on all three corpora.It also has the lowest WER on 9 of 25 tasks; its epoch-indexed counterpart degrades on RESPIN, falling 0.24% below CTC on average.Source: Table 2 · p. 3, §4.2 · p. 4
4.1. Training criteria
- Both token-level criteria improve over CTC on all 25 tasks without per-language tuning; the criteria share graph construction and differ in wildcard arcs and schedules.The hybrid combines token- and word-level bypasses, with entropy-indexed relaxation removing the RESPIN degradation of the epoch-indexed hybrid.Source: §4.2 · pp. 3–4, §4.1 · p. 3
4.3. Sensitivity to the schedule
- On a 10 h LibriSpeech sweep, OTC-Te is sensitive to τ and performs best at 0.775, while OTC-TH improves over CTC throughout the tested κ range and is less sensitive, with a best value of 1.3.These settings are used in the remaining experiments.Source: Figure 2 · p. 4, §4.3 · p. 4
4.4. Bypass probability and annotator disagreement
- Validator transcripts define disputed characters as those changed by at least one validator when validators do not all propose the same alternative; bypass probability is computed from alignment posteriors, not decoded output.All systems are probed at the same fixed bypass weight, and CTC is scored through the same graph as a control.Source: §4.4 · p. 4
- Token-level arcs raise disputed-character bypass probability by 27–36% over CTC while leaving agreed-character values essentially unchanged, across all nine languages on both splits.Every system places an order of magnitude more bypass probability on disputed than agreed characters; word-level arcs raise agreed words as much as disputed ones.Source: §4.4 · p. 4
5. CONCLUSION
The hybrid token- and word-level graph with entropy-indexed decay achieved the lowest mean WER on every corpus, while token-level OTC improved over CTC across all 25 tasks.
5. CONCLUSION
- 9.45% average relative WER reduction over CTC; the hybrid graph with entropy-indexed decay also had the lowest mean WER on every corpus.The hybrid combined token- and word-level bypasses.Source: §5 · p. 4
- Token-level OTC improved over CTC on all 25 tasks across 19 languages and three corpora, while published word-level OTC stayed close to CTC on LibriSpeech and RESPIN.The trends held with a Conformer encoder.Source: §5 · p. 4
- Validator transcriptions showed more bypass probability on disputed characters than agreed ones for token-level models, while agreed-character probabilities remained essentially unchanged.Future work will aim to learn wildcard weights and evaluate against multiple references.Source: §5 · p. 4