Source-linked AI summary
Learning New Words from Unlabeled Test Data in Automatic Speech Recognition
Mengqi Wang, Mark A. Hasegawa-Johnson, Haolong Zheng, Chang D. Yoo
TL;DR
Unknown words are often misunderstood in part because ASR lacks an adequate language-model representation of their usage contexts, and many OOV methods assume the spelling is known. The system detects mismatches between CTC alternatives and language-model context, then adds new tokens using top-K CTC spellings and adapts their representations and spelling distributions. On the OOV subsets, adaptation achieved up to 14.97% relative OOV-CER reduction; on SAP development, reductions versus corresponding shallow-fusion systems were 6.67%, 4.27%, and 3.85% for GPT-2, Qwen-3, and Gemma-3.
Problem
Unknown words are often misunderstood in part because ASR lacks an adequate language-model representation of their usage contexts, and many OOV methods assume the spelling is known.
Source: §1 · p. 1Method
Results
On the OOV subsets, adaptation achieved up to 14.97% relative OOV-CER reduction; on SAP development, reductions versus corresponding shallow-fusion systems were 6.67%, 4.27%, and 3.85% for GPT-2, Qwen-3, and Gemma-3.
Source: §5 · p. 4Takeaways & Limitations
Unsupervised estimation of new-word models permits ASR to recognize future instances of the same word with reduced error rate.
Source: §7 · p. 4The conservative strategy limits benefits for OOV words that occur only once.
Source: §6 · p. 4
Abstract
from arXiv · showhide
New words are invented every day. A human listener can learn a new word by hearing it clearly once and inferring its usage from sentence context. This paper proposes granting ASR a similar ability to learn the contextual representations and spellings of new words from unlabeled test data at test time. A frozen CTC acoustic model provides spellings, a frozen language model provides contextual evidence for out-of-vocabulary (OOV) word detection, and an adaptation module expands the vocabulary by learning the lexical token representations with distributions over CTC-generated candidates. The spelling model of each token is optimized by minimizing a Kullback-Leibler divergence (KLD) objective. We demonstrate that the CTC-weighted language model log likelihood ratio can be interpreted as the KLD between the unknown correct ASR and the unsupervised learned ASR, and that, using a Pinsker bound, the square root of KLD can be interpreted as an upper bound on the total variation distance between the true and estimated spelling of the unknown word. Experiments show relative OOV character-error-rate reductions of up to 14.97% on LibriSpeech and 6.67% on dysarthric Speech Accessibility Project data for recurring OOV words, relative to the corresponding rescoring system.
1. INTRODUCTION
Words with unseen spellings and pronunciations are difficult for ASR when language models lack evidence about their usage. The paper proposes learning new lexical tokens and spelling distributions from unlabeled speech at test time.
1. INTRODUCTION
- Unknown words are hard to transcribe partly because the language model lacks an adequate model of their usage contexts, while many prior OOV studies assume the spelling is known.When spelling is also unknown, context-biasing methods are less effective.Source: §1 · p. 1
- The proposed test-time lexical adaptation detects candidate mismatches, initializes new language-model tokens from CTC-generated lexical identities, and updates token representations and spelling distributions using unlabeled speech.The target words have never been seen in spelling or pronunciation.Source: §1 · p. 1
- CTC maps speech to a character sequence, then deterministically compresses it into language-model tokens under monotonicity and compressivity constraints.The acoustic model is trained to maximize the training dataset's log probability.Source: §1 · p. 1
- Text corpora contain several orders of magnitude more language-model tokens than speech corpora, and shallow fusion selects the transcript with the maximum acoustic-language-model fusion score.The fusion score includes an LM probability and a transcript-length penalty; the paper analyzes it using an explicit spelling model.Source: §1 · p. 1
- The paper models p(c|y) and p(c) explicitly rather than assuming permissible spellings have conditional probabilities proportional to their marginals.Under that proportionality condition, the paper states that its two scoring equations are equal.Source: §1 · p. 1
2. UNSUPERVISED LEARNING OF NEW WORDS
The paper expands a language-model vocabulary with new tokens learned from unlabeled speech. It estimates their spelling distributions from CTC candidates and relates the KLD objective to convergence toward the true spelling distribution.
2. UNSUPERVISED LEARNING OF NEW WORDS
- The method considers low-confidence acoustic segments when no known language-model token has both high language-model probability and high acoustic-model probability.The passage introduces this as the criterion for locating candidate new-word spans.Source: §2 · p. 2
- A new vocabulary item can be added by creating its input embedding and output weight vector, then optimizing these with spelling-model parameters under an unsupervised criterion.The new word's meaning is chosen to maximize its language-model probability.Source: §2 · p. 2
- Spelling estimation uses unlabeled speech samples, at least one believed to contain the new word, and a well-trained acoustic model p(c|x).The text introduces estimating the marginal probability of character sequences from these resources.Source: §2 · p. 2
- The spelling model q(c|ŷ) is estimated by minimizing KLD between its marginal q(c) and the known character-sequence marginal p(c).The marginal q(c) is formed from token probabilities and their spelling distributions.Source: §2 · p. 2
- When no cross-utterance recurrence can be identified, KLD can be estimated from one example, but the resulting spelling model is more uncertain.With sufficiently many unlabeled examples, Pinsker's inequality bounds the total variation distance between p(c) and q(c).Source: §2 · p. 2
- The divergence and total variation objectives are convex with a shared global minimum at q(c)=p(c), so KLD convergence to zero guarantees pointwise convergence of q(c) to p(c).A larger BPE vocabulary may be reasonable because BPE units can encode common training-corpus words as individual tokens.Source: §2 · p. 2
3. OVERALL ARCHITECTURE
The system finds possible OOV spans through disagreement between acoustic and language-model preferences, then adds and adapts placeholder tokens from CTC-generated spelling candidates. Existing placeholders can be reused and refined as further utterances provide evidence.
3. OVERALL ARCHITECTURE
- CTC-LM disagreement identifies candidate OOV spans by comparing acoustically plausible alternatives with language-model preferences.The system aligns N-best hypotheses to the 1-best transcript and scores the alternatives in context.Source: §3 · p. 2
- The highest-mismatch span is initialized from top-K CTC spellings with a uniform prior, and retained only if its spelling model differs sufficiently from existing placeholders.The reuse check uses Levenshtein edit distance and a threshold.Source: §3 · p. 2
- For a new placeholder, only its input embedding and output softmax parameters are fine-tuned in context, while the utterance is buffered for possible final rescoring.Buffering allows rescoring after further occurrences provide evidence and can reduce self-adaptation bias.Source: §3 · p. 2
- An existing placeholder replaces the known span only when it improves both contextual LM likelihood and local acoustic evidence; its spelling model and lexical parameters are then adapted.The spelling model, embedding, and softmax weights receive 100 gradient-descent steps.Source: §3 · pp. 2–3
4. EXPERIMENTAL SETTINGS
Experiments use frozen pretrained acoustic and language models with a dynamic OOV adaptation module, evaluated on LibriSpeech and dysarthric speech. Evaluation focuses on recurring OOV words without supplying their identities or candidate spellings at test time.
4. EXPERIMENTAL SETTINGS
- HuBERT-XLarge-LS960-FT supplies acoustic scores, while GPT-2, Qwen-3-0.6B-Base, and Gemma-3-270M language models remain frozen as adaptation modules handle new words.The adaptation module is instantiated from the corresponding language model; reported initialization uses K=5 CTC spellings.Source: §4 · p. 3
- Evaluation uses LibriSpeech test-clean and test-other, plus the Speech Accessibility Project development set after HuBERT-XLarge is fine-tuned on approximately 740 hours of SAP training speech.The SAP recordings are from speakers with speech disabilities.Source: §4 · p. 3
- OOV words are absent from supervised CTC training, and evaluation includes only words occurring more than once, with no target identity, spelling, or candidate list supplied at test time.For SAP, 921 qualifying utterances were identified; 200 were reserved for tuning and 721 used for evaluation.Source: §4 · p. 3
5. RESULTS
On frequent-OOV subsets, test-time adaptation reduced recognition errors across language-model backbones, with the largest LibriSpeech OOV-CER reduction reaching 14.97%. On SAP-dev, adaptation also reduced errors relative to shallow-fusion systems, with OOV-CER reductions up to 6.67%.
5. RESULTS
- 14.97% relative OOV-CER reduction at best; adaptation consistently reduced WER, CER, and OOV-CER versus corresponding base systems across language models on OOV subsets.Most improvements were statistically significant or highly statistically significant; the largest OOV-CER reduction was highly significant.Source: Table 1 · p. 4, §5 · p. 4
- Table 1 reports LibriSpeech results in percentages; parentheses give relative error reductions over corresponding shallow-fusion systems, and asterisks indicate significant differences from corresponding plain-rescoring baselines.The OOV subset contains test utterances with frequent OOV words; * denotes p < 0.05 and ** denotes p ≤ 0.001, measured with the NIST sc stats tool.Source: Table 1 · p. 4
- 6.67%, 4.27%, and 3.85% relative OOV-CER reductions were observed with GPT-2, Qwen-3, and Gemma-3, respectively, over corresponding SAP-dev shallow-fusion systems.The passage describes the error reductions with adaptation as small but significant on the dysarthric OOV subset.Source: Table 2 · p. 4, §5 · p. 4
6. DISCUSSION
Adaptation depends on locating true OOV spans and generating acoustically plausible spellings, and failures at either stage can lead to erroneous adaptation. Its conservative revision policy reduces self-adaptation risk but limits gains for words heard only once.
6. DISCUSSION
- Accurate OOV-span detection and reliable CTC spellings are key dependencies; ambiguous segments or rare in-vocabulary words can misdirect adaptation.A confident CTC misrecognition can prevent triggering adaptation because CTC-LM disagreement is absent.Source: §6 · p. 4
- If the correct spelling is absent from the first-occurrence top-K candidates, later placeholder reuse may introduce more errors.The adaptation module can propose spellings only from the options CTC supplies on that first occurrence.Source: §6 · p. 4
- The conservative policy preserves the first-utterance hypothesis unless later occurrences update the placeholder, limiting benefit for OOV words heard only once.The design is intended to reduce self-adaptation bias; broader revision coverage while preserving reliability remains future work.Source: §6 · p. 4
7. CONCLUSION
The paper concludes that ASR can learn incidental words from context by adding a new language-model token and estimating its representation and spelling. Unsupervised estimation permits reduced-error recognition of later instances of the same word.
7. CONCLUSION
- ASR can learn incidental words by adding a new token to the language-model vocabulary and estimating it from contextual use.The conclusion frames this as learning words from their use in context.Source: §7 · p. 4
- The new token's contextual representation is estimated from language-model context, while its spelling is estimated from top-K character sequences produced by a CTC acoustic model.Source: §7 · p. 4
- Unsupervised estimation of new-word models permits reduced-error recognition of future instances of the same word.Source: §7 · p. 4