Source-linked AI summary
BaLEEN: Biasing with Latent Encoded Entities for Context-Aware ASR
Chihiro Taguchi, Yotaro Kubo, Rujikorn Charakorn
TL;DR
Recognizing domain-specific entities and rare proper nouns remains challenging for ASR, while existing adaptation can require domain-specific data or add inference cost. BaLEEN compresses contextual keywords into latent vectors and injects biases into a frozen ASR model. On the test set, it improves WER by 21% and CER by 28% relative to the non-fine-tuned baseline, with synthesized speech limiting the evidence about real-world acoustic conditions.
Problem
ASR needs external context to transcribe irregular named entities, while domain-specific fine-tuning can require sufficient audio–text data and costly dedicated training.
Source: §1 · p. 1Method
Results
21% WER and 28% CER reductions relative to the non-fine-tuned baseline were achieved by hidden-state biasing across all ASR layers under realistic dynamic context conditions.
Source: §6 · p. 4Takeaways & Limitations
Precomputed context bias vectors allow BaLEEN to provide contextual adaptation with virtually zero inference-time computational overhead.
Source: §1 · p. 1Evaluation uses synthesized speech, limiting evidence about generalization to real-world human voices, accents, and diverse acoustic environments.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Transcribing domain-specific entities and rare proper nouns remains a major challenge in automatic speech recognition (ASR). In this paper, we propose BaLEEN (Biasing with Latent Encoded Entities), a lightweight, hypernetwork-based framework for dynamic contextual adaptation without fine-tuning the underlying ASR model. BaLEEN encodes variable-length contextual keywords using a pretrained language model, compresses them into a fixed sequence of latent vectors via a Perceiver bottleneck, and injects context-dependent bias vectors directly into the intermediate encoder representations of the ASR model. Because both the language model and the backbone ASR model remain entirely frozen during training, BaLEEN operates as a plug-and-play adapter that incurs zero computational overhead at inference time when context biases are precomputed. We evaluate our method on a CTC-based ASR model using a Wikipedia-derived corpus with annotated named entities and synthetic speech. Experimental results demonstrate that BaLEEN reduces keyword miss rate by 8.7% on the test set relative to the unbiased baseline while simultaneously improving overall word error rate by 21% and character error rate by 28%.
1. INTRODUCTION
ASR systems need external context to recognize irregular named entities, but existing approaches can add inference cost or require costly domain-specific training. BaLEEN adapts a frozen ASR model using context-generated offsets and is evaluated on a synthetic Wikipedia-derived dataset.
1. INTRODUCTION
- Irregular named-entity spellings make external contextual information important for accurate transcription in mobile and domain-specific ASR.The paper cites contact names and specialized terminology as examples.Source: §1 · p. 1
- Cross-attention biasing and external-language-model rescoring can add inference computation and search latency; cross-attention cost scales with keyword-list size.Fine-tuning also requires sufficient domain-specific audio–text data and can entail training a dedicated model for each domain.Source: §1 · p. 1
- BaLEEN generates layer-wise offsets from context embeddings and injects them into ASR hidden representations without updating the target model’s parameters.The hypernetwork approach is intended to relax training-data constraints and avoid fine-tuning or retraining the backbone.Source: §1 · p. 1
- 14.3%: BaLEEN reduces keyword miss rate relative to the unbiased baseline while improving overall word and character error rates.The evaluation uses a synthetic dataset of 144k Wikipedia-derived audio samples.Source: §1 · p. 1
2. RELATED WORK
Prior contextual-biasing methods intervene during decoding, train bias encoders, or modify CTC-related representations. BaLEEN instead uses a hypernetwork to inject latent bias vectors without changing the decoding pipeline.
2. RELATED WORK
- Prior adaptation work spans decoding-time biasing, learned bias encoders, and approaches that bias CTC decoding or encoder representations.Examples include shallow fusion, CLAS [1], and CTC-oriented methods [19].Source: §2 · pp. 1–2
- Attention over one embedding per phrase scales linearly with keyword-list size and can become increasingly diffuse on large lists.Neural associative-memory approaches address this tension with two-pass top-K retrieval.Source: §2 · p. 2
- Unlike methods that add computation for keyword biasing, BaLEEN injects latent contextual bias vectors into the backbone without changing its decoding pipeline.Figure 1 notes that a linear injection layer is instantiated for each target ASR layer.Source: §2 · p. 2, Figure 1 · p. 2
3. METHOD
BaLEEN compresses variable-length context into fixed-length latent representations, projects them into selected ASR encoder layers, and adds the resulting biases to intermediate representations. Its context bias is audio-independent and can be disabled or precomputed for fixed context.
3. METHOD
- A Perceiver bottleneck compresses variable-length language-model representations into a fixed-length latent sequence using learnable query vectors.The default latent sequence has N = 16 queries of dimension d = 512.Source: §3 · p. 2
- The compressed representation is mean-pooled and projected through layer-specific linear maps to produce biases for selected ASR encoder layers.Figure 1 specifies a separate linear injection layer for each target ASR layer.Source: §3 · p. 2
- Each contextual bias is added across all acoustic frames, and the generated bias is independent of the audio input.The scaling parameter is initialized to zero, and setting it to zero lets users disable contextual biasing.Source: §3 · p. 2
- The proposed design relies on the backbone’s attention and nonlinear projections to amplify bias when acoustic evidence aligns with an entity and suppress it otherwise.The paper presents this as the motivation for time-invariant bias injection across encoder layers.Source: §3 · p. 2
- The context encoder and ASR backbone remain frozen during training; only the context bottleneck and projection layers are updated.For fixed context, the bias can be precomputed once, avoiding recomputation through the language model or bottleneck and yielding virtually zero runtime overhead.Source: §3 · p. 3
4. DATASET
The dataset provides diverse audio, transcription, and keyword-list triplets for training and evaluating contextual biasing. It is built from Wikipedia articles, entity annotations, and synthesized speech, with an option to include hard-negative distractors.
4. DATASET
- The synthetic corpus contains 144k audio samples derived from 993 English Wikipedia articles to provide diverse triplet data for context adaptation.The triplets pair audio and transcription with a keyword list.Source: §4 · p. 3
- Wikipedia articles are filtered, segmented into sentences, annotated with named entities, and converted to speech using randomly sampled voices from 30 speaker profiles.The articles are drawn from Featured, Good, and Vital Articles; sentences are tagged with GPT-5.4 Mini and synthesized with Gemini 3.1 Flash TTS.Source: §4 · p. 3
- Keyword lists can contain only entities spoken in the audio or all entities in the source article, with the latter adding contextually relevant hard-negative distractors.The paper says this more challenging configuration is intended to foster stronger context adaptability.Source: §4 · p. 3
- Train, validation, and test splits use distinct source articles to prevent context leakage during evaluation.The passage specifies an 80%/10%/10% split.Source: §4 · p. 3
5. EXPERIMENTS
The experiments evaluate contextual keyword biasing with a spelling-aware encoder and a frozen CTC ASR backbone, using keyword-recognition and general transcription metrics. They compare layer-selection and context-list settings against the non-fine-tuned baseline.
5. EXPERIMENTS
- The ASR backbone is an English CTC model with 17 Conformer encoder layers and a 1,025-token case-sensitive vocabulary; keywords use ByT5 layer 7 representations.Transcript normalization expands digits into words, removes specified punctuation, and filters utterances with unsupported characters; KMR is the proportion of ground-truth keywords omitted, alongside WER and CER.Source: §5 · p. 3
- Eight settings combine four encoder-layer selections with training contexts that either include sampled hard negatives or omit them, against the non-fine-tuned baseline.Dynamic distractors are used only during training; evaluation contexts contain keywords that appear in the utterance.Source: §5 · p. 3
- Table 1 reports validation-set metrics as percentages, with DC denoting dynamic context containing probabilistically mixed hard-negative keywords.The context bottleneck has 37.7M parameters, and each linear-layer projection adds 2.6M parameters.Source: §5 · p. 3, Table 1 · p. 4
- The direct-biasing baseline adds a cross-attention term over compressed context vectors to the CTC output logits.Experiments use a 5 × 10^-5 learning rate, 10 epochs, batch size 32, and one Perceiver encoder block; speech shorter than 1 second or longer than 20 seconds is excluded from training, and metrics use greedy decoding.Source: §5 · pp. 3–4
6. RESULTS
On the test set, all-layer hidden-state biasing with dynamic context achieves the strongest balance of keyword recognition and overall transcription accuracy. Validation results also favor all-layer injection with hard-negative distractors, while partial-layer injection generalizes weakly across splits.
6. RESULTS
- The validation-set best configuration—linear biases at every encoder layer with dynamic hard-negative context—reduces WER by 22.6%, CER by 31.7%, and KMR by 14.3% relative to baseline.Across variants, adding hard-negative distractors improves robustness; direct logit cross-attention exhibits training instability and appends an extraneous token.Source: §6 · p. 4
- On the test set, direct logit biasing without distractor keywords reduces KMR by 11.0% relative to baseline, but its decoding instability degrades WER and CER.The KMR gain occurs only with keyword lists restricted to terms uttered in the audio; the method also appends spurious tokens.Source: §6 · p. 4
- 21% lower WER and 28% lower CER, with an 8.7% relative KMR reduction, are achieved by all-layer hidden-state biasing under realistic dynamic context versus the non-fine-tuned baseline.This setting yields the best balanced test performance; Table 2 reports test-set metrics as percentages.Source: Table 2 · p. 4, §6 · p. 4
- Unlike direct logit biasing, the proposed method injects bias without conditioning on the audio input.Source: §6 · p. 4
- Among partial four-layer subsets, the selected layers do not significantly change performance, but partial-layer injection generalizes weakly across validation and test sets.All-layer biasing with dynamic context improves over the non-fine-tuned baseline.Source: §6 · p. 4
- The evaluation uses synthesized speech, so generalization to real human voices, accents, and diverse acoustic environments remains a practical limitation.The synthetic dataset was constructed to test contextual biasing with dense, arbitrary keyword lists.Source: §6 · p. 4
7. CONCLUSION
BaLEEN compresses variable-length contextual entities into fixed-length vectors and injects them into a frozen ASR model without fine-tuning its backbone. The paper reports improved recognition with all-layer biasing and hard-negative training contexts.
7. CONCLUSION
- BaLEEN compresses arbitrary variable-length entities into fixed-length latent vectors and injects them into intermediate representations of a frozen ASR model.The approach uses a hypernetwork and does not modify or fine-tune the backbone weights.Source: §7 · p. 4
- Precomputed bias vectors allow the method to incur zero runtime computational overhead and latency during inference for a static context list.Source: §7 · p. 4
- The benchmark is a large-scale Wikipedia-based synthetic dataset of audio, transcripts, and contextual keyword lists.Source: §7 · p. 4
- All-layer linear bias injection achieves the best overall performance, reducing KMR by 14.3% while improving WER and CER over baseline; hard-negative training contexts enhance generalization under realistic deployment conditions.Source: §7 · p. 4