Source-linked AI summary
TurboBias 2.0: Streaming Context-Biasing for Production-Efficient ASR Systems
Vladimir Bataev, Lilit Grigoryan, Andrei Andrusenko, Nikolay Karpov, Vitaly Lavrukhin, Boris Ginsburg
TL;DR
Production ASR needs contextual phrase recognition that handles rare user terminology while meeting streaming, batching, personalization, and latency requirements. TurboBias 2.0 addresses these needs with case-insensitive GPU boosting, independent per-stream graphs, and streaming Transducer decoding. Experiments report improved contextual F-score and WER across Earnings22 and medical-domain evaluations.
Problem
Production ASR must recognize rare, domain-specific, and user-defined phrases accurately, but many context-biasing methods do not cover streaming, personalized batched decoding, and low-overhead requirements.
Method
TurboBias 2.0 combines a case-insensitive GPU phrase-boosting graph with per-stream context models and streaming Transducer greedy or beam-search decoding.
Results
TurboBias 2.0 consistently improves contextual F-score and WER on Earnings22 and an internal medical-domain set, with realistic per-stream global boosting reaching 81.5 F-score greedily and 87.5 with beam search.
Takeaways & Limitations
Independent per-stream context lists avoid cross-stream distractors while the framework supports efficient personalized contextualization in production-style batched streaming ASR.
Abstract
from arXiv · showhide
Contextualization is essential for production automatic speech recognition (ASR) systems, where user-provided phrases must be recognized accurately under strict latency constraints. Although many context-biasing methods improve recognition accuracy, they often do not address the practical requirements of modern production ASR systems: streaming inference, efficient batched decoding, user-specific context lists, and low runtime overhead. We propose TurboBias 2.0, a production-oriented framework for efficient phrase boosting in Transducer-based ASR systems. The framework extends GPU-accelerated TurboBias with a case-insensitive boosting graph and per-stream batched decoding, allowing each utterance in a batch to use an independent context-biasing configuration. This enables personalized context biasing for multiple simultaneous users without sharing or mixing their context lists. The proposed framework supports both offline and streaming inference and can be used with greedy and beam-search decoding. Experiments show that TurboBias 2.0 improves contextual phrase recognition while preserving low latency and high throughput.
I. INTRODUCTION
TurboBias 2.0 targets production ASR gaps by combining case-insensitive phrase boosting, independent per-stream context lists, and streaming Transducer decoding. The framework is designed to improve contextual recognition while retaining efficient batched GPU inference and low latency.
- Motivation: Rare, domain-specific, and user-defined terms remain difficult for general-purpose ASR despite strong benchmark accuracy.These terms are important for transcript usability in meetings, medical transcription, earnings calls, and voice assistants.
- Contributions: Case-insensitive boosting avoids expanding each phrase into lowercase, uppercase, and title-case variants.Direct case-insensitive matching reduces the tree-size, compilation-time, and memory costs associated with phrase-list expansion.
- Contributions: Per-stream boosting gives each batched request an independent phrase list, preventing distractors and false acceptances from other users.This supports personalized context biasing for simultaneous meetings, medical conversations, earnings calls, and customer-support sessions.
- Contributions: TurboBias 2.0 supports streaming Transducer greedy and beam-search decoding for real-time applications.The framework also extends phrase boosting to batched inference while keeping context-biasing overhead small relative to ASR inference.
- Evaluation: The evaluation covers Contextual Earnings-22 and an internal medical-domain test set using modern streaming ASR models.The study compares case-insensitive, case-sensitive, and phrase-expanded graphs across offline and streaming modes and examines F-score, beam size, and latency.
II. METHOD DESCRIPTION
TurboBias 2.0 converts context phrases into a case-insensitive GPU boosting graph that accommodates alternative cased BPE segmentations without duplicating phrase variants. Score potentials preserve consistent phrase bonuses across paths with different arc structures.
- Original TurboBias: Original TurboBias tokenizes each context phrase into a GPU phrase-boosting tree with suffix links and adds a shallow-fusion bonus for matched tokens.Its tree construction is efficient but case-sensitive, so differently capitalized hypotheses may not match the same user phrase.
- Case-Insensitive Boosting: Phrase expansion can cover common casing variants, but it increases boosting-tree paths and misses arbitrary mixed-case outputs.A character-level graph instead uses parallel arcs for case variants, preserving a compact capitalization-independent match.
- Case-Insensitive Boosting: Variative BPE decomposes greedy SentencePiece tokens into canonical character pieces and adds valid single-piece and merged-token arcs.This handles cased spellings whose BPE segmentations differ from the original phrase representation.
- Case-Insensitive Boosting: Lowercasing vocabulary pieces and phrase paths allows different cased BPE segmentations to converge on the same graph states.The resulting acyclic graph can contain multiple arcs that skip different character spans before converging.
- Score Assignment: State potentials assign arc scores as destination-minus-source differences, preventing path-dependent bonuses across character and merged-token paths.Potentials derive from the lowercased greedy BPE tokenization, while τ controls whether scores are distributed uniformly or concentrated near token completion.
B. Per-Stream Context Biasing
Per-stream context biasing merges multiple compiled boosting models into GPU tensors while selecting each stream’s model through an identifier. This preserves independent phrase lists and nearly the computational profile of one global tree.
- Motivation: Production batches require request-specific phrase lists so one customer’s names or entities do not affect another stream.A single global tree can introduce irrelevant distractors across simultaneous customer sessions.
- Merged Representation: TurboBias 2.0 concatenates the arc and state tensors of active boosting trees into merged multi-model storage.Per-model state and arc ranges are retained as offsets, together with each model’s boosting weight.
- Batched Decoding: Each batched stream carries a biasing model id that selects its phrase-list tensors, while id −1 disables biasing.The GPU kernel uses the selected offsets before running the standard suffix-link traversal.
- Batched Decoding: Independent stream selection prevents phrase lists from sharing distractors, while unbiased streams receive zero boosting scores and no valid next states.Thus, streams with id −1 are decoded exactly as unbiased streams.
- Efficiency: The merged layout adds only model-id and offset reads before the usual traversal, with no observable decoding-speed degradation in measurements.Compilation and transfer remain the main lifecycle costs; individual models are typically below 1 MB, and caching supports practical deployment.
C. Streaming Beam Search
The streaming beam-search extension adapts offline batched Transducer decoding to chunk-by-chunk encoder outputs. It maintains beam-search state across chunk boundaries for real-time recognition.
- Streaming Extension: Streaming decoding processes encoder outputs chunk by chunk, unlike offline decoding.This requires beam-search state to persist across chunk boundaries.
A. ASR models
The evaluation uses three 600M-parameter Transducer-based ASR models covering offline, streaming, and unified inference settings, all with punctuation and capitalization support.
- TDT-v2 is a 600M-parameter offline baseline using a FastConformer encoder and TDT decoder.
- Nemotron-streaming is a 600M-parameter cache-aware streaming model with an RNNT decoder that reuses encoder states across audio chunks.
- Unified is a 600M-parameter model supporting offline and streaming inference with shared parameters, chunk-limited attention, and dynamic chunked convolutions.
- All evaluated models support punctuation and capitalization, enabling study of case-insensitive phrase boosting.
B. Evaluation data
The study evaluates TurboBias 2.0 on earnings-call and medical speech, using contextual lists that represent local, global, and specialized terminology settings.
- Contextual Earnings-22 is a public earnings-call benchmark focused on person, company, and product names.Each 15-second sample includes at least one contextual keyword, a manually reviewed transcript, and a context list.
- C-Earnings22 provides local lists containing target-clip keywords and global lists drawn from the full call-level inventory with distractors.
- The internal medical-domain test set contains approximately 3 hours of English medical speech with diseases, procedures, medications, and clinical conditions.
- Validation data select context-biasing parameters, while test data support final evaluation.
C. Compared methods
The comparison includes TurboBias 2.0 variants and a standard CTC-WS baseline implemented with separate TDT and CTC inference paths; exact reproduction of a modified benchmark integration is unavailable.
- The compared-methods section includes the CTC-WS evaluation script reference and the evaluation-dataset table caption.
- The CTC-WS baseline produces the main transcript with greedy TDT decoding and detects contextual phrases using a separate TDT-CTC model.Its final text correction combines outputs from two independent ASR inference paths.
- The benchmark’s modified CTC-WS integration cannot be reproduced exactly because required implementation details are not provided.
- The study therefore uses publicly available ASR models and the standard CTC-WS method as closely as possible.
D. Metrics
The evaluation measures full-output recognition quality, contextual phrase detection, decoding efficiency, list-setting behavior, and streaming latency under greedy and beam-search decoding.
- WER measures recognition quality on the full ASR output, while precision, recall, and F-score measure contextual phrases only.A phrase counts as correct when it appears in the hypothesis and matches its reference after text normalization.
- RTFx measures decoding efficiency as audio duration divided by ASR decoding time, with higher values indicating faster decoding.
- The experiments compare full, per-stream global, and per-stream local context-list configurations.These settings range from one list for all streams to segment-relevant lists within each stream.
- Streaming latency is reported as worst-case algorithmic latency, computed from chunk size plus right context when applicable.
- Models are evaluated with greedy and beam-search decoding, using beam size 32 for ASLD++.
- Figure 2 presents contextual-biasing comparisons using WER and F-score.
IV. RESULTS
TurboBias 2.0 improves contextual recognition across offline and streaming Transducer decoding, while case-insensitive and per-stream configurations address practical production constraints.
- Offline decoding: 83.0 F-score and 13.3% WER are achieved by offline TDT-v2 with per-stream global boosting and beam search on Earnings22.The Unified model reaches 87.5 F-score and 12.6% WER, close to the reported Argmax CTC-WS F-score with lower WER.
- Streaming decoding: 81.7 F-score and 14.3% WER are obtained by Unified streaming at 1.12s latency after global boosting and beam search on Earnings22.The configuration uses a 0.56s chunk and 0.56s right context, improving from 62.2 F-score and 16.3% WER.
- Case-insensitive boosting: The case-insensitive tree reaches 75.4 F-score with greedy decoding and 81.4 with beam search, avoiding dependence on context-list capitalization.It slightly surpasses the target-cased case-sensitive tree and remains robust to lowercase or incomplete cased variants.
- Per-stream boosting: Per-stream global context reaches 81.5 F-score and 87.5 with greedy and beam search, while WER falls to 13.1% and 12.6%, respectively.Per-stream local context performs better but assumes knowledge of the exact phrases in each utterance, making global context the more realistic operating point.
- Efficiency: Keeping compiled models in CPU memory adds approximately 9-17% recognition-speed overhead, while decoder-level registration can hide model-loading and update overhead.The speed evaluation covers 128 concurrent streams.
V. CONCLUSION
TurboBias 2.0 provides efficient contextual phrase boosting for Transducer ASR by combining case-insensitive graphs, per-stream batched biasing, and streaming decoding. Across Earnings22 and a medical-domain set, it improves contextual accuracy and WER with low overhead without auxiliary CTC inference, post-recognition correction, or model retraining.
- Contribution: TurboBias 2.0 is an efficient framework for contextual phrase boosting in Transducer-based ASR systems.It supports offline and streaming inference with greedy and beam-search decoding.
- Contribution: The case-insensitive boosting graph handles capitalization differences without expanding phrases into multiple case variants.This makes decoding robust to differences between user phrase lists and formatted ASR output.
- Contribution: Per-stream context biasing gives each batched stream an independent context list and avoids interference from unrelated distractors.This supports personalized biasing for simultaneous streams.
- Results: Experiments on Earnings22 and an internal medical-domain set show consistent improvements in contextual F-score and WER.Beam search provides substantial gains over greedy decoding in offline and streaming settings.
- Implication: Decoder-level phrase boosting delivers strong contextual accuracy with low overhead without auxiliary CTC inference, post-recognition correction, or model retraining.The conclusion states this scope for the evaluated Transducer-based framework.