Source-linked AI summary
Domain-Adaptive ASR for Telephony AI Agents: Fine-tuning Canary Flash Models for Enterprise Contact Center Applications
Chanameth Boonpramuk, Winn Voravuthikunchai, Songpol Bunyang
TL;DR
Telephony ASR must handle channel distortion, noise, speaker variability, and limited Thai telephony data. This report adapts Canary Flash models with NeMo using a hybrid telephony dataset, finding large in-domain accuracy gains while retaining real-time deployment practicality.
Problem
Off-the-shelf ASR models underperform on Thai telephony because their training conditions differ from narrowband telephone audio and public Thai telephony corpora are extremely limited.
Method
The study fine-tunes NVIDIA Canary 180M Flash and Canary 1B Flash with NVIDIA NeMo on internally collected telephony and business-domain speech.
Results
Fine-tuning reduces BOTNOI telephony CER from 23.31% to 9.04%, while names-and-addresses adaptation reduces business-critical CER from 16.98% to 3.78%.
Takeaways & Limitations
Domain-adaptive fine-tuning improves in-domain telephony and business-specific speech, while Canary Flash variants remain practical for real-time deployment.
Takeaways & Limitations
Telephony and domain-specific evaluations use internally collected Thai data without an external Thai telephony benchmark, so absolute CER values may not transfer directly.
Abstract
from arXiv · showhide
This technical report describes Botnoi Group's methodology and results for rapidly fine-tuning the open-source NVIDIA Canary 180M Flash and NVIDIA Canary 1B Flash multitask models for speech-to-text tasks using the NVIDIA NeMo framework, with a focus on telephony-grade audio. To support this adaptation, we construct a telephony-oriented fine-tuning dataset from live voicebot system recordings and prompted speech with telephony-oriented augmentation. We evaluate four targeted experiments-language adaptation (Thai), telephony robustness, domain-specific jargon (names and addresses), and latency-using character error rate (CER) for accuracy and real-time factor (RTFx) for inference speed. Results show that fine-tuning substantially improves recognition in noisy telephony environments, reducing CER from 23.31% to 9.04% on BOTNOI telephony data, and further improves business-critical names and addresses from 16.98% to 3.78% CER through domain-specific adaptation. Overall, our results show that domain-adaptive fine-tuning enhances business-critical terminology while preserving real-time responsiveness for production voicebot deployments.
1 Introduction
The report addresses the mismatch between off-the-shelf ASR training data and Thai telephony deployment conditions. It presents targeted fine-tuning and evaluation of multilingual Canary Flash models for accurate, low-latency enterprise voice applications.
- Motivation: Thai telephony ASR is challenged by 8 kHz narrowband audio, codec compression, packet loss, handset noise, and limited public corpora.These conditions differ from the 16 kHz wideband audio on which most ASR systems are predominantly trained.
- Approach: The study builds an in-house Thai telephony corpus from diverse handsets, locations, background conditions, and business-critical names and addresses.The collection yields 77 hours of general telephony audio and an additional 104 hours focused on names and addresses.
- Evaluation: The models are fine-tuned with NVIDIA NeMo and evaluated through Thai adaptation, telephony robustness, domain terminology, and latency experiments.Accuracy uses character error rate (CER), while inference efficiency uses real-time factor (RTFx).
- Findings: 23.31% to 9.04% CER: in-domain telephony fine-tuning produces a 61% relative improvement over the language-only baseline.The result is reported on BOTNOI telephony evaluation data.
- Findings: 16.98% to 3.78% CER: names-and-addresses adaptation further improves recognition of business-critical utterances.Canary 180M Flash also closely matches the approximately 6× larger Canary 1B Flash while running at RTFx > 600.
2 Model and Framework Overview
The report situates Canary Flash within modern multilingual encoder–decoder ASR and uses NVIDIA NeMo to support efficient domain adaptation. Its architecture and Flash variants are designed to balance accuracy, model size, and inference speed for real-time workloads.
- ASR architectures: Modern ASR commonly uses CTC, transducer, or attention-based encoder–decoder paradigms with jointly learned acoustic and linguistic representations.Transformer and Conformer architectures dominate the encoder side because of their scalability and robustness.
- Canary models: Canary combines a FastConformer encoder with a Transformer-based decoder for unified multilingual ASR and speech-to-text translation.The hybrid design uses convolution for local acoustic patterns and self-attention for long-range dependencies.
- Canary models: FastConformer uses 8× subsampling and linearly scalable attention to achieve roughly 2.8× faster inference while retaining, and often improving, Conformer accuracy.These properties are particularly valuable for real-time telephony workloads.
- Flash variants: Canary Flash variants target low-latency inference and provide a model-size, accuracy, and speed trade-off for voicebots and contact center automation.The report emphasizes carefully curated moderate-scale data and multitask objectives as part of Canary’s efficiency profile.
- Framework: NVIDIA NeMo supplies modular ASR components, augmentation, mixed precision, and distributed execution for efficient fine-tuning on domain-specific speech.The framework is used to adapt pretrained models to Thai telephony data.
3 Telephony Data Collection and Model Training Configuration
The telephony dataset combines authentic live-call audio with scalable prompted recordings transformed through telephony-oriented augmentation. Fine-tuning follows a NeMo-based configuration using a single NVIDIA A100 GPU and mixed-precision optimization.
- Data collection: The dataset combines live voicebot system audio with prompted messaging-application recordings to balance realism, lexical control, and collection efficiency.The hybrid pipeline is designed to reflect deployment conditions while scaling data volume.
- Real-call branch: Live voicebot calls are segmented with Voice Activity Detection, manually transcribed, and contribute 25 hours of authentic telephony speech.These recordings capture telephone-channel characteristics, spontaneous speech, and realistic conversational pacing.
- Prompted branch: Prompted messaging-app recordings provide known transcripts, controlled lexical coverage, and substantially more scalable collection than real-call data.Their cleaner environment and different channel do not fully match real telephony acoustics.
- Augmentation: Prompted speech is converted to mu-law and mixed with public and internal environmental noise to approximate telephone-channel conditions.The two-step augmentation reduces mismatch between clean voice messages and production voicebot audio.
- Dataset scope: The final telephony dataset includes 52 hours of noise-augmented training data, while all recordings were collected internally with consent.Names and addresses were synthetically generated rather than drawn from real individuals.
- Training configuration: Fine-tuning uses NeMo EncDecMultiTaskModel with AdamW, inverse square root annealing, fp16 mixed precision, and a single 40GB NVIDIA A100 GPU.The schedule specifies 1 million training steps with an audio bucket size of 320 seconds.
4 Experiments and Results
The experiments evaluate Canary fine-tuning for Thai language adaptation, telephony robustness, business-specific jargon, and inference latency using CER and RTFx. Results show improved domain performance, while model-size comparisons inform production deployment choices.
- Experimental design: Four experiments assess language adaptation, telephony robustness, domain-specific terminology, and latency using CER and RTFx.The study compares fine-tuned Canary models with open-source and commercial baselines.
- Datasets: 1,593 hours of Thai public speech, 77 hours of BOTNOI telephony audio, and 104 hours of names-and-addresses data support the experiments.The BOTNOI datasets use 90% for training and 10% for evaluation.
- Thai language adaptation: Fine-tuned Canary 180M Flash achieves competitive Thai CER and outperforms Whisper-large-v3 on Common Voice 23 and Fleurs.ElevenLabs Scribe v2 performs better on this clean read-speech evaluation, while the reported advantage for Canary emerges under telephony conditions.
- Telephony adaptation: Fine-tuning with BOTNOI telephony data substantially improves performance on telephony audio.The experiment retains general Thai capability by including the Thai language data during training.
- Domain-specific jargon: Domain-focused fine-tuning improves names-and-addresses accuracy but increases general telephony CER from 9.04% to 10.89%.The report characterizes this as a trade-off from specializing toward business-domain data.
- Latency: Canary 1B Flash achieves slightly higher accuracy but slower inference, leading to selection of Canary 180M Flash for production.The 180M model is selected for its lower memory footprint and efficiency in real-time applications.
5 Conclusion
Targeted fine-tuning adapts NVIDIA Canary Flash models for enterprise telephony ASR, improving in-domain and business-specific speech recognition. Both variants remain practical for real-time deployment, with different accuracy–efficiency trade-offs.
- Telephony adaptation reduces CER from 23.31% to 9.04% on BOTNOI telephony data.This demonstrates substantial gains from targeted fine-tuning for enterprise telephony conditions.
- Names-and-addresses adaptation further reduces CER from 16.98% to 3.78% on business-specific speech.The result demonstrates the value of domain-specific data for production voicebot settings.
- Both Canary Flash variants are practical for real-time deployment, with Canary 1B Flash achieving the best accuracy.Canary 180M Flash offers a more favorable efficiency–accuracy trade-off for production use.
- The findings suggest that moderately sized, carefully collected telephony data combined with multilingual pretrained models supports robust contact-center ASR.The conclusion connects data collection and efficient fine-tuning with deployment-oriented ASR performance.
6 Limitations
The evaluation is constrained by internally collected Thai data, participant-recorded calls, and a single latency-testing configuration. These boundaries limit cross-deployment transferability and generalization to spontaneous customer speech.
- Telephony and domain-specific experiments use only internally collected Thai data, with no external Thai telephony benchmark for cross-validation.Absolute CER values may not transfer directly to other deployments.
- Internal participants recorded the real-call audio, so genuine customer speaking styles and spontaneity may be underrepresented.The recordings reproduce telephony channel conditions but may not fully capture spontaneous customer interactions.
- Names-and-addresses adaptation raises general telephony CER from 9.04% to 10.89%, exposing a specialization trade-off.A model specialized for one domain trades off against broader telephony robustness.
- Latency comparisons use one hardware and batch-size configuration, so real-time factor may vary under different serving conditions.The reported latency does not establish performance across all deployment configurations.