Source-linked AI summary
WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit
Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, Xin Lei
TL;DR
End-to-end speech recognition offers simplified training, but practical deployment remains difficult, particularly for streaming and for maintaining separate streaming and non-streaming systems. WeNet addresses these issues with a production-oriented toolkit centered on the U2 two-pass model, whose dynamic chunk strategy supports both modes and whose runtime design targets deployment. The paper reports comparable non-streaming performance, promising streaming results, and production-oriented benchmarks and integrations.
Problem
Deploying E2E ASR in production is difficult because streaming can require substantial effort or reduce accuracy, while separate streaming and non-streaming systems increase development, training, and deployment costs.
Method
WeNet uses U2, a two-pass joint CTC/AED model with dynamic chunk training to unify streaming and non-streaming recognition, alongside PyTorch-based production runtimes.
Results
WeNet provides comparable full-attention results, promising limited-chunk streaming results, and reported runtime latency and real-time-factor benchmarks.
Takeaways & Limitations
WeNet offers a unified, production-oriented E2E ASR solution intended for deployment across server and embedded platforms.
Abstract
from arXiv · showhide
In this paper, we propose an open source, production first, and production ready speech recognition toolkit called WeNet in which a new two-pass approach is implemented to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model. The main motivation of WeNet is to close the gap between the research and the production of E2E speechrecognition models. WeNet provides an efficient way to ship ASR applications in several real-world scenarios, which is the main difference and advantage to other open source E2E speech recognition toolkits. In our toolkit, a new two-pass method is implemented. Our method propose a dynamic chunk-based attention strategy of the the transformer layers to allow arbitrary right context length modifies in hybrid CTC/attention architecture. The inference latency could be easily controlled by only changing the chunk size. The CTC hypotheses are then rescored by the attention decoder to get the final result. Our experiments on the AISHELL-1 dataset using WeNet show that, our model achieves 5.03\% relative character error rate (CER) reduction in non-streaming ASR compared to a standard non-streaming transformer. After model quantification, our model perform reasonable RTF and latency.
1. Introduction
WeNet targets the practical barriers to deploying end-to-end speech recognition, especially streaming, unified streaming and non-streaming operation, and production integration. It presents a production-oriented toolkit designed to address these barriers across models, runtimes, and platforms.
- E2E ASR simplifies training compared with conventional hybrid ASR frameworks.
- Streaming inference is essential for low-latency scenarios, but models such as LAS and Transformer are difficult to run streamingly without substantial effort or accuracy loss.
- Unifying streaming and non-streaming systems in one model can reduce development, training, and deployment costs.
- WeNet addresses streaming, unification, and production challenges through the U2 two-pass framework and PyTorch-based development and deployment.
- WeNet supports production use across x86 servers and ARM-based Android platforms through portable runtimes.
- Experiments describe WeNet as an easy-to-learn toolkit with an end-to-end path from research to production.
2.1. Model Architecture
U2 combines a shared limited-right-context encoder with CTC and attention decoders to support unified streaming and non-streaming recognition. Its two-pass runtime generates CTC hypotheses and uses attention rescoring as the production decoding solution.
- Architecture: U2 comprises a shared Transformer or Conformer encoder, a CTC decoder, and an attention decoder in a joint architecture.The shared encoder uses limited right context to balance latency.
- Training: The combined training objective weights CTC and AED losses through λ: Lcombined (x, y) = λLCTC (x, y) + (1 −λ)LAED (x, y).Here, x is the acoustic feature, y is its label, and λ balances the two losses.
- Unified streaming and non-streaming operation: Dynamic chunk training lets the same model operate in streaming or non-streaming mode by varying chunk size from limited values to the utterance length.Each chunk attends to itself and all previous chunks, while latency in the first-pass CTC decoder depends on chunk size.
- Decoding: WeNet provides attention, CTC greedy, CTC prefix beam, and attention-rescoring decoding modes for research-stage comparison.These modes respectively use AED beam search, CTC greedy search, CTC prefix beam search, or CTC-generated n-best candidates followed by AED rescoring.
- Production runtime: Development runtime supports attention rescoring בלבד because the authors designate it as the ultimate production solution.It first generates n-best candidates with CTC prefix beam search and then rescores them with the attention decoder.
2.2. System Design
WeNet’s system design uses a PyTorch-based stack spanning research development, model export and quantization, production runtime, and deployment platforms. Its workflow also integrates training, decoding, validation, and production-hosting tools.
- System stack: WeNet’s design stack is fully based on PyTorch and its ecosystem.TorchScript, Torchaudio, distributed training, JIT export, quantization, and LibTorch support the research-to-production workflow.
- Training: Training uses on-the-fly feature extraction, joint CTC/AED training, and distributed multi-GPU execution.These components support augmentation from raw PCM, faster convergence and stability, and higher linear speedup across workers and GPUs.
- Validation: Python recognition and accuracy tools support different decode modes for model validation and debugging before production deployment.The tools support all decoding algorithms described in Section 2.1.2.
- System stack: TorchScript enables direct model export through torch JIT and runtime hosting with LibTorch.Both float-32 and quantized int-8 models are supported.
- System stack: Quantized models can double inference speed or improve it further on ARM-based Android and iOS devices.The passage describes this as a hosting benefit for embedded devices.
- System stack: WeNet supports x86 server and Android on-device runtimes with C++ APIs and runnable demos.Users can also build customized systems with the provided C++ library.
3. Experiments
Experiments evaluate WeNet’s unified model on AISHELL-1, including decoding modes, quantization, runtime efficiency, latency, and an industry-scale Mandarin dataset. Results support dynamic chunking with attention rescoring as a practical production configuration.
- Unified model evaluation: AISHELL-1 experiments compare a non-streaming full-attention baseline M1 with unified model M2 decoded using full, 16, 8, or 4-frame chunks.Full attention represents non-streaming inference, while limited chunk sizes represent streaming inference.
- Unified model evaluation: The unified model achieves comparable full-attention results and promising streaming results across chunk sizes 16, 8, and 4.The authors attribute this to the dynamic chunk training strategy.
- Unified model evaluation: Attention rescoring improves CTC results in both non-streaming and unified modes, while smaller chunks substantially degrade greedy and prefix-beam CTC performance.Attention rescoring also alleviates degradation of CTC prefix beam search as chunk size decreases.
- Unified model evaluation: Attention rescoring is faster and has better RTF than autoregressive attention decoding, so the unified model with rescoring is selected for production runtime.Only attention rescoring is supported at runtime.
- Runtime benchmarks: Quantization preserves comparable CER, provides about 2 times speedup on Android, and yields a slight improvement on x86.The CER comparison uses runtime tools; the float-model CER differs slightly from Python-tool results.
- Runtime benchmarks: RTF increases as chunk size decreases because smaller chunks require more forward-computation iterations.RTF and latency benchmarks cover server x86 and on-device ARM Android platforms.
- Runtime benchmarks: Latency benchmarks model WebSocket streaming on x86, with final latency dominated by rescoring cost and increasing slightly from chunk sizes 4 to 8 to 16.Rescoring cost is nearly invariant to chunk size, so reducing it can further reduce final latency.
- Industry-scale evaluation: A 15,000-hour Mandarin dataset evaluates U2 on three test sets using a Conformer shared encoder and Transformer decoder.The Conformer uses causal convolution for chunk training and adds three-dimensional pitch features to 80-dimensional FBANK features.
4. Conclusions
WeNet is an open-source, production-oriented E2E speech recognition toolkit offering unified streaming and non-streaming applications, with broad system support and reported benchmark performance.
- WeNet provides a unified solution for streaming and non-streaming E2E speech recognition applications.The toolkit is open source, lightweight, and evaluated on both an open dataset and an internal large dataset.