Source-linked AI summary
OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee-weon Jung, Shinji Watanabe
TL;DR
Whisper’s full development pipeline is unavailable publicly, and previous OWSM models use a Transformer encoder. OWSM v3.1 uses E-Branchformer at multiple scales without additional training data; it outperforms OWSM v3 on most evaluated sets and offers faster inference.
Problem
Whisper’s full development pipeline, including training-data details and learning dynamics, is unavailable publicly, while OWSM v3 retains a Transformer encoder.
Source: §1 · p. 1, §2.1 · p. 1Method
The authors build E-Branchformer-based OWSM v3.1 models at three scales using the same amount of training data as OWSM v3.
Source: §1 · p. 1, §2.1 · p. 1Results
Takeaways & Limitations
The work provides a low-restriction model and plans public releases of code, pretrained weights, and training logs to promote transparency.
Source: §4 · p. 4The work does not improve training-data quantity or quality, which might lead to suboptimal performance in low-resource languages.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Recent studies have highlighted the importance of fully open foundation models. The Open Whisper-style Speech Model (OWSM) is an initial step towards reproducing OpenAI Whisper using public data and open-source toolkits. However, previous versions of OWSM (v1 to v3) are still based on standard Transformer, which might lead to inferior performance compared to state-of-the-art speech encoder architectures. This work aims to improve the performance and efficiency of OWSM without additional data. We present a series of E-Branchformer-based models named OWSM v3.1, ranging from 100M to 1B parameters. OWSM v3.1 outperforms its predecessor, OWSM v3, in most evaluation benchmarks, while showing an improved inference speed of up to 25%. We further reveal the emergent ability of OWSM v3.1 in zero-shot contextual biasing speech recognition. We also provide a model trained on a subset of data with low license restrictions. We will publicly release the code, pre-trained models, and training logs.
1. Introduction
Open Whisper-style models address the lack of public access to Whisper’s full development pipeline, while earlier OWSM versions retain a standard Transformer encoder. OWSM v3.1 replaces that encoder with E-Branchformer and improves performance and inference efficiency using the same amount of training data.
1. Introduction
- The full development pipeline behind Whisper, including training-data details and learning dynamics, is unavailable publicly, raising concerns about data leakage, fairness, and bias.OWSM is an open reproduction effort using public datasets and an open-source toolkit.Source: §1 · p. 1
- E-Branchformer-based OWSM v3.1 targets better performance and efficiency than OWSM v3 without additional training data.The authors select E-Branchformer after preliminary encoder comparisons and present models from 101M to 1.02B parameters.Source: §1 · p. 1
- OWSM v3.1 outperforms OWSM v3 on 8 of 9 English ASR, 10 of 11 multilingual ASR, 13 of 19 speech translation, and 3 of 4 SLUE-PERB test sets, with inference speedups up to 25%.The paper also reports an improved English-ASR performance–efficiency trade-off against Whisper and zero-shot contextual-biasing ASR ability.Source: §1 · p. 1
2. OWSM v3.1
OWSM v3.1 replaces the Transformer encoder with E-Branchformer and offers three model scales. Its training setup includes transcript preprocessing, a low-restriction data subset, and a piecewise-linear warmup schedule for large-model training.
2.1. Model architecture
- E-Branchformer enhances Branchformer with parallel branches for local and global information, merged by convolutions; OWSM v3.1 provides 101M, 367M, and 1.02B variants.The models otherwise mostly follow OWSM v3’s design, and their smaller decoder contributes to faster inference at the same scale.Source: §2.1 · pp. 1–2
2.2. Data preparation
- Training data uses scripts from OWSM v3, with WSJ excluded and selected transcripts lowercased; the main models use 180K hours, while the low-restriction model uses 70K hours of ASR data and no ST data.The low-restriction subset comprises datasets with low license restrictions; “cmn” and “zho” language codes are merged as “zho”.Source: §2.2 · p. 2
2.3. Training setups
- A piecewise-linear warmup first raises the learning rate to a small value, then to its peak, before exponential decay; the authors report it enables successful OWSM v3.1 training.The two warmup stages each last 30K steps, with the first reaching an example value of 5e-5.Source: §2.3 · p. 2
3. Experiments
The encoder comparison favors E-Branchformer for faster convergence, and English ASR results show broad gains over OWSM v3 alongside faster inference. The convergence comparison is limited to the first 105K steps.
3.1. Comparison of encoder architectures
- E-Branchformer converges faster than Transformer and Conformer in a comparison of similarly sized models trained on 10% of the data, so the authors use it in the main experiments.The comparison tracks validation loss over the first 105K steps.Source: §3.1 · p. 2
3.2. English speech recognition
- OWSM v3.1 medium performs better than OWSM v3 on 8 of 9 English ASR test sets, achieves lower average WER than Whisper at each scale despite less English ASR data, and is faster than peers at the same scale.The models use the Branchformer encoder; the smaller decoder is identified as the primary reason for faster inference.Source: §3.2 · pp. 2–3
- The encoder convergence comparison covers only the first 105K steps because of budget and time limits, before full convergence.The authors report that full convergence takes more than a week with 16 GPUs.Source: §3.2 · p. 2
- Table 3 reports multilingual ASR WER/CER and training-data sizes, while Table 4 reports X-to-English speech-translation BLEU and training-data sizes.Both captions state that OWSM v3.1 uses the same amount of training data as OWSM v3.Source: Table 3 · p. 3, Table 4 · p. 3
3.3. Multilingual speech recognition
OWSM v3.1 medium improves multilingual ASR over OWSM v3 in most test sets, but performance relative to Whisper varies with available training data.
3.3. Multilingual speech recognition
- 18.8% to 15.2%: OWSM v3.1 medium reduces average multilingual ASR error rate and outperforms OWSM v3 in 10 of 11 test sets.It still trails Whisper in many European languages, but performs strongly on Chinese and Japanese when training data are sufficient.Source: §3.3 · p. 3
- Limited training data is associated with weaker results in many European languages, while sufficient data supports strong Chinese and Japanese performance.The authors plan to add more public-source data, such as YODAS, to improve OWSM.Source: §3.3 · p. 3
3.4. Speech translation
Speech translation evaluation covers selected CoVoST-2 directions, with OWSM v3.1 improving on OWSM v3 in both reported direction groups and decoding faster.
3.4. Speech translation
- Evaluation uses all 15 English-to-X directions, but reports X-to-English only where OWSM had more than 100 training hours.Directions with very limited training data, such as Japanese- or Chinese-to-English, usually do not work for OWSM; decoding time is also recorded.Source: §3.4 · p. 3
- 13.0 to 13.3: English-to-X average BLEU improves with OWSM v3.1, which beats OWSM v3 in 9 of 15 directions and decodes 25% faster.Whisper cannot perform translation in these English-to-X directions.Source: Table 5 · p. 3, §3.4 · p. 3
- 18.8 to 20.1: OWSM v3.1 medium raises average X-to-English BLEU over OWSM v3 and is 16% faster at inference.It remains worse than Whisper, though faster at inference at each scale; the authors attribute the speed advantage to a 40 ms rather than 20 ms encoder time shift and a smaller decoder.Source: §3.4 · p. 3
3.5. Long-form speech recognition
On long-form TEDLIUM English ASR, OWSM v3.1 medium substantially lowers WER relative to OWSM v3, while remaining behind Whisper amid data and training-inference differences.
3.5. Long-form speech recognition
- Long-form inference takes an entire recording and generates transcript chunks of fixed 30-second length, shifted based on predicted timestamps.OWSM v3.1 still trails Whisper, which the authors link to its smaller training-data volume and the mismatch from training on segmented short audio.Source: §3.5 · pp. 3–4
- 5.7% versus 9.2%: OWSM v3.1 medium achieves lower WER than OWSM v3 on long-form TEDLIUM ASR.The authors say this demonstrates robustness to long-form audio, and suggest predicted timestamps might also be more accurate.Source: §3.5 · pp. 3–4
- The authors plan to add more long-form training data to mitigate the mismatch between segmented short-audio training and long-form inference.Many public datasets used by OWSM do not provide unsegmented long-form data.Source: §3.5 · p. 4
3.6. Language identification
OWSM v3.1 language-identification accuracy on FLEURS degrades relative to OWSM v3, but improves markedly with model scaling and exceeds Whisper medium at the medium scale.
3.6. Language identification
- 41.9% to 75.6%: OWSM v3.1 accuracy rises from base to medium on FLEURS, while Whisper accuracy rises from 47.6% to 54.8%.The authors report OWSM v3.1 medium is much better than Whisper medium despite a degradation relative to OWSM v3.Source: §3.6 · p. 4
- OWSM v3.1 benefits more from scaling than Whisper on FLEURS language identification, possibly because OWSM supports more ASR languages and speech-translation language pairs.The authors suggest this broader task and language coverage is more challenging for smaller models to learn.Source: §3.6 · p. 4
3.7. Spoken language understanding via fine-tuning
The section evaluates OWSM through fine-tuning on four spoken language understanding tasks. OWSM v3.1 medium outperforms OWSM v3 by a large margin on NER, NEL, and DAC.
3.7. Spoken language understanding via fine-tuning
- OWSM v3.1 medium outperforms OWSM v3 by a large margin on NER, NEL, and DAC.Source: §3.7 · p. 4
- The evaluation freezes the pre-trained speech encoder and trains a randomly initialized shallow decoder on task-specific SLU data.Source: §3.7 · p. 4
- The four evaluated tasks are sentiment analysis, named entity recognition, named entity localization, and dialog act classification.Source: §3.7 · p. 4
3.8. Emergent ability for zero-shot contextual biasing
The section tests zero-shot contextual biasing by prompting ASR with biasing words. Small and medium OWSM v3.1 models reduce biased WER while mostly maintaining unbiased WER, unlike the base model.
3.8. Emergent ability for zero-shot contextual biasing
- During training, the previous sentence in the recording serves as a prompt with probability 0.5; users may provide a prompt during inference.Source: §3.8 · p. 4
- Small and medium models greatly reduce B-WER and mostly maintain U-WER, while the base model has only minor B-WER improvement and much larger U-WER degradation.Source: §3.8 · p. 4, Table 9 · p. 4
- The test uses 100 biasing words as a space-separated prompt with greedy decoding and no text normalizer.Source: §3.8 · p. 4
4. Conclusion and future work
OWSM v3.1 generally outperforms OWSM v3 across evaluation sets and offers faster inference despite using the same amount of training data. The conclusion also notes a low-resource-language limitation and identifies directions for future work.
4. Conclusion and future work
- OWSM v3.1 achieves better results than OWSM v3 in the vast majority of evaluation sets while showing up to 25% faster inference.The comparison uses the same amount of training data.Source: §4 · p. 4
- A model trained on a subset of data with low license restrictions is provided to extend model accessibility.Source: §4 · p. 4
- The work does not improve training-data quantity or quality, which might lead to suboptimal performance in low-resource languages.Source: §4 · p. 4
- Future directions include studying data diversity, adding public data such as YODAS, model compression, and further downstream applications.Source: §4 · p. 4