Source-linked AI summary
Pisets: A Robust Speech Recognition System for Lectures and Interviews
Ivan Bondarenko, Daniil Grebenkin, Oleg Sedukhin, Mikhail Klementev, Roman Derunets, Lyudmila Budneva
TL;DR
Pisets addresses reliable transcription for lectures, interviews, and other recordings where Whisper can produce errors and hallucinations. It combines Wav2Vec2 segmentation, AST filtering, and Whisper transcription, with uncertainty modeling and Russian-focused training. The resulting architecture is reported to reduce errors across varied acoustic conditions, while homophones and phonetically similar words remain a limitation.
Problem
Whisper-associated errors and hallucinations limit reliable speech recognition for scientists, journalists, and recordings processed across varied acoustic conditions.
Method
Pisets combines Wav2Vec2 speech segmentation, AST false-positive filtering, Whisper transcription, curriculum learning, and uncertainty modeling.
Results
The three-component architecture significantly reduces errors and hallucinations across various acoustic conditions and produces high-quality transcription in multiple scenarios.
Takeaways & Limitations
Pisets is reported as a reliable system for high-quality transcription in automatic dictation and conversational AI systems.
Takeaways & Limitations
The system currently performs insufficiently on homophones and words or phrases with similar phonetic characteristics.
Abstract
from arXiv · showhide
This work presents a speech-to-text system "Pisets" for scientists and journalists which is based on a three-component architecture aimed at improving speech recognition accuracy while minimizing errors and hallucinations associated with the Whisper model. The architecture comprises primary recognition using Wav2Vec2, false positive filtering via the Audio Spectrogram Transformer (AST), and final speech recognition through Whisper. The implementation of curriculum learning methods and the utilization of diverse Russian-language speech corpora significantly enhanced the system's effectiveness. Additionally, advanced uncertainty modeling techniques were introduced, contributing to further improvements in transcription quality. The proposed approaches ensure robust transcribing of long audio data across various acoustic conditions compared to WhisperX and the usual Whisper model. The source code of "Pisets" system is publicly available at GitHub: https://github.com/bond005/pisets.
1 Introduction
Pisets is an offline speech recognition system designed to improve accuracy while reducing Whisper-associated errors and hallucinations. Its pipeline combines Wav2Vec2 segmentation, AST false-positive filtering, and Whisper transcription, supported by curriculum learning and diverse Russian speech data.
- Pisets combines Wav2Vec2 primary recognition, AST false-positive filtering, and Whisper final transcription in a three-component architecture.
- Wav2Vec2 identifies speech-containing segment boundaries using contextual information from large-scale audio training.
- Curriculum learning progressively introduces accents, background noise, and varied acoustic conditions after simpler annotated data.
- AST analyzes acoustic signals to reduce non-existent speech fragments, particularly in noisy or acoustically complex conditions.
- BIRM and a speech environment built from diverse, carefully annotated audio were used to improve Whisper recognition outcomes.
- The architecture forms a transformation chain from initial recognition to final output that reduces errors and hallucinations.
2 Related Works
Whisper unified acoustic and linguistic processing but retains hallucination and latency limitations. Pisets addresses these limitations with Wav2Vec2-based VAD, post-recognition AST filtering, and verification against Wav2Vec2 output.
- Whisper generates coherent punctuated text directly from audio but can produce hallucinations and incur latency through autoregressive decoding.
- WhisperX improves speed and accuracy through non-autoregressive decoding, constrained beam search, VAD, and parallel chunk processing.
- Pisets uses Wav2Vec2 VAD for more nuanced acoustic analysis than WhisperX’s fixed-threshold approach.
- Unlike WhisperX, Pisets applies AST filtering after initial recognition to validate segments before Whisper’s final transcription.
- Comparing Whisper transcription with Wav2Vec2 output provides a verification step absent in WhisperX for mitigating inaccuracies.
3 Uncertainty modeling
Pisets models transcription uncertainty using token probabilities, disagreement between pipeline stages, and disagreement between Whisper runs on original and stretched audio. These signals support highlighting or rejecting dubious transcription segments.
- Uncertainty scores can highlight doubtful places for manual correction without requiring review of the entire transcription.
- Uncertainty-based refusal to transcribe hard-to-hear phrases can prevent incorrect text from disrupting later LLM summarization or harming reputation.
- The evaluated uncertainty methods use Whisper token probabilities, Whisper–Wav2Vec2 disagreement, and disagreement between Whisper predictions on original and stretched audio.
- Wav2Vec2 predictions can support uncertainty modeling in addition to segmenting long audio, while its preprocessing can enable parallel Whisper execution.
- Transcription disagreement is analyzed by aligning outputs, locating insertions, deletions, and replacements, then splitting or merging differences for linguistic matching.
- Optional heuristics and language-model validation reduce differences between base and additional model transcriptions.
- For word-based uncertainty, token probabilities are aggregated with minimum, sum, or mean operations; minimum and sum perform better than mean.
- Whisper tokenization distributes a spoken word’s probability across alternative token sequences, a distinction not incorporated into the current method.
4 Experiments
The experiments evaluate lexical and semantic assessment, long Russian lecture recordings under varied noise, Pisets configurations against WhisperX, and uncertainty modeling. Pisets provides higher recognition quality than WhisperX, while uncertainty scores identify many errors among a small marked word fraction.
- Evaluation criteria: The evaluation combines lexical WER with semantic BERT score (F1) because WER may miss semantically accurate but lexically different output.The paper recommends both measures for a more complete assessment of modern autoregressive speech recognition systems.
- Test data: Seven 20–40-minute Russian scientific lecture recordings span diverse lexical and speech domains, with quiet lecture-hall audio and added speech-like or musical noise at 1 dB SNR.The recordings cover subjects including philology, mathematics, and history.
- ASR comparison: Pisets provides higher recognition quality compared to WhisperX, while Whisper-Podlodka slightly trails Whisper-Large under favorable acoustics but gains advantages as background noise increases.These comparisons are reported for the long-audio lecture testing setup.
- Uncertainty metrics: The uncertainty evaluation aligns predicted words with ground truth, labels incorrect words from replacement or deletion operations, and treats uncertainty assessment as binary classification.Uncertainty ratio measures marked predicted words, while error-detection recall measures incorrect predicted words marked uncertain.
- Caveats: The uncertainty calculations exclude ground-truth words that the model failed to predict, and the test set is limited in size.The authors note that predicting only a small set of confident words could artificially improve uncertainty metrics while worsening WER.
- Uncertainty results: About 5% of words marked uncertain capture 35% of all errors, excluding errors caused by missed words in transcription.Across seven long Russian audios, model-disagreement and ensemble methods do not consistently outperform Whisper scores as uncertainty sources.
5 Conclusion
The conclusion presents Pisets as a framework combining segmentation, false-positive filtering, and final transcription to address recognition errors, hallucinations, and acoustic-linguistic variability. The resulting system is reported as robust across varied scenarios, while future work targets broader language adaptation and uncertainty handling.
- Conclusion: Pisets combines Wav2Vec2 segmentation, AST false-positive filtering, and Whisper transcription to address hallucinations, domain adaptability, and acoustic-linguistic variability.The framework also integrates diverse Russian speech corpora and BIRM fine-tuning for unfamiliar domains.
- Conclusion: Advanced uncertainty modeling techniques provided practical recommendations for improving transcription quality.The conclusion links these enhancements to the development of the Pisets system.
- Conclusion: The resulting system is described as capable of high-quality transcription in scenarios including automatic dictation and conversational AI systems.
- Future work: Future work will expand uncertainty handling and adaptation to multilingual datasets, including English speech by non-native speakers, Bengali, and Spanish.
6 Limitations
The system has insufficient performance on homophones and words or phrases with similar phonetic characteristics.
- Limitations: Pisets currently performs insufficiently on homophones and words or phrases with similar phonetic characteristics.The authors plan to incorporate pragmatic context through large multimodal models such as Qwen-Audio.
A Dictation mistakes overview
The system was evaluated during Russia’s “Total Dictation” event in a crowded classroom, combining clear narration with substantial background noise and reverberation.
- On April 20, 2024, the ASR system participated in Russia’s “Total Dictation” event, where thousands write text read by a narrator.The evaluation involved the system alongside other writers.
- The dictation took place in a 200-person classroom with a professional philologist reading into a microphone.The narrator pronounced the text clearly and loudly, favoring recognition.
- More than a hundred participants generated conversations, movement, coughing, and paper noise that hindered recognition.The auditorium’s high reverberation further reduced speech audibility.
A.2 Linguistic Conditions of the Text
The dictation used accessible Russian prose and a demanding multi-pass delivery format, requiring the system to separate target text from spoken instructions and comments.
- A.2 Linguistic Conditions of the Text: The Russian text used a free, conversational style, common vocabulary, and a clear paragraph structure.Its subject was diaries and their role in a person’s life.
- A.2 Linguistic Conditions of the Text: The narrator read the text fully, repeated sentences quickly, dictated slowly by parts, and repeated complete sentences quickly.Some parts were repeated at listeners’ requests.
- A.2 Linguistic Conditions of the Text: Additional comments and paragraph announcements created a requirement to distinguish the main text from extraneous speech.The announcements indicated sentence and paragraph boundaries.
- A.2 Linguistic Conditions of the Text: A Longformer model was trained to detect out-of-context and line-break insertions during postprocessing.It did not remove enough insertions but split the text into paragraphs correctly.
A.3 Typology of model mistakes
The evaluation found relatively strong spelling, punctuation, and paragraph handling, alongside omissions, boundary errors, and sound-based word substitutions that sometimes altered meaning.
- A.3 Typology of model mistakes: Two spelling and three punctuation errors were recorded.The spelling errors involved noun and adjective endings, while punctuation errors involved several construction types.
- A.3 Typology of model mistakes: Eight words out of 276 were omitted at sentence ends, often disrupting sentence structure.The system also failed to add a full stop before starting the next sentence with a capital letter.
- A.3 Typology of model mistakes: The system ignored two instances of narrator-indicated parceling and combined sentences despite placing punctuation correctly.Such combinations would not have counted as errors for other writers.
- A.3 Typology of model mistakes: Eight mishearings replaced words with acoustically similar forms that were usually far from the originals in meaning.Examples included substitutions such as “чернилами” for “черепами” and “свадьбой” for “спать”.
- A.3 Typology of model mistakes: The model failed to recognize the construction “читай – не хочу”, recording it as “Считай, не хотите”.
- A.3 Typology of model mistakes: A sentence-level mishearing changed “и так смело” to “и та смело”, violating the sentence’s content.The passage notes that a similar error was common among other writers.
- A.3 Typology of model mistakes: The system handled spelling, punctuation, repetitions, and paragraph placement well, but sometimes omitted or replaced sentence-final words.Experts rated the system B (“good”), while many human writers received F.
- A.3 Typology of model mistakes: Tables 3 and 4 compare ASR pipeline configurations on noisy and clean audio.
C Testing computational efficiency
The computational-efficiency test found that Wav2Vec2-based “smart” chunking was faster at inference than Whisper’s uniform chunking.
- C Testing computational efficiency: Wav2Vec2 “smart” chunking outperformed the original Whisper model’s uniform chunking in inference time.
D Uncertainty places in final transcription
The system highlights dubious transcription places using uncertainty estimation based on model disagreement. Comparative tables report ASR pipeline configurations across quiet-noise, loud-noise, and noised-audio settings, including timing results.
- Model disagreement uncertainty estimation highlights dubious places in the final transcription.Figure 3 illustrates this highlighting approach.
- Tables 3 and 4 compare ASR pipeline configurations for quiet-noise and loud-noise audio.
- Table 5 reports ASR pipeline configuration times in seconds for noised audio.