Source-linked AI summary
Whisper-Based Speech Transcription from Videos Across Multiple Languages for Cross-Cultural Understanding
Michael Picheny
TL;DR
Cross-cultural understanding tools need usable transcripts from naturally produced multilingual video speech, but realistic audio remains difficult for open-source recognizers. The paper develops an accessible Whisper-based workflow and evaluates fine-tuning for seven languages. Average error rates were about 30% out of the box and could fall to close to 20% with modest fine-tuning data, while associated speech and metadata were released for further research.
Problem
Naturally produced multilingual video speech remains challenging to transcribe accurately for cross-cultural understanding tools.
Method
The paper uses accessible Whisper and WhisperX tools with a practical video-processing workflow, fine-tuning, and community data release.
Results
Average transcription error decreased from about 30% to close to 20% with modest fine-tuning data across the investigated languages.
Takeaways & Limitations
The methodology enables non-speech researchers to generate more usable transcripts and provides released data for further improvement.
Abstract
from arXiv · showhide
Cross-cultural understanding has become increasingly important in today's highly connected, cross-national world. The success of LLM-based technologies is now driving the development of automated tools to aid understanding for nonnative people trying to succeed in cross-cultural environments. Building such automated tools is often done by leveraging in-thewild text, audio, and video data. This paper presents techniques for improving speech recognition-based transcript creation in multiple languages from videos to better train these automated tools. The focus is on processes and speech tools that can easily be used by cross-cultural tool builders without requiring deep speech processing expertise. Using publicly available videos from YouTube and Whisper-based tools, average transcription error rate across seven languages (Spanish, Japanese, Korean, Mandarin, Turkish, Russian, and Hebrew) of 30% are observed. With a modest amount of fine-tuning data, the average error rate can be reduced to 20% making such output much more usable for downstream processing. Speech and metadata associated with these videos that can be used by the community to further refine these experiments are released as well.
I. INTRODUCTION
The paper targets reliable multilingual transcription of naturally produced video speech for cross-cultural tools. It evaluates accessible Whisper-based processes intended for researchers without deep speech-processing expertise.
- Cross-cultural tools need accurate transcripts from naturally produced, multilingual audio and video rather than only stylized benchmark speech.Realistic speech remains challenging, especially for languages with less data.
- The study estimates open-source transcription accuracy, develops a practical collection and system-building process, and shares representative speech data.These goals address evaluation, accessibility, and further community improvement.
- WhisperX was selected alongside Whisper because it offers multilingual recognition, fine-tuning, speedups, speaker diarization, and an easy-to-use API.Diarization is useful for conversation analysis, while speed and cost support large-scale processing.
- The paper describes a reproducible workflow spanning recognizer choice, language selection, processing, video selection, experiments, discussion, and data release.
II. DATA PROCESSING
The study evaluates Whisper and WhisperX for multilingual video transcription, emphasizing accessible processing, speaker diarization, and speed for large video collections.
- Whisper was chosen for its easy-to-use API, multilingual performance, fine-tuning capabilities, and open-source availability.
- WhisperX combines speech recognition, speaker diarization, and a sped-up Whisper implementation for processing thousands of videos.Its diarization component supports identifying speaker turns in conversations.
- Both Whisper and WhisperX were evaluated using the large-v2 Whisper model.The paper notes that large-v3 might improve some languages but could be more prone to hallucinations on noisy data.
B. Language Data
The language set follows the DARPA CCU program, with Hebrew added as a lower-coverage challenge language. Available training-data coverage varied substantially across languages.
- The selected languages were Mandarin, Spanish, Korean, Japanese, Russian, Turkish, and Hebrew.
- Table I reports the size of Whisper training data for the selected languages.The supplied table passage provides its scope but not the individual language values.
- Speech-recognition performance could not be evaluated for the original CCU data because manual transcripts were not supplied.
- Six study languages came from DARPA CCU, while Hebrew was added because Whisper covered it with substantially less training data.Hebrew had 688 hours of Whisper training data, whereas each CCU language had over 4000 hours.
C. Processing Pipeline
The pipeline collects multilingual YouTube videos using language-aware search and licensing filters, then prepares caption-derived audio segments for transcription alignment and model development.
- Videos lacking sufficient qualifying material for Mandarin, Korean, Turkish, and Hebrew were processed from scratch.
- Table II consolidates language metadata by speech duration.The supplied table passage identifies the metadata scope but not individual duration values.
- Jtubespeech tools search YouTube using language-related words, subtitles, Creative Commons criteria, and time restrictions.The process uses Wikimedia title indexes and extracts whether subtitles are automatic or manually produced.
- The tools were modified to process multiple parallel batches and restrict search titles to language-appropriate UTF-8 character ranges.
- VTT files provide caption text and start-end times, but adjacent short segments often do not mark useful word or sentence boundaries.
- Caption segments were merged up to 20 seconds or until a 0.1-second silence gap to improve audio-word and phrase alignment.Short VTT timing boundaries were frequently inaccurate for these alignments.
D. Video Selection Process
The video-selection process assembled multilingual speech data from curated CCU files and YouTube crawling, using duration, licensing, and transcript-quality constraints. It also assumed that newly crawled videos were suitable for cultural-understanding research.
- Videos with WER above 50% were discarded, and audio was divided into training, development, and test files.The duration and split procedures were constrained by processing practicality and language-specific data availability.
- Data sources varied by language: Creative Commons CCU data were used when at least 10 hours were available, while other languages relied on additional YouTube crawling.Spanish, Russian, and Japanese used the original CCU data when possible; Mandarin, Korean, Turkish, and Hebrew required more YouTube data.
- The selection targeted videos longer than five minutes to increase the chance of capturing interactions and cultural markers.The authors noted that CCU required thousands of videos per language to obtain enough cultural markers for model training and evaluation.
III. EXPERIMENTS
The experiments compared out-of-the-box Whisper systems with fine-tuning using small and complete training-data settings across all seven languages.
- Three experiment sets evaluated out-of-the-box recognition, fine-tuning with small training sets, and fine-tuning with all available training data.Recognition used the small test set, while fine-tuning used the small development set for rapid experiment turnover.
A. Out-of-Box performance
Out-of-the-box Whisper and WhisperX performance generally produced 25%–30% error rates across languages, with WhisperX slightly ahead and faster. Language comparisons varied with the evaluation metric and data source.
- 25%–30% WERs were typical across languages, with WhisperX showing a slight performance edge and considerably faster processing than Whisper.These results summarize the out-of-the-box comparison shown in Figure 1.
- Mandarin had substantially lower error than the other languages, probably because it used CER, while Japanese CER remained high due largely to deletions.The metric difference limits direct comparison of Mandarin with languages evaluated using WER.
- No obvious overall WER difference appeared between CCU and non-CCU data or between Hebrew and the other languages.The balance of substitution, deletion, and insertion errors differed between CCU and non-CCU sources.
B. Performance after Fine-Tuning
Fine-tuning initially worsened average performance because of hallucination-driven insertions, but constrained decoding and complete-data training improved results. WhisperX ultimately reached the lowest reported average WER.
- Fine-tuning with the small data setting initially worsened performance because insertions increased, with hallucinations accounting for most of that increase.The authors suggest poor parameter tuning or the absence of more sophisticated adaptation schemes as possible explanations.
- 24.4% WER was obtained for WhisperX after optimized Whisper fine-tuning produced 26.0% WER, compared with 29.7% out of the box.No WhisperX parameters were tuned in this comparison.
- 21.7% average WER was achieved by fine-tuning WhisperX on all training data, improving over the 29.7% out-of-the-box Whisper result.Whisper results with complete-data fine-tuning were slightly worse and are not presented.
- All ten pairwise condition comparisons were significant at p < .001 except the out-of-box Whisper versus optimized Whisper comparison, which had p = .012.The authors interpreted the result as suggesting that 10 hours or less per language was too little for reliably large improvements.
- Hebrew showed no substantial performance difference from the other languages despite much lower Whisper training-data representation, while YouTube-harvested languages performed better than CCU languages.The authors tentatively attributed the latter pattern to stricter LDC screening producing more complex data.
IV. DISCUSSION
The study finds that open-source Whisper-based transcription of difficult multilingual speech can improve substantially with modest fine-tuning, but its adequacy for downstream cultural-marker extraction remains uncertain. Comparisons with prior NLP robustness findings suggest that the impact of transcription errors depends on the downstream component and language.
- Nearly 33% relative improvement lowered average transcription error from 30% to close to 20% using at most 200 hours of data for one language.The authors suggest larger gains may require substantially more training data, computing, and personnel resources.
- The study cannot directly establish whether 20% WER is adequate for accurate cultural-marker extraction because controlled datasets and open-source evaluation code are unavailable.Related NLP-component studies provide only a crude proxy for full cultural-marker-system evaluation.
- Prior evidence finds dialog-act classification tolerates 44%-71% WER increases, whereas summarization and question answering can degrade at much lower error levels.Sensitivity varies across models, with some remaining robust through WER ranges of 20%-30%.
- At approximately 30% WER, out-of-the-box Whisper or WhisperX may degrade downstream NLP performance, while fine-tuning toward 20% WER might approach perfect-transcription results.The authors characterize this expectation as highly speculative because effects may vary across languages and compound across multiple components.
- Reported WERs measure deviations from undocumented closed captions rather than true error against gold-standard transcripts.The authors note that lower error after fine-tuning and inference-parameter tuning suggests the transcripts are not completely inaccurate.
V. DATA RELEASE
The release provides speech data, transcriptions, and train/dev/test metadata, but licensing uncertainty and unavailable videos restrict the release to a subset of the collected material. Fine-tuning still produced significant improvements across all languages despite smaller released training sets.
- Only a subset of the collected data could be released because about half lacked explicit license information and some videos were no longer available.Spanish, Russian, and Japanese had little or no license information explicitly set in the video metadata.
- The release contains segmented speech data, transcriptions, and TSV files dividing examples into training, development, and test subsets.The data are provided after the described segmentation process.
- Significant fine-tuning improvements were observed for all languages even though the released training sets were smaller than those used in the original configurations.Table V summarizes released-data statistics and pre- versus post-fine-tuning error rates using Whisper large-v2 with optimized configurations.
VI. SUMMARY
The paper describes an accessible methodology for improving speech recognition in cross-cultural understanding research and releases multilingual data for future refinement.
- 30–200 hours of fine-tuning data produced substantial speech-recognition improvements with Whisper and WhisperX when manual transcriptions were available.The approach targets researchers without deep speech-recognition expertise.
- The methodology uses open-source data and tools to generate inputs for cross-cultural understanding and other natural-language training systems.
- 260 hours of Mandarin, Korean, Turkish, and Hebrew training, development, and test data were released for community research.The release supports future efforts to improve speech recognition for cultural-understanding research.