Source-linked AI summary
Automatic Speech Recognition for Multilingual Oral History Research
Sidney Wong, Chelsea Wong She, Eda Tang, Tiana Marshall Wong, Debbie Sew Hoy, Chelsea Wong
TL;DR
Cantonese oral histories are important for community-led language revitalisation, but code-switching creates an understudied challenge for ASR transcription. This paper evaluates Whisper on multilingual oral-history recordings and finds that it can produce a rapid first-pass transcript, while non-English segments remain difficult to transcribe accurately.
Problem
Evidence is limited on how effective ASR toolkits are for code-switched oral-history recordings containing Cantonese, English, Taishanese, and te reo Māori.
Method
The study evaluated Whisper on a community-led corpus of multilingual oral-history recordings and examined both model metrics and predicted outputs.
Results
The language-unspecified LARGE-V3 achieved a WER of 12.10, while model outputs often transliterated or mishandled non-English segments.
Takeaways & Limitations
Whisper is useful for producing first-pass transcriptions, with English-language segments suitable for this purpose and non-English segments requiring correction and editing.
Takeaways & Limitations
The evaluation used a limited reference set, and Whisper struggled to distinguish and transcribe Taishanese and other non-English segments accurately.
Abstract
from arXiv · showhide
This paper offers a unique perspective on how speech technologies are being adopted by community-led heritage language preservation and revitalisation initiatives. As a community-led language maintenance strategy, oral histories play a crucial role in Cantonese language revitalisation in New Zealand. The development of Automatic Speech Recognition (ASR) toolkits, such as Whisper, have expedited what has often been a resource and time-intensive process of transcribing oral history collections. However, there is limited research into the effectiveness of ASR toolkits when applied to code-switched language contexts. Based on Word Error Rate (WER), the best performing Whisper model configuration achieved a WER of 12.10 at the expense of accurately transcribing unsupported non-English segments. However, Whisper remains a useful tool by providing a first-pass transcription using only 1% of the estimated time otherwise needed for manual transcription.
1. Introduction
Cantonese is a large, established heritage language community in New Zealand, but sustained language shift has weakened transmission. Community-led oral histories help preserve linguistic and cultural knowledge, while transcription barriers limit access to these recordings.
- 54,417 speakers made Cantonese New Zealand’s seventh most spoken language in the 2023 Census.
- 70.96% of people in surveyed Wellington Cantonese-speaking families no longer spoke Cantonese across three generations.
- Legislative discrimination, language-teacher bans, social exclusion, and assimilation pressures contributed to Cantonese language shift.
- Community-led oral histories document lived experiences and place communities in control of the narratives produced during interviews.
- Oral histories preserve linguistic and cultural knowledge, but translating and transcribing them remains difficult and limits access to Cantonese-speaking elders’ recordings.
2. Related Works
ASR has become an important tool for language documentation and revitalisation, evolving from statistical systems to transformer-based toolkits such as Whisper. Researchers use these systems to create first-pass transcripts that speakers can correct, including for multilingual oral histories.
- ASR transforms speech into word sequences and is used to support documentation and revitalisation in indigenous and heritage language communities.
- In under-resourced languages, ASR primarily produces first-pass transcripts that speakers then correct and edit.
- Open-source ASR toolkits such as KALDI developed from statistical language models and later adopted deep neural network architectures.
- Whisper has been applied to multilingual oral-history transcription and Cantonese ASR, but Cantonese transcription involves more than speech-to-text.
- Transformer-based ASR systems treat speech-to-text as a sequence-to-sequence task, accelerating neural-network ASR development.
3. Methodology
The paper investigates how oral historians can use ASR toolkits in practice, focusing on a community-led corpus of interviews with New Zealand Chinese elders proficient in English and Cantonese-related varieties.
- The study addresses how oral historians can use ASR toolkits and examines their limitations in an applied real-world context.
- The researchers created a corpus of oral-history interviews with community elders proficient in English, Cantonese, and related varieties including Taishanese.
- The research team included members of New Zealand Chinese communities and heritage-language speakers of Cantonese.
3.1. Data Collection and Corpus Dimensions
The corpus comprises interviews with older first- and second-generation New Zealand Cantonese speakers, recorded by bilingual project members and designed to permit multilingual speech. Participants also consented to archival and educational uses of the recordings.
- Two trained project members conducted the interviews, using English and varying levels of Cantonese proficiency.
- The corpus included five speakers aged 65 or older who were first- or second-generation New Zealand Cantonese born or migrated before 1980.
- The five-speaker corpus comprised two women and three men.
- Interviews began with 20–30 minutes of demographic questions, and participants were encouraged to use any language.
- Participants consented to uses including podcasts, teaching resources, documentaries, and publications, with recordings intended for a public archive.
3.2. Speech Recognition
The study tested Whisper on a community-led multilingual oral-history corpus containing English, Cantonese, Taishanese, and te reo Māori code-switching. Evaluation used a manually transcribed reference set and multiple ASR error metrics.
- Corpus and data preparation: The pilot corpus contained 12 hours and 42 minutes of multilingual oral-history audio, estimated to require 508 hours for manual transcription.The estimate excludes time needed to translate between language conditions.
- Corpus and data preparation: The researchers combined dual-microphone recordings into one audio file using PYDUB after manipulating audio with FFMPEG.
- Pretrained models: The evaluation included eight Whisper models ranging from tiny with 39 million parameters to large with 1,550 million parameters, plus English-specified and language-unspecified conditions.The language-unspecified condition used Whisper’s internal language detection to determine the primary language.
- Evaluation: Word Error Rate measures word differences between reference and prediction, while CER, MER, WIP, and WIL capture character, alignment, and information-preservation differences.WER was the primary ASR performance measure.
- Evaluation: The reference set was a 9-minute-33-second recording containing 1,198 words and code-switching among English, te reo Māori, Cantonese, and Taishanese.Its development exposed differing orthographic conventions between English and Sinitic languages.
4. Results
Whisper performance varied substantially by model size and language condition, with large models generally outperforming smaller ones. Although language-unspecified LARGE-V3 achieved the best WER, qualitative inspection showed that models often transliterated or failed to distinguish non-English code-switched speech.
- Language-unspecified models: 12.10 WER made language-unspecified LARGE-V3 the best-performing configuration, while SMALL recorded the worst WER at 40.65%.
- Language-unspecified models: Six of eight language-unspecified configurations detected te reo Māori as the primary language, whereas BASE and TURBO detected English.
- Qualitative inspection: More than half of the models transliterated non-English segments, and all models failed to distinguish phonetic differences between Cantonese and Taishanese.Some outputs also contained hallucinations, while certain models used modified Hànyǔ Pīnyīn rather than Jyutping.
5. Discussion
Whisper performed well on aggregate transcription metrics, but code-switched oral histories exposed substantial weaknesses in recognizing and transliterating non-English segments. The evaluation therefore supports Whisper as a rapid first-pass tool, with manual effort still required for multilingual correction and verification.
- 12.10 WER was achieved by the language-unspecified LARGE-V3 configuration, alongside 8.73 CER, 11.89 MER, and 84.39 WIP.Its WER fell within reported English and Hong Kong Cantonese benchmark ranges, but the configuration still struggled with non-English segments.
- Whisper configurations struggled to detect and transliterate non-English segments in recordings containing English, Cantonese, Taishanese, and te reo Māori.The limitation concerns both identifying non-English speech and rendering it accurately.
- Taishanese segments produced limited transliteration success, including hallucinations, because the model failed to distinguish phonetic differences from Cantonese.The excerpt also shows that Cantonese–Taishanese boundaries can be unclear even to the speaker.
- The evaluation used a limited reference set covering only a small part of the full corpus, prioritising ecological validity over scale.This constrains how broadly the findings should be generalized.
- Whisper is useful for first-pass transcription, but this corpus still requires correction and editing of non-English segments.The authors report that the tool meets oral historians’ needs when used as an initial transcription rather than a finished record.
6. Conclusion
The paper demonstrates that Whisper can support heritage language revitalisation by rapidly transcribing oral history collections. Its practical value is strongest as a first-pass system, while code-switched multilingual speech remains a target for improvement and community-guided refinement.
- Whisper was applied across the entire corpus to support heritage language revitalisation through automatic transcription.The paper presents this use of existing state-of-the-art language technology as its primary contribution.
- Five hours transcribed 12 hours and 42 minutes of audio end-to-end, taking less than 1% of the estimated manual transcription time.The estimate excludes time needed to verify the transcripts.
- Improving code-switched multilingual speech performance may require fine-tuning with existing data and consultation with speakers and the wider community.The paper also stresses that first-pass outputs require additional correction and editing.