Source-linked AI summary
Fine-Tuning Whisper for Automatic Speech Recognition in Baniwa: A Preliminary Study
Leonardo Duart, Tiago Fonseca, Thiago Chacón
TL;DR
Indigenous and low-resource languages remain underserved by ASR because speech resources and language technologies are scarce. This study fine-tunes Whisper on a small manually transcribed Baniwa corpus and finds a best WER of 37.5% and CER of 7.45%, establishing an initial baseline for Baniwa ASR.
Problem
Low-resource and indigenous languages lack sufficient transcribed speech and modern ASR evaluation, including in the Amazon region.
Method
The study uses supervised fine-tuning to adapt Whisper to Baniwa with a manually transcribed corpus and standard ASR evaluation metrics.
Results
The best model achieved a Word Error Rate of 37.5% and a Character Error Rate of 7.45%.
Takeaways & Limitations
The results establish an initial Baniwa ASR baseline and demonstrate the potential of multilingual foundation models for indigenous language technologies.
Takeaways & Limitations
Performance is scoped to isolated-word and short-utterance recognition because the corpus primarily contains elicited speech rather than conversational speech.
Abstract
from arXiv · showhide
Automatic Speech Recognition (ASR) technologies have achieved remarkable performance in recent years through the use of large multilingual foundation models. However, most advances remain concentrated on high-resource languages, while indigenous languages continue to suffer from a lack of speech resources and language technologies. This work presents a preliminary study on the adaptation of Whisper for Automatic Speech Recognition in Baniwa, an indigenous Arawakan language spoken in Brazil, Colombia, and Venezuela. The experiments were conducted using a corpus of 1,373 manually transcribed recordings obtained from a linguistic documentation project. The corpus contains approximately 0.54 hours of speech and consists primarily of isolated words and short elicited utterances. The Whisper Small model was fine-tuned using supervised learning and evaluated using Word Error Rate (WER) and Character Error Rate (CER). The best model achieved a WER of 37.5% and a CER of 7.45%, demonstrating that multilingual foundation models can be successfully adapted to extremely low-resource indigenous languages. The results establish an initial baseline for Baniwa Automatic Speech Recognition and provide a foundation for future research involving larger datasets, language-specific adaptation strategies, and post-processing techniques.
1 Introduction
ASR advances have largely benefited high-resource languages, while indigenous languages such as Baniwa remain constrained by scarce resources. This study investigates whether Whisper can be adapted to Baniwa using limited manually transcribed speech.
- Indigenous and low-resource languages remain underrepresented in ASR despite major advances from deep learning and multilingual speech corpora.
- Baniwa is an indigenous Arawakan language spoken across Brazil, Colombia, and Venezuela.
- Fine-tuning large pre-trained speech models can reduce the language-specific data required for low-resource ASR.
- The study evaluates supervised Whisper fine-tuning on a manually transcribed Baniwa corpus using standard ASR metrics.
2 Related Work
Low-resource ASR is hindered by limited transcribed speech and linguistic infrastructure, motivating transfer learning with multilingual models. This study applies that approach to Baniwa, addressing limited evaluation of modern ASR models for Amazonian indigenous languages.
- Low-resource ASR faces limited transcribed speech, linguistic resources, and computational infrastructure.
- Transfer learning and multilingual training reduce the target-language annotation required for speech recognition systems.
- Whisper was trained on approximately 680,000 hours of supervised multilingual and multitask speech data.
- Whisper fine-tuning has been studied for low-resource ASR using limited transcribed target-language speech.
- Few studies have examined modern ASR models for indigenous languages of the Amazon, making Baniwa evaluation an important research direction.
- This work explores adapting Whisper to Baniwa within the growing field of indigenous language technology.
3 The Baniwa Language
Baniwa is an Arawakan language with cultural and sociolinguistic importance, spoken across the Upper Rio Negro region. Its orthographic and phonological conventions create specific challenges for low-resource ASR.
- Linguistic Background: Baniwa belongs to the Arawakan language family and is spoken in the Upper Rio Negro region across Brazil, Colombia, and Venezuela.
- Linguistic Background: Baniwa has distinct phonological, lexical, and grammatical characteristics within the broader Arawakan language network.
- Sociolinguistic Context: Documentation, literacy, and educational efforts support Baniwa maintenance and transmission, amid language shift toward Portuguese and Nheengatu in some regions.
- Speech Technology Relevance: Speech technologies may facilitate transcriptions, digital archives, educational resources, and linguistic documentation for language preservation initiatives.
- Orthographic Characteristics: These orthographic and phonological characteristics make Baniwa a challenging case for multilingual ASR in low-resource settings.
- Orthographic Characteristics: Baniwa orthography represents long vowels through duplicated vowel graphemes, which may affect lexical distinctions.
- Orthographic Characteristics: Aspirated consonants are marked with h, while digraphs and doubled consonants represent additional phonological distinctions.
4 Corpus Description
The study uses a small, manually transcribed Baniwa corpus drawn from a language documentation project. Its recordings are primarily elicited isolated words and short utterances rather than conversational speech.
- Data Collection: The corpus comes from the Baniwa-Koripako Multimedia Dictionary, a project documenting and disseminating Baniwa lexical and cultural knowledge.
- Data Collection: Recordings primarily contain elicited lexical items and short expressions, differing from conversational ASR datasets.
- Data Collection: Manual transcriptions from the original documentation project provide the ground-truth annotations.
- Corpus Statistics: 1,373 recordings contain approximately 0.54 hours of speech, predominantly isolated words and short elicited utterances.
- Corpus Statistics: Audio durations range from 0.48 to 5.09 seconds, with a mean of 1.42 seconds and median of 1.37 seconds.
- Data Splits: The dataset was randomly split into approximately 90% training, 5% validation, and 5% testing subsets.
5 Methodology
The study fine-tuned Whisper Small on the available Baniwa speech corpus using supervised learning, with preprocessing and decoding choices adapted to the low-resource setting.
- Whisper Small was selected as a compromise between computational efficiency and recognition performance for low-resource experimentation.
- Audio recordings were resampled to 16 kHz and converted into log-Mel spectrograms using the Whisper feature extractor.
- Transcriptions were tokenized with the Whisper tokenizer, without language-specific normalization, spelling correction, or post-processing.
- Spanish transcription prompts were used because Baniwa is not among Whisper’s predefined language tokens.
- Supervised fine-tuning used Hugging Face Transformers, with training, validation, and test subsets divided according to the stated proportions.
- 300 optimization steps were adopted after longer schedules showed signs of overfitting without consistent validation improvement.
- Evaluation used periodic validation checks, without an external language model, pronunciation lexicon, augmentation, or post-processing.
6 Experimental Setup
The experimental setup summarizes the fine-tuning configuration and evaluates recognition with word- and character-level error metrics, where lower values indicate better performance.
- Whisper fine-tuning was documented through a training-configuration table and checkpoint evaluation procedure.
- WER measures the proportion of word-level errors between reference and predicted transcriptions.Substitutions, deletions, and insertions are normalized by the total number of reference words.
- CER evaluates transcription quality at the character level, motivated here by the corpus’s predominance of isolated words and short utterances.It counts character substitutions, deletions, and insertions relative to the reference character count.
- Lower WER and CER values indicate better recognition performance.
7 Results
Fine-tuning reduced training and validation losses and improved recognition metrics, with the strongest WER at step 200 and evidence that later training may overfit.
- Training loss decreased from 1.0883 at step 100 to 0.0669 at step 300, while validation loss decreased from 0.4313 to 0.3652.
- WER decreased from 55.0% at step 100 to 37.5% at step 200, while CER decreased from 10.1% to 7.5%.
- 37.5% was the best WER at step 200, while CER reached 7.3% at step 300.
- The results indicate that Whisper adapted to Baniwa using only 0.54 hours of speech recordings.
- WER increased from 37.5% to 40.0% between steps 200 and 300 despite continued validation-loss reduction.This divergence may indicate the onset of overfitting, making the step-200 checkpoint most effective by the primary metric.
- Meaningful Baniwa speech recognition performance was achieved with approximately 32 minutes of manually transcribed speech.
8 Discussion
Whisper produced meaningful Baniwa recognition results from extremely limited data, but corpus composition, overfitting, and language-specific orthographic features constrain interpretation and motivate further development.
- 0.54 hours of manually transcribed speech yielded a minimum WER of 37.5% and CER below 8% with Whisper adapted to Baniwa.The result demonstrates meaningful recognition performance in a highly constrained low-resource scenario.
- Baniwa’s corpus primarily contains elicited lexical items and short utterances averaging approximately 1.4 seconds.This reduces linguistic and acoustic variability relative to continuous speech recognition.
- Reported performance should be interpreted for isolated-word and short-utterance recognition rather than large-vocabulary continuous speech recognition.
- 200 training steps produced the best WER, while further optimization lowered losses without improving word recognition, suggesting overfitting.The behavior reflects limitations imposed by the corpus’s small size.
- Baniwa orthographic features including vowel duplication, digraphs, and doubled letters introduce transcription challenges for a multilingual model trained mainly on high-resource languages.
- Usable performance from a few hundred recordings highlights transfer learning’s potential for indigenous language technologies with scarce digital resources.
- Future work should examine larger corpora, language-specific normalization, post-processing, external language models, and detailed error analysis.These directions target both recognition accuracy and understanding of linguistically difficult phenomena.
9 Conclusion
This preliminary study evaluated Whisper fine-tuning for Baniwa using 1,373 manually transcribed recordings. Whisper Small achieved useful recognition performance, while corpus size limited adaptation and indicated overfitting beyond 200 steps.
- 1,373 manually transcribed recordings totaling approximately 32 minutes were used to evaluate multilingual-model adaptation for extremely low-resource Baniwa ASR.
- 37.5% WER and 7.45% CER were achieved by the best-performing Whisper Small model.
- Recognition performance did not improve beyond 200 training steps despite decreasing training and validation losses, suggesting overfitting.
- The results support multilingual ASR models as a potential basis for indigenous language technologies and provide an initial Baniwa speech-recognition benchmark.
Declarations
The declarations report no specific funding, conflicts of interest, or applicable ethics and consent requirements. The speech corpus is access-restricted, code is available upon request, and author contributions are specified.
- The authors received no specific funding and declared no conflict of interest.
- Ethics approval and consent to participate and publish were reported as not applicable.
- The speech corpus is not publicly available because of ownership and community access considerations, though access may be granted upon authorization.
- The training and evaluation source code is available from the corresponding author upon reasonable request.
- The authors’ contributions span experimentation, computational implementation, analysis, supervision, linguistic expertise, corpus access, and manuscript revision.