PaperlayerView exact revision on arXiv

Source-linked AI summary

SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition

Biraj Subedi

arXiv:2609.01737v1cs.CL

TL;DR

Visually mediated mobile payments in Nepal limit independent access for visually impaired users, while Nepali ASR research and public datasets have not addressed financial commands. The paper builds a Nepali financial-speech dataset and adapts Whisper large-v2 with LoRA; reported results include a 67.2% relative WER reduction and a rise in Transaction Success Rate from 1.67% to 33.33%.

  • Problem

    Nepali ASR systems target general speech, and no public Nepali financial-speech dataset existed to measure performance on commands with dense numerals and proper nouns.

    Source: §1 · p. 1

  • Method

    The paper constructs NepFinSpeech-403 and conducts a controlled comparison of zero-shot and LoRA-adapted Whisper for Nepali financial speech.

    Source: §1 · p. 2

  • Results

    67.2% relative WER reduction and Transaction Success Rate improvement from 1.67% to 33.33% are reported for LoRA adaptation.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The Transaction Success Rate gain suggests that WER understates domain adaptation’s practical impact for slot-critical applications.

    Source: §8 · p. 9

    The 403-utterance dataset supports a feasibility demonstration rather than a claim of state-of-the-art absolute performance, and numeral accuracy is not yet production-ready.

    Source: §9 · p. 10

Abstract

from arXiv · show

Mobile payment applications in Nepal are graphically mediated and largely inaccessible to visually impaired users. This paper presents SpeakPay, a voice-first digital wallet, and documents the central technical contribution: a controlled study of domain adaptation for low-resource financial speech recognition. We introduce NepFinSpeech-403, a 403-utterance dataset of Nepali financial voice commands (send, load, and balance operations spanning 237 unique numerals), and fine-tune Whisper large-v2 with LoRA. On the held-out test set, the domain-adapted model reduces Word Error Rate from 129.95% (zero-shot baseline) to 42.58% --- a 67.2% relative reduction --- and improves Devanagari numeral recognition accuracy from 0.0% to 73.9%. We find that word-level metrics understate the practical task-level impact: domain adaptation improves the Transaction Success Rate from 1.67% to 33.33%, a roughly 20x gain. The improvement is consistent at the individual-utterance level (sign test, $p < 10^{-17}$) and across all command types. A data efficiency analysis shows that as few as 100 domain-specific utterances are sufficient to halve the zero-shot WER, with performance plateauing around 300 examples. Error analysis reveals systematic numeral confusion patterns (zero insertion/deletion, prefix hallucination) that account for the majority of remaining transaction failures. The trained system is deployed as a publicly accessible voice-first web application. All code, dataset, model weights, and this paper are released at https://github.com/subedibiraj/speakpay.

1. Introduction

Visually mediated mobile wallets make independent digital finance effectively inaccessible to many visually impaired people in Nepal, while Nepali ASR and public financial-speech data have not addressed financial commands. SpeakPay assembles a dataset and evaluates zero-shot against domain-adapted ASR, reporting large gains and a deployed voice-first wallet.

1. Introduction

  • Nepal’s visually mediated eWallet interfaces make independent digital finance effectively impossible for visually impaired users without sighted assistance.The population is estimated at roughly 95,000 individuals.Source: §1 · p. 1
  • Existing Nepali ASR systems target general speech, and no public Nepali financial-speech dataset existed to measure or improve performance on financial commands.Such commands require recognition of dense numeral sequences, proper nouns, and recurring sentence structures.Source: §1 · p. 1
  • The paper constructs a dataset and conducts a controlled comparison of zero-shot and domain-adapted ASR, replacing an earlier project version without rigorous evaluation or a reproducible benchmark.The rewrite adds a characterized dataset, reproducible LoRA fine-tuning pipeline, statistically grounded evaluation, and deployed application.Source: §1 · p. 2
  • 67.2% relative WER reduction and Transaction Success Rate improvement from 1.67% to 33.33% are the headline gains reported for LoRA fine-tuning.The work also reports a public voice-first eWallet application and a data-efficiency analysis finding most adaptation gain with 100–300 utterances.Source: §1 · p. 2

2. Related Work

Prior Nepali and multilingual ASR work largely evaluates general-domain speech, while this paper examines financial-language domain adaptation and task-level evaluation in Nepali. Its contribution is a domain-specific measurement and application study, not a new architecture or training method.

Nepali ASR

  • General-domain Nepali ASR systems report CER of 10.3% and 11.14%, but neither evaluates financial or transactional speech.The work distinguishes financial speech by its vocabulary distribution and numeral density.Source: §2 · p. 2

Multilingual and low-resource speech recognition

  • Multilingual benchmarks and models expand low-resource coverage, but the cited benchmark work measures general-domain performance and does not address this specific financial-domain gap.The paper positions its study as measuring domain mismatch for a high-stakes application.Source: §2 · p. 2

Parameter-efficient fine-tuning

  • LoRA freezes the base model and adds trainable low-rank matrices [11], with prior work finding it competitive for domain-specific Whisper adaptation at lower cost [13].QLoRA extends the approach with quantization; these approaches can make adaptation feasible on a consumer GPU and small dataset.Source: §2 · p. 2

Task-level ASR evaluation

  • Task-level metrics address a limitation of WER: Semantic Distance and IRER assess downstream meaning or joint intent-and-slot correctness.This work adopts an IRER variant, Transaction Success Rate, to test whether transcription errors cause transaction failures.Source: §2 · pp. 2–3

Accessible FinTech

  • Prior accessible voice-payment work focuses on architecture, USSD automation, and biometric security, whereas this study focuses on Nepali ASR adaptation and evaluation.The paper describes Nepali as lacking an existing accessible financial voice interface.Source: §2 · p. 3

Study scope

  • The study does not propose a new architecture or training method; it tests whether domain-specific LoRA closes the gap for low-resource financial voice interaction and deploys the result.Source: §2 · p. 3

3. Dataset: NepFinSpeech-403

NepFinSpeech-403 was assembled from prompted recordings of three types of Nepali financial operations, with manually verified transcripts. The dataset statistics section also identifies a residual “other” category as capturing an unanticipated funds-received intent.

3.1 Collection

  • Recordings cover fund transfers, wallet loads or deposits, and balance enquiries, collected through a browser microphone and manually verified against audio.Contributors were undergraduate students and staff; devices and acoustic conditions varied, files were WAV at 16 kHz, and speaker overlap across splits could not be ruled out because identities were not recorded per utterance.Source: §3.1 · p. 3

3.2 Statistics

  • Table 1 is identified as the composition table for NepFinSpeech-403.Source: Table 1 · p. 3
  • “Other” is a keyword-matched heuristic residual class, and it captures previously unaccounted-for third-person statements about funds received.The authors treat this pattern as a finding rather than annotation noise.Source: §3.2 · p. 3

4. Method

The method adapts Whisper large-v2 with LoRA while freezing most model parameters, a choice motivated by the small training set. Training uses a single consumer GPU and a fixed 300-step schedule.

4. Method

  • Whisper large-v2 is a 1.55B-parameter encoder-decoder Transformer that operates on log-Mel spectrograms.Source: §4 · p. 3
  • LoRA uses rank 32 and scaling 64 across attention and feed-forward projections, yielding about 8M trainable parameters while 99.4% of the base model remains frozen.Source: §4 · p. 3
  • With only 303 training utterances, full fine-tuning risks overfitting and catastrophic forgetting, making LoRA’s parameter efficiency important to feasibility.Source: §4 · p. 4
  • Training uses AdamW and fp16 on one RTX 3060 GPU, with effective batch size 16 and a maximum of 300 steps.The final checkpoint is selected at step 300 after validation checks every 75 steps; audio is resampled to 16 kHz.Source: §4 · p. 4

5. Results

Domain adaptation raises Transaction Success Rate from 1.67% to 33.33%, while word-level metrics understate its task-level impact. Results also show gains across utterances and command types, robustness under GSM band-limiting, and diminishing returns as training data grows.

5.1 Aggregate benchmark

  • The general-domain Nepali fine-tune (Dragneel/whisper-small-nepali, trained on OpenSLR-54) has 0.0% NumAcc; the domain-adapted model’s improvement is attributed specifically to financial-domain data with digit-form transcriptions.The general-domain model outputs number words rather than Devanagari digits, so the downstream intent parser cannot extract a digit sequence.Source: §5.1 · p. 4

5.2 Per-utterance significance

  • 130.33% mean per-utterance WER for zero-shot falls to 42.61% with domain adaptation, which performs better on essentially every test utterance.The paired comparison covers 60 utterances; the authors use a non-parametric sign test.Source: Table 3 · p. 5, §5.2 · p. 5

5.3 Task-level Interpretation Accuracy

  • Transaction Success Rate rises from 1.67% to 33.33%, though it remains too low for fully autonomous financial use without user confirmation.The authors report that this task-level gain is substantially larger than the relative WER improvement.Source: Table 4 · p. 5, §5.3 · p. 5

5.4 Per-intent breakdown

  • Domain adaptation improves WER across all three command groups, with the largest absolute gain on send utterances.Send is the most frequent intent and has the highest numeral and proper-noun density.Source: Table 5 · p. 6, §5.4 · p. 6

5.5 Acoustic Robustness

  • GSM band-limiting adds 4.1 percentage points of WER, while performance collapses at 0 dB SNR.The authors identify GSM band-limiting as the most relevant simulated condition for phone-based payments.Source: Table 6 · p. 6, §5.5 · p. 6

5.6 Data Efficiency

  • WER drops sharply with the first 100 training examples, then improves gradually; Transaction Success Rate peaks at N = 300.Returns diminish beyond N = 200, with a slight regression at N = 403; the authors characterize 100–300 utterances as sufficient for most adaptation gain.Source: Figure 1 · p. 7, §5.6 · p. 7

6. Error Analysis

Domain adaptation improves numeral exact-match accuracy, but the remaining errors expose both systematic numeral confusions and gaps in the dataset’s three-intent taxonomy. These findings point to confirmation or explicit handling of utterances outside the defined command types.

6.1 Numeral Confusions

  • Numeral exact-match accuracy rose from 0.0% to 73.9% after domain adaptation.The remaining test errors include single-numeral mistakes and utterances outside the dataset’s send/load/balance taxonomy.Source: §6.1 · p. 7
  • In 25 of 60 test utterances, the model transcribed exactly one numeral incorrectly; patterns included trailing-zero insertion/deletion, prepended 5s, and phonetic confusions such as 12 and 1, distributed in Figure 2.Source: §6.1 · pp. 7–8
  • Because numeral errors can undermine transactions, the paper argues financial ASR should pair raw transcription with secondary confirmation steps.Source: §6.1 · p. 8

6.2 Taxonomy Gaps

  • The “other” category includes third-person reports of received funds, which contributors may have included after interpreting prompts broadly.Source: §6.2 · p. 8
  • A three-class intent classifier could misclassify these “receive” statements; production systems should add a fourth class or reject out-of-taxonomy utterances.The current application parser was not retrained to add the class.Source: §6.2 · p. 8
  • Unrelated demographic statements occurred in 4 of 60 test utterances and appeared identically in both models’ transcriptions.The authors say the small count cannot support a strong causal claim, though it is consistent with occasional prompt-following drift during collection.Source: §6.2 · p. 8

7. System Architecture and Deployment

SpeakPay connects a web wallet to a self-hosted LoRA-adapted speech recognizer and presents spoken confirmation before financial actions. Early informal testing informed design changes, while a structured usability study with visually impaired participants remains future work.

7. System Architecture and Deployment

  • The Next.js wallet uses Supabase-backed PostgreSQL for accounts and balances, while a self-hosted FastAPI service runs LoRA-adapted inference and a rule-and-confidence classifier parses transcripts.The database uses an atomic SQL transfer function; inference is self-hosted after unreliable cold starts on the free hosted-inference tier.Source: §7 · p. 9

7.1 Informal Usability Observations

  • Informal testing with 3–5 blindfolded sighted volunteers was exploratory, and testers completed some tasks but encountered interaction friction.The sessions had no formal protocol, timing instrumentation, written consent process, preserved logs, or quantitative metrics.Source: §7.1 · p. 9
  • Those observations motivated adapted ASR, an explicit voice-confirmation step before financial actions, and self-hosted inference.Source: §7.1 · p. 9
  • A structured evaluation of the current system with visually impaired participants remains future work and is identified as the key gap in accessibility evaluation.Source: §7.1 · p. 9

8. Discussion

The discussion emphasizes that task-level gains exceed the word-level improvement, while remaining errors and evaluation scope constrain deployment and cross-study comparisons. It also reports data-efficiency findings and a general-domain baseline whose numeral format limits downstream use.

8. Discussion

  • Transaction Success Rate rose from 1.67% to 33.33% (~20x), alongside a 67.2% relative WER reduction, indicating larger task-level than word-level gains.The authors argue that WER understates practical impact for slot-critical applications.Source: §8 · p. 9
  • With 50 transcribed utterances, the adapted model already halved zero-shot WER, and performance plateaued around 300 examples.The slight drop at N = 403 versus N = 300 is consistent with mild overfitting or train/test-split noise at this sample size.Source: §8 · pp. 9–10
  • The 33.33% Transaction Success Rate remains insufficient for autonomous payments, with zero insertion/deletion accounting for most remaining transaction failures.The authors suggest numeral-sequence post-processing or a small numeral-specific language model as possible approaches without retraining.Source: §8 · p. 10
  • The 42.58% WER should not be directly compared with the 10–11% CERs reported for general-domain read speech, given their different domains and evaluation protocols.The authors identify their own zero-shot versus adapted results on the same held-out set as the relevant comparison.Source: §8 · p. 10
  • On the test set, general-domain Whisper small, fine-tuned on 154 hours of general Nepali speech, achieved 106.32% WER versus zero-shot Whisper large-v2’s 129.95%, but 0% numeral accuracy because training transcribed numbers in word form.Source: §8 · p. 10

9. Limitations and Future Work

The evaluation is limited by a small dataset and an imperfect baseline comparison, and the paper identifies unresolved data, annotation, taxonomy, and accessibility-evaluation gaps.

9. Limitations and Future Work

  • 403 utterances (303 for training) make this a feasibility demonstration, not a claim of state-of-the-art absolute performance; 73.9% NumAcc is not production-ready for payment use.Numeral transcription is identified as the application's most safety-critical metric.Source: §9 · p. 10
  • The general-domain baseline uses Whisper small rather than large-v2, and its 0% numeral accuracy is partly attributable to word-form versus digit-form transcription.A controlled comparison would fine-tune the same large-v2 base on a size-matched sample of general Nepali speech.Source: §9 · p. 10
  • Because contributor identity was not recorded, speaker overlap across train and test cannot be ruled out, and speaker-disjoint splits could give a more conservative estimate.Source: §9 · p. 10
  • Transcripts were manually verified but not independently double-annotated, so no inter-annotator agreement statistic is available.Source: §9 · p. 10
  • The three-class intent scheme omits a recognizable “funds received” pattern; future work should add it as a fourth class and re-evaluate the NLP parser.Source: §9 · pp. 10–11
  • No usability study has been conducted with visually impaired users, the system's intended audience; a System Usability Scale study is planned.Source: §9 · p. 11

10. Conclusion

The conclusion presents NepFinSpeech-403 and reports substantial gains from LoRA fine-tuning Whisper large-v2, alongside data-efficiency findings and remaining error patterns. The system is deployed as a voice-first eWallet application, and the project materials are publicly available.

10. Conclusion

  • NepFinSpeech-403 is presented as a domain-specific Nepali financial speech dataset.Source: §10 · p. 11
  • LoRA fine-tuning of Whisper large-v2 reduced WER by 67.2% relative to the zero-shot baseline and improved 59 of 60 individual test utterances (p = 3.5 × 10−18).The conclusion also reports recovery of basic output coherence and gains across command types.Source: §10 · p. 11
  • As few as 100 utterances halve zero-shot WER, with performance plateauing around 300 examples; remaining errors include numeral confusions and a gap in the intent taxonomy.The reported numeral patterns include zero insertion/deletion and prefix hallucination.Source: §10 · p. 11
  • The system is deployed as a working voice-first eWallet application, and the codebase, training pipeline, dataset, model weights, and paper are available publicly.Source: §10 · p. 11

Ethics Statement

Contributors recorded prompted phrases voluntarily and consented to release their recordings; the dataset contains scripted commands rather than real transactions, and the application uses simulated balances.

Ethics Statement

  • Audio contributors voluntarily recorded prompted phrases through a web platform and consented to release recordings under CC-BY 4.0.Source: Ethics Statement · p. 11
  • No personally identifiable information, including names, phone numbers, or account details, was collected or stored.Source: Ethics Statement · p. 11
  • The dataset contains only scripted financial commands, while the deployed application uses simulated wallet balances and does not process real money.Contributor identity was not tracked per utterance, limiting demographic reporting while minimizing privacy risk.Source: Ethics Statement · p. 11

Related papers

Loading 2609.01737v1…