Source-linked AI summary
Mind the Accent Gap: British Accent Robustness in Speech-Driven Financial Voice Assistants
Aadam Haq, Oggi Rudovic, Malcolm Chadwick, Jay Rainey, Shucong Zhang, Ricardo Guerrero, Sourav Bhattacharya, Maja Pantic
TL;DR
Regional British accents challenge cascaded financial voice assistants, and production ASR must also balance robustness with streaming efficiency. This paper introduces CavaBench and evaluates ASR models through downstream tool-calling across accents and acoustic conditions. WER is strongly correlated with Exact Match overall, but accent-specific and task-critical errors can make it an incomplete indicator of task performance.
Problem
Public benchmarks do not target spoken financial queries annotated by British-accent group, limiting evidence on accent robustness in this setting.
Source: §1 · p. 1Method
The authors introduce CavaBench and evaluate ASR models across self-reported accents and acoustic conditions, measuring both WER and downstream tool-calling Exact Match.
Source: §1 · p. 1, §3 · p. 2, §5.2 · p. 3Results
WER and Exact Match are strongly correlated overall, but the relationship weakens for Scottish speech (r = −0.29), and task-critical token preservation matters beyond total word errors.
Source: §5.2 · p. 3, §5.3 · p. 4Takeaways & Limitations
CavaBench supports evaluating financial voice assistants at both transcription and downstream tool-calling levels across accents and acoustic conditions.
Source: §1 · p. 1, §3 · p. 2, §5.2 · p. 3
Abstract
from arXiv · showhide
AI voice assistants often use Automatic Speech Recognition (ASR) with LLM-based reasoning, yet existing systems struggle with regional British accents, including Scottish, Irish, and Welsh accents, since most ASR models are trained predominantly on American English voice data. Consequently, errors can carry through to the LLM stage, corrupting tool-call arguments and producing wrong or missing responses, which is especially costly in finance. Deployable ASR must also meet tight latency and memory budgets, making an accent-robust model choice even harder. We introduce CavaBench, the first internally collected benchmark of spoken financial queries, and use it to evaluate a range of ASR models and their end-to-end ASR-LLM pipeline behaviour across self-reported British accents. We find that WER strongly predicts downstream tool-calling accuracy ($r = -0.93$) but can fail to reflect task-level performance, with accent-related failures varying substantially across models and acoustic conditions. These findings guide the design of more inclusive, reliable voice-based financial assistants.
1. INTRODUCTION
Regional British accents expose weaknesses in cascaded voice assistants, while financial deployments also need ASR that is both accurate and responsive. CavaBench addresses the lack of a British-accent-labeled financial-query benchmark and evaluates accent and acoustic robustness through downstream tool-calling.
1. INTRODUCTION
- Most ASR systems are optimized for American English, leaving cascaded assistants fragile to Scottish, Irish, and Welsh accents and compounding errors that can misrepresent intent.The resulting underperformance is systematic for specific accent groups, rather than uniform across users.Source: §1 · p. 1
- Production-grade financial ASR must jointly balance accent robustness and streaming efficiency, because the accent gap changes with the latency operating point.A deployed component must emit incremental transcripts while supporting correct downstream tool-calling.Source: §1 · p. 1
- CavaBench fills the absence of a publicly available, British-accent-group-labeled spoken financial-query benchmark and evaluates ASR WER alongside downstream tool-calling Exact Match.It is internally collected, human-labeled, balanced across four British-accent categories, and tests accent and acoustic robustness.Source: §1 · p. 1
- Errors in merchant names, amounts, dates, durations, and temporal expressions can change a financial tool call, while errors in less informative words may have little task-level effect.The benchmark therefore links transcription error to downstream tool-calling rather than treating all word errors as equally consequential.Source: §1 · p. 1
2. RELATED WORK AND METHODS
Prior work documents demographic and accent disparities in ASR, but existing accent benchmarks do not evaluate financial queries through a cascaded assistant. This work motivates task-level evaluation because transcription errors can impair downstream tool-calling.
2. RELATED WORK AND METHODS
- Earlier audits report ASR disparities across speaker demographics and British varieties, while VCTK, EdAcc, and Common Voice evaluate general accent robustness rather than financial queries.Reported American-versus-British WERs include 9.4% versus 15.2–16.6% on VCTK and approximately 11% versus 14–15% on Common Voice.Source: §2 · pp. 1–2
- CavaBench pairs accent variability with financial queries and evaluates the full cascaded pipeline, addressing a gap left by prior work focused on ASR in isolation.Prior studies found higher WER for African American speakers, performance correlated with country of origin [4], and British-variety bias reflecting existing prejudice.Source: §2 · pp. 1–2
- The evaluated systems span FastConformer-transducer, attention encoder–decoder, audio-conditioned language-model, and hosted-API families, with differing streaming and offline capabilities.Parakeet TDT uses full-context attention for offline decoding, whereas Nemotron uses cache-aware limited-context processing and an RNN-T decoder for incremental low-latency processing.Source: §2 · p. 2
- Because the transcript is the LLM’s sole textual representation of intent, errors in amounts, dates, or merchant names can redirect tool calls and risk materially incorrect financial decisions.This motivates measuring tool-calling Exact Match alongside transcription-level metrics.Source: §2 · p. 2
3. CAVABENCH: DATASET
CavaBench is an internally recorded benchmark of realistic spoken financial queries, grouped by self-reported accent and designed to support accent comparisons separately from acoustic robustness testing.
3. CAVABENCH: DATASET
- CavaBench contains 2,919 retained utterances from 90 participants, recorded in a clean acoustic environment to isolate accent as the primary variable.Participants recorded at least one round of 20 naturally phrased queries sampled from a pool of 200; reference transcripts came directly from the prompts.Source: §3 · p. 2
- The dataset groups self-reported accents into Scottish, Southern English, Other British, and Non-British categories, with approximately 20–26 speakers per group.Other British bundles Northern Irish, Mancunian, Yorkshire, Welsh, and Midlands accents; group counts range from approximately 685 to 784 utterances.Source: §3 · p. 2
- The 200-query pool covers spending summaries, budget tracking, merchant lookups, trend analysis, and numerical queries, with 3,079 utterances recorded before quality inspection.Utterances average 4.73 seconds, range from 2.16 to 20.82 seconds, and total 3.94 hours; acoustic robustness is evaluated separately.Source: §3 · p. 2
4. EXPERIMENTAL SETUP
The evaluation compares streaming and offline ASR models, including in-house fine-tuned variants, and measures downstream tool-calling accuracy under multiple acoustic conditions. It also reports deployment-relevant latency and memory trade-offs.
4. EXPERIMENTAL SETUP
- The evaluation includes Nemotron Streaming at two lookahead settings with baseline and fine-tuned variants, Parakeet offline variants, chunked Qwen3-ASR, Whisper references, and Cohere Offline.Parakeet v2 is fine-tuned in-house with a gated encoder adapter and TDT loss; Qwen3-ASR pseudo-streaming processes audio in one-second chunks.Source: §4 · pp. 2–3
- In-house fine-tuning is limited to open-weight models by regulatory and data-governance constraints, and adaptation investigates additional in-domain data.The Nemotron corpus includes British-accented financial queries under telephony-style channel and noise conditions; shorter lookaheads and a larger Parakeet-RNNT model underperformed.Source: §4 · pp. 2–3
- Nemotron Streaming has roughly 3.5× lower latency and less than half the memory footprint of chunked Qwen3-ASR, whose non-causal encoder re-decodes a growing prefix.None of the other evaluated models are natively streaming.Source: §4 · p. 3
- Tool-calling Exact Match requires all five fields—metric, time period, group by, filter, and compare with—to match the gold call.Gold labels use majority votes from ten-fold LLM validation of each of the 200 reference queries, followed by manual inspection; recordings are tested across six acoustic-condition groups.Source: §4 · p. 3
5. RESULTS
Across accent and acoustic conditions, ASR performance varies by model, and aggregate WER does not always capture downstream tool-calling outcomes. Error analysis shows that preserving task-critical words and intent can matter more than the total number of transcription errors.
5.1. Accent-robustness results (Clean condition)
- Qwen3-ASR-1.7B Chunking had the largest accent spread (9.4%), driven by Scottish WER of 22.2%; Cohere Offline had 15.3% overall WER and a 1.6% spread.Source: §5.1 · p. 3
- Nemotron fine-tuning improved WER (18.2% to 16.1%) and accent spread (2.3% to 1.5%); Parakeet’s spread fell (2.7% to 1.7%) despite marginal WER change (16.6% to 16.5%) and no EM gain.In the Clean-only condition, EM ranged from 75.1% (Nemotron Base-LA6) to 80.7% (Qwen3-ASR-1.7B Offline); Nemotron Streaming LA13 improved from 75.5% to 78.2% with fine-tuning.Source: Table 1 · p. 3, §5.1 · p. 3
5.2. Robustness across acoustic conditions
- Overall WER and EM were strongly anti-correlated across models (r = −0.93), but the relationship weakened for Scottish (r = −0.29); longer lookahead modestly improved both metrics.Across LA6 to LA13, WER improved by 1.5–1.8% and EM by 1.5–1.9%.Source: §5.2 · p. 3
- WER worsened monotonically from Clean (6.0%) through Reverb (9.8%), Noise (11.8%), Telephony (12.9%), and Overlap (20.0%) to Environment (28.6%).Whisper Turbo reached 29% WER in Environment, while Qwen3-ASR-1.7B Offline was most robust throughout, retaining the highest EM except for Reverb; fine-tuned models reduced the Telephony gap.Source: §5.2 · p. 3, Figure 2 · p. 4, Figure 3 · p. 4
5.3. Error analysis
- In Clean, 21.9% of tool-call predictions were incorrect, with time period the largest failure mode (33.5%); Other British was lowest-EM under most individual conditions for every model.Compare with (25.4%) and group by (25.3%) were the next largest failure modes; the authors identify accent diversity, rather than any single accent, as the harder challenge.Source: §5.3 · p. 4
- A common UK retailer name was transcribed correctly in only 56.3% of 2,703 utterances, while “last quarter” survived verbatim in just 43.3% of 61,786 transcriptions.Day counts were 85.9–98.4% correct, but a systematic 60→“sixteen”/“six” substitution could return a confident wrong answer, computing a 60-day query over 6 or 16 days instead.Source: §5.3 · p. 4
- In two inspected cases, merchant-name destruction or a dropped imperative triggered abstention, while a lower-WER system preserved the reference on the same audio.The authors characterize abstention as a retryable stall, unlike a silent wrong answer.Source: §5.3 · p. 4
- WER is necessary but not sufficient: Qwen3-ASR-1.7B Chunking had the worst observed WER (49.8% at overlap +5 dB) yet retained 67% EM there, with “groceries” at 86.8% versus 76.3% for Whisper Turbo.Source: §5.3 · p. 4
6. CONCLUSION AND FUTURE WORK
Across CavaBench, WER and Exact Match are strongly correlated overall, but accent-specific challenges and task-critical tokens complicate aggregate performance. The robustness gap traces to training data rather than architecture, and targeted fine-tuning substantially closed it for a smaller streaming model.
6. CONCLUSION AND FUTURE WORK
- WER and Exact Match are strongly correlated overall, but a Scottish WER spike breaks the correlation and Other British accent diversity poses a distinct challenge.Other British bundles several distinct regional accents into one category.Source: §6 · p. 4
- Preserving task-critical merchant names, amounts, and dates matters more than reducing aggregate word errors.Source: §6 · p. 4
- The robustness gap traces to training data rather than architecture.Source: §6 · p. 4
- Targeted fine-tuning closed much of the robustness gap for the smaller streaming model.The authors point to efficient streaming models trained on more diverse data as a path toward closing the gap with offline counterparts.Source: §6 · p. 4
- The CavaBench study examines how accent, model architecture, and acoustic conditions jointly shape end-to-end ASR accuracy.Source: §6 · p. 4