Source-linked AI summary
AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASR
Eugen Beck, Sarah Beranek, Uma Moothiringote, Daniel Mann, Wilfried Michel, Katie Nguyen, Taylor Tragemann
TL;DR
English ASR evaluation has limited coverage of spontaneous, long-form conversations across diverse accents. AppTek introduces a privately collected, role-played call-center corpus and benchmarks open-weight systems under different segmentation strategies. Results vary substantially by accent and segmentation, and lower average WER does not necessarily mean stronger accent robustness.
Problem
Many public English ASR benchmarks use short, read or prepared speech and provide limited evidence for long-form conversation across diverse accents.
Source: §1 · p. 1Method
The authors built an evaluation corpus of spontaneous, role-played call-center dialogues across fourteen English accents and benchmarked open-weight ASR systems using multiple segmentation strategies.
Source: §1 · p. 1, §3.1 · p. 2Results
Manual segmentation gives the lowest WER for most systems, but accent-level errors vary substantially and average WER does not automatically indicate accent robustness.
Source: §6 · p. 4Takeaways & Limitations
The results show that evaluating long-form conversational ASR requires attention to both segmentation strategy and accent-level performance.
Source: §6 · p. 4No formal inter-annotator agreement metric was computed, so residual transcription uncertainty may remain, especially in acoustically challenging segments.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Evaluating English ASR systems for conversational AI applications remains difficult, as many publicly available corpora are either pre-segmented into short segments, consist of read or prepared speech, or lack explicit dialect annotations to evaluate robustness for a diverse user base. This work presents the AppTek Call-Center Dialogues corpus, a collection of spontaneous, role-played agent-customer conversations spanning fourteen English accents covering sixteen service-oriented scenarios. The dataset was commissioned specifically for evaluation and none of the audio or text was publicly available prior to release, reducing the risk of overlap with existing large-scale pretraining corpora. We benchmark a set of open-source ASR systems under different segmentation approaches. Results show substantial variation across accents and segmentation methods, indicating that good performance on general American English benchmarks does not necessarily generalize to other accents.
1. Introduction
Standard English ASR benchmarks provide limited evidence about long-form spontaneous conversation across diverse accents, a gap relevant to conversational AI. The paper introduces a privately commissioned call-center dialogue corpus and evaluation results across accents and segmentation strategies.
1. Introduction
- Public benchmarks often emphasize short, read or prepared speech, leaving long-form spontaneous conversations and diverse accents less systematically evaluated.These conditions matter for call-center interactions containing disfluencies, repairs, named entities, and domain-specific vocabulary.Source: §1 · p. 1
- Using material unavailable before release is a pragmatic way to reduce possible overlap with training data for open-weight ASR models.The paper notes that training data may include transcripts or near-duplicates of publicly available test sets.Source: §1 · p. 1
- The authors present spontaneous, role-played call-center dialogues spanning fourteen English accents and sixteen topics.The corpus was created specifically for evaluation, with transcripts produced by professional annotators using multi-stage quality assurance.Source: §1 · p. 1
2. Related Work
Prior English ASR benchmarks tend to emphasize short, read speech, while conversational corpora address differing acoustic settings and often lack explicit global accent coverage. AppTek’s work targets long-form, interactive agent-customer conversations with accent-specific evaluation.
2. Related Work
- Widely used benchmarks such as LibriSpeech, Mozilla Common Voice, and FLEURS largely contain short, read speech, which underrepresents spontaneous phenomena.The paper notes that known intended text can reduce transcription effort and that pretrained models may have encountered it during training.Source: §2 · p. 1
- Hub5, AMI, and CHiME-6 cover telephone, meeting, and noisy-home speech, respectively, but not the target call-center setting or global English accent coverage.Source: §2 · p. 1
- Earnings-22 [7] includes varied English accents, but company-country labels weaken accent-specific conclusions and its calls include prepared speech with limited interaction.Its public audio and transcripts also may have been used to train large foundational models.Source: §2 · pp. 1–2
3. Dataset description
The dataset combines spontaneous role-played agent-customer conversations across English accents and service domains with verbatim transcription, documented speaker recruitment, and multi-stage quality assurance. A translated subset is also planned for blind evaluation.
3.1. Overview
- The evaluation corpus contains 128.6 hours of speech from 156 speakers across fourteen accents and 1,746 recordings, with about 8–11 hours per accent.It is designed for evaluation and analysis rather than model training.Source: §3.1 · p. 2
3.2. Transcription Conventions
- Transcription preserves disfluencies, repetitions, repairs, filled pauses, and non-standard grammar while rendering dialectal pronunciations in standard orthography.Dialect-specific lexical items are retained; numerals and symbols are written in spoken form.Source: §3.2 · p. 2
3.3. Speaker Recruitment and Demographics
- Recruitment defined fourteen accent categories, requiring adult speakers native to the target locale; accents were self-identified and verified, with 10–16 speakers per accent.Natural within-accent variation was accepted even though accents are treated as discrete evaluation groups.Source: §3.3 · p. 2
3.4. Recording Setup and Conditions
- Pairs recorded free-form agent-customer dialogues without scripted text, with limited scenario planning and sessions averaging 10.4 minutes.Topics covered sixteen service domains and included named entities and numerical expressions; participants used fictional but plausible entities for privacy.Source: §3.4 · p. 2
3.5. Transcription Process
- Professional annotators transcribed recordings without automatic transcript bootstrapping, followed by review and automated validation checks.Guided recognition flagged segments when edit distance exceeded the threshold and recognition confidence met the threshold; 10% were flagged, and about 40% of that sample had minor issues fixed.Source: §3.5 · p. 2
3.6. Multilingual Extension
- About five hours of recordings were translated into Chinese, German, Japanese, and Spanish for blind evaluation at the 2026 IWSLT shared task.The translated subset follows the original segmentation and includes speakers from multiple accents.Source: §3.6 · pp. 2–3
4. Benchmarking
The benchmark evaluates open-weight ASR systems under multiple segmentation strategies and finds performance sensitive to both segmentation and accent. Manual segmentation performs best for most systems, while accent-level error gaps remain substantial.
4.1. Evaluation Setup
- The evaluation compares diverse open-weight systems using manual, AppTek, Silero, and fixed 30 s and 60 s segmentation strategies, with identical segmentation across models.WER was aggregated per session and scoring included normalization; dataset-specific normalization reduced WER by approximately 0.8–1.1% absolute across models and test sets.Source: §4.1 · p. 3
4.2. Results
- Manual segmentation yields the best WER for nearly all evaluated systems, while Qwen3-ASR models reach their lowest WER with fixed 60 s chunking.Without external segmentation, meaningful results were reported only for Kyutai STT 2.6B (13.9% WER) and Parakeet 0.6B TDT (v2: 8.8%, v3: 10.4%).Source: Table 2 · p. 3, §4.2 · p. 3
- Several models show more than 10% absolute WER difference between their best- and worst-performing accents under Silero segmentation.Table 3 reports WER by accent for the evaluated models.Source: §4.2 · p. 3, Table 3 · p. 4
- Singapore, Chinese, Scottish, and Indian English accents tend to have higher error rates, while Australian and US General English tend to have lower WER.Source: §4.2 · p. 4
- Accent robustness does not track average WER: Canary-1B has a 26% relative accent gap at 11.2% average WER, while Parakeet V3 has a 48% gap at 9.2%.The paper concludes that long-form conversation, accent diversity, and segmentation sensitivity challenge evaluations based on short or read speech.Source: §4.2 · p. 4
5. Limitations
The dataset’s scope and sample characteristics constrain how broadly its results can be interpreted. Representation and transcription uncertainty also remain relevant limitations.
5. Limitations
- The corpus covers role-played call-center interactions, so its findings are bounded to that setting.Participants may also be unfamiliar with some domain-specific terms or expressions.Source: §5 · p. 4
- Gender representation is uneven across accent groups, which may influence acoustic variability.The corpus includes 102 female and 54 male speakers; the authors advise taking the imbalance into account.Source: §5 · p. 4
- Accent labels are self-reported and verified, but discrete categories do not fully capture within-accent and regional dialect diversity.Results should be read as performance on the represented speaker sample, not exhaustive coverage of each accent community.Source: §5 · p. 4
- No formal inter-annotator agreement metric was computed, so residual transcription uncertainty may remain, especially in acoustically challenging segments.The authors identify rapid speech and reduced articulation as particularly challenging.Source: §5 · p. 4
6. Conclusion
The AppTek Call-Center Dialogues provide a newly collected, long-form English ASR test set spanning accents and service scenarios. Benchmark results show sensitivity to accent and segmentation, with manual segmentation performing best for most systems.
6. Conclusion
- The corpus contains 129 hours of transcribed spontaneous, role-played conversations across fourteen English accents and sixteen service-oriented scenarios.It was collected from scratch rather than relying on publicly available sources, minimizing potential overlap with web-scraped training data.Source: §6 · p. 4
- Manual segmentation yielded the lowest WER for most evaluated systems, while error rates varied widely across accents and strong average performance did not ensure accent robustness.The gap between best- and worst-performing accents remained large even for strong average-performing models.Source: §6 · p. 4
7. Generative AI Use Disclosure
The disclosure describes generative AI tools used for proofreading and scoring-normalization support, with author review of generated output.
7. Generative AI Use Disclosure
- ChatGPT (GPT5.2 [17]) was used to proofread the paper.Any generative AI output was vetted by at least one author before inclusion.Source: §7 · p. 5
- The local gpt-oss-120B [18] model helped generate scoring-normalization mapping files and verify proper US English spelling.Any generative AI output was vetted by at least one author before inclusion.Source: §7 · p. 5
- At least one author vetted any generative AI output before it was included in the work.Source: §7 · p. 5