Source-linked AI summary
VoiceCodeBench: Evaluating Exact Structured-Token Recovery in Automatic Speech Recognition
Tyler Baumgartner, Brandon Tai, Lisa Kaelin-Martin, Candice Fan, Luc Debaupte, Bill Wang, Yi Zhong
TL;DR
VoiceCodeBench addresses the problem that low-WER ASR transcripts can still corrupt exact values required by software workflows. It introduces a raw-audio-only English benchmark with entity-sensitive recovery metrics, finding that WER is informative but insufficient for exact workflow-value recovery.
Problem
Common ASR evaluation emphasizes WER, although production voice workflows require exact written values that may be stored, routed, compared, or executed.
Method
VoiceCodeBench evaluates raw-audio ASR on human-recorded workplace segments using canonical entity recovery, CTEM, TSR, and per-type exact recovery alongside WER.
Results
WER is informative but does not determine CTEM or TSR, and the strongest baseline still leaves many recordings with at least one unrecovered workflow-critical value.
Takeaways & Limitations
Reporting WER with CTEM, TSR, and per-type recovery gives teams a more actionable view of readable transcripts, exact-value preservation, and needed safeguards.
Takeaways & Limitations
The benchmark is English-only, and exact scoring uses an LLM-assisted recoverability verifier that depends on verifier-model and prompt handling.
Abstract
from arXiv · showhide
Automatic speech recognition (ASR) systems are commonly evaluated with word error rate (WER), yet many voice workflows depend on exact written values for identifiers, paths, and measured quantities. A transcript can appear fluent and achieve low WER while corrupting a value that a downstream system must parse, store, or execute. We introduce VoiceCodeBench, a benchmark for evaluating exact structured-token recovery in English ASR. It contains 300 human-recorded workplace segments spanning eight workflow domains and 1,482 audited target entities across 26 entity types, each with a canonical written form recoverable from the audio. Under a raw-audio-only protocol, systems receive audio bytes without additional context or metadata. Alongside WER, we evaluate Canonical Token/Entity Match (CTEM), Task Success Rate (TSR), and per-type exact recovery. Across 12 baseline ASR systems, lower WER generally corresponded to better structured-token recovery but did not fully determine it: Spearman correlations were -0.73 for both WER versus CTEM and WER versus TSR. The strongest baseline by TSR reached only 68.7%, leaving nearly one third of recordings with at least one unrecovered workflow-critical value. These results show that entity-sensitive metrics are needed to assess whether ASR output preserves exact values that production systems must parse, route, store, compare, or execute.
Introduction
VoiceCodeBench addresses the gap between broad ASR transcript evaluation and production workflows that require exact written-value recovery. It benchmarks this requirement with entity-sensitive metrics alongside WER.
- Exact written values matter because voice transcripts can serve as software input for storage, routing, comparison, or execution.
- WER can underrepresent structured-value errors because punctuation, casing, separators, token boundaries, digits, and normalization may determine the value.
- Existing ASR benchmarks primarily target transcript fidelity, semantic similarity, named-entity recognition, or context use rather than exact recovery from raw audio alone.
- VoiceCodeBench contains 300 human-recorded workplace segments spanning eight domains and 1,482 target entities across 26 entity types.
- The benchmark uses raw-audio-only evaluation and reports WER alongside CTEM, TSR, and per-type exact recovery.
Methods
The methods construct an entity-first, human-recorded English benchmark with synthetic structured values and evaluate raw-audio ASR using recoverability-based entity scoring. The design supports controlled per-type analysis while treating exact written values as workflow-relevant outputs.
- The dataset contains 300 English segments across eight workflow domains and 1,482 target entities across 26 entity types.
- VoiceCodeBench is a diagnostic, test-only resource rather than a general-purpose ASR corpus or training set.
- Entity-first construction specifies domains, difficulty, counts, and types before generating synthetic values and workplace-style transcripts.
- Human speakers record acoustic forms containing dictated punctuation, spelling, casing, and formatting cues, while canonical forms encode the exact downstream written values.
- All systems receive only raw audio at inference, with provider defaults and no benchmark-specific prompting, custom vocabulary, or entity hints.
- Entity extraction accepts formatting variation only when the transcript preserves enough evidence to recover the exact canonical value.
Results
VoiceCodeBench evaluates structured-token recovery across a broad workplace benchmark and shows that WER is informative but insufficient for exact workflow values.
- WER correlated with CTEM and TSR at Spearman ρ = -0.73 for each metric, indicating strong but incomplete association.Lower WER is better, whereas higher CTEM and TSR are better.
- 2,489 of 17,784 model-entity decisions were failures, with URLs, commands, email addresses, file paths, and postal addresses comprising 59.6% of failures.The paper treats raw failure counts descriptively because prevalence and difficulty both affect them.
- Median CTEM was lowest for commands, URLs, file paths, postal addresses, and email addresses, while conventional numeric forms were recovered more reliably.The hardest classes depend on symbols, separators, or exact token boundaries.
- Baseline examples distinguish recoverable formatting variation from unrecoverable loss of digits, symbols, separators, or units.
Discussion
The results show that WER cannot substitute for entity- or workflow-level correctness: strong systems can still corrupt critical values, especially in punctuation-sensitive classes.
- WER is useful but incomplete as a predictor of application-facing ASR reliability.Lower WER generally corresponded to better structured-token recovery, but the association did not fully determine correctness.
- 68.7% was the best TSR, leaving nearly one third of recordings with at least one unrecovered workflow-critical value.
- Deepgram Nova-3 batch had the lowest WER and highest TSR, while ElevenLabs Scribe v2 batch achieved the highest CTEM.Streaming usually reduced CTEM and TSR relative to paired batch systems, with OpenAI as the exception in this run.
- A provider that looks best under WER may not minimize value-level repair work, and entity recovery may still fail to produce successful workflows.
- The largest performance gaps occurred for URLs, commands, file paths, email addresses, postal addresses, and environment variables.These classes require preserving symbols, separators, digit sequences, casing cues, or token boundaries.
- CTEM, TSR, and per-type recovery distinguish broad transcript accuracy, value-level correction burden, workflow automation readiness, and fragile entity classes.These metrics support targeted mitigations such as selective confirmation or rejecting systems whose failures cluster in costly values.
Punctuation and Normalization as Correctness
VoiceCodeBench treats punctuation and normalization as components of structured-value correctness because formatting can determine whether downstream software receives the intended value.
- Dashes, dots, slashes, underscores, equals signs, leading zeros, decimal points, units, and token boundaries may define the value itself.
- The benchmark distinguishes acoustic forms from canonical forms and evaluates whether transcripts preserve enough evidence to recover the intended canonical value.A phone number can remain recoverable when hyphens are omitted if its digit sequence is intact.
- Baseline failures involving URLs, commands, paths, email domains, units, and environment variables change the value available to downstream software.
- Workflow-oriented ASR evaluation should treat punctuation-sensitive and normalization-sensitive values as first-class targets rather than secondary presentation features.
Intended Use
VoiceCodeBench is intended as a comparative diagnostic benchmark for structured-token reliability, not a general ASR quality measure. Its focused English workplace scope and scoring assumptions constrain how results should guide deployment decisions.
- VoiceCodeBench supports provider comparison, regression tracking, per-entity risk analysis, and application-specific safeguard decisions.Strong aggregate performance may still leave weak recovery for file paths, email addresses, or currency amounts requiring validation, confirmation, constrained decoding, or repair.
- Low WER and readable transcripts are insufficient deployment gates when ASR output feeds operational workflows.Applications should evaluate recoverability of acted-on values and may use confirmation prompts, typed validation, constrained inputs, or human review for high-risk classes.
- The benchmark is comparative and diagnostic rather than a general ASR measure, training corpus, fine-tuning set, or hidden leaderboard.Its scope excludes all speech styles, languages, acoustic environments, and conversational settings, while public labels require reporting configuration and evaluation details.
- VoiceCodeBench is English-only, so multilingual structured-token evaluation requires additional design rather than direct translation.Punctuation, spelling, number, address, and spoken-symbol conventions vary across languages.
- The benchmark focuses on compact workplace-style dictation and should not be generalized to all speech-recognition use cases.It does not cover casual conversation, meetings, overlapping speech, broadcast audio, voice search, noisy field recordings, or long-form dictation.
- The scenarios and values are synthetic despite human-recorded audio, which may limit representation of naturally occurring workflow speech.Generated scenarios may be cleaner or more compact than real support calls, developer conversations, clinical documentation, logistics workflows, or financial operations.
- Exact scoring uses policy choices and currently depends on LLM-assisted recoverability verification.The flexible scoring handles formatting variation but requires borderline-case handling and introduces dependence on the verifier model and prompt.
- Commercial ASR systems change over time, so benchmark results should include evaluation dates and configuration details.The benchmark is most useful as a repeatable evaluation protocol rather than a one-time static ranking.
Ethical and Privacy Considerations
VoiceCodeBench addresses workflow risks from corrupted structured values while emphasizing responsible interpretation of its diagnostic results. Privacy protections reduce exposure of sensitive values, but released audio can still carry speaker-identifying characteristics.
- The primary privacy risk is speaker identifiability because released audio can contain voice characteristics that identify or profile speakers.Contributors consent to dataset use and release, metadata is limited, and sensitive or routable structured values are avoided by construction.
- Future extensions could broaden evaluation across languages, speaker characteristics, recording conditions, and contextual ASR settings.Proposed additions include multilingual coverage, responsible accent and region analysis, acoustic variation, and domain labels or candidate values.
- The benchmark connects transcript evaluation to workflow risk by testing exact recovery of values that downstream systems depend on.Its design combines entity-first construction, acoustic and canonical annotations, raw-audio-only evaluation, and entity-sensitive metrics.
- WER is informative but does not determine CTEM or TSR, while difficult cases concentrate in punctuation-, separator-, and boundary-sensitive entity types.Reporting WER with CTEM, TSR, and per-type recovery identifies where additional safeguards are needed.
Data Availability
VoiceCodeBench is released publicly as a transparent, test-only diagnostic benchmark with the artifacts needed to reproduce evaluation. It is not intended for training or fine-tuning.
- VoiceCodeBench is publicly released as a test-only benchmark rather than a training, fine-tuning, or post-training resource.The dataset and evaluation scripts are available through a Hugging Face dataset repository.
- The release includes audio, reference transcripts, entity annotations, metadata, scoring scripts, baseline outputs, result tables, and documentation.Because labels are public, reported results should disclose model versions, evaluation dates, inference settings, and benchmark-specific processing.
Appendix: Domain and Difficulty Slices
Appendix slices provide descriptive CTEM and TSR context by workflow domain and difficulty band, averaged across the 12 released baseline systems. They are not intended as provider rankings.
- Tables 8 and 9 report descriptive CTEM and TSR slices averaged across the 12 baseline systems.The slices cover workflow domains and difficulty bands, respectively.
- Domain and difficulty slices should be interpreted as secondary context because entity mix and entity load differ across groups.
Appendix: Generation Tooling
The dataset-generation workflows used repository-aware Codex skills to create and iterate transcript data while preserving domain, entity, acoustic, and canonical constraints.
- Codex operated as a repository-aware LLM agent through separate skills for single-segment generation and dataset-wide iteration.
- The workflows preserved each segment’s domain, scenario, difficulty, entity constraints, and acoustic and canonical annotations.
- The generated metadata retained template, acoustic, and canonical transcript layers in data/metadata.jsonl.
- The workflows checked recoverability and taxonomy alignment during dataset construction.
Appendix: Verifier Prompt
The verifier checks each listed entity against only the ASR transcript, requiring enough evidence to recover the exact canonical value and returning schema-conforming JSON.
- The verifier uses the ASR output, called the “STT transcript,” as the transcript being checked.
- Each gold entity is checked independently, with one result returned for every target_index.
- The entity taxonomy includes structured types such as email addresses, phone numbers, paths, URLs, commands, measurements, dates, times, and domain terms.
- The target acoustic field supplies the expected spoken form, while the canonical field supplies the exact value to recover.
- An entity is marked present only when the transcript contains enough evidence to recover its exact canonical value.
- Casing, punctuation, spacing, and formatting differences are accepted only when the same value remains recoverable.
- Wrong, missing, extra, or substituted letters, digits, separators, units, dates, times, amounts, or words that change the target value are rejected.
- The output copies target_index, type, and canonical exactly, includes supporting transcript evidence and a short reason, and contains only valid JSON.